Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a595dd96ae |
No files matched your search
@@ -29,6 +29,7 @@ first look at a new machine with one command.
|
||||
|
||||
```sh
|
||||
make build
|
||||
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
./sfdupes scan /srv
|
||||
./sfdupes report > dupes.tsv
|
||||
./sfdupes trees > dupetrees.tsv
|
||||
@@ -42,14 +43,10 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
Both cover the whole database, or only the paths given after them, as in
|
||||
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||
count.
|
||||
`info` prints the database's path, size and record count.
|
||||
|
||||
Root keeps the database in `/var/lib/sfdupes/db.sqlite`. Any other user keeps it
|
||||
in `sfdupes/db.sqlite` in the XDG state directory, `~/.local/state` by default
|
||||
on Linux. `SFDUPES_DATABASE` names another file (see "Location"). The first
|
||||
`scan` creates the database.
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
|
||||
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
||||
errors go to stderr. stdout and stderr are a program's two output streams. The
|
||||
@@ -117,27 +114,22 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||
directory it cannot read, and about a new or changed file it cannot read. It
|
||||
skips the path and deletes the records at or below it (see "Rules for the
|
||||
walk").
|
||||
|
||||
An unmounted disk leaves its mount point empty, so `scan` deletes every record
|
||||
under it, and the next scan reads every file on the disk again. Test the mount
|
||||
point of each disk at or below `/srv` with its own `mountpoint -q`, joined with
|
||||
`&&`, and test `/srv` only when it is one. As the file `/etc/cron.d/sfdupes`:
|
||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
||||
|
||||
```
|
||||
30 3 * * * root if mountpoint -q /srv; then /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log; else echo "a disk is not mounted; scan of /srv skipped"; fi
|
||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||||
```
|
||||
|
||||
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
|
||||
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` directly
|
||||
before `/usr/local/bin/sfdupes`.
|
||||
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the
|
||||
command on the same line.
|
||||
- `scan` writes nothing to stdout. Its stderr, appended here to
|
||||
`/var/log/sfdupes.log`, holds:
|
||||
- one line naming the database;
|
||||
- progress lines, as described under "Progress";
|
||||
- a warning for each path it skips;
|
||||
- a line for each system, temporary or cache directory the walk leaves out
|
||||
(see "Rules for the walk");
|
||||
- a line for each system or temporary directory the walk leaves out (see
|
||||
"Rules for the walk");
|
||||
- a line for each path left out because its filesystem is one the kernel,
|
||||
the core of the operating system, makes up or keeps in memory, such as
|
||||
`tmpfs`;
|
||||
@@ -156,9 +148,8 @@ point of each disk at or below `/srv` with its own `mountpoint -q`, joined with
|
||||
- `report`, `trees` and `info` need only read access to the database. The umask
|
||||
is the setting that removes permissions from the files a process creates.
|
||||
Under the usual umask of `022`, the first scan creates the database readable
|
||||
by every user. Another user can then run those three commands against root's
|
||||
database, with `SFDUPES_DATABASE=/var/lib/sfdupes/db.sqlite` before the
|
||||
command.
|
||||
by every user. An unprivileged user can then run those three commands against
|
||||
root's database.
|
||||
|
||||
### Reading the reports
|
||||
|
||||
@@ -267,8 +258,8 @@ In order:
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||
sfdupes report [PATH...] > dupes.tsv
|
||||
sfdupes trees [PATH...] > dupetrees.tsv
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
|
||||
sfdupes --version
|
||||
@@ -297,9 +288,9 @@ The reference below is also the program's specification.
|
||||
### Settings
|
||||
|
||||
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
|
||||
database is at its default path (see "Location"). There is no command-line
|
||||
flag for it. The path names the file exactly, whatever characters it holds. A
|
||||
relative path is relative to the working directory.
|
||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
||||
it. The path names the file exactly, whatever characters it holds. A relative
|
||||
path is relative to the working directory.
|
||||
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
||||
while it lists the trees. It also sets how many files `scan` reads at once
|
||||
while it hashes them. The default is the number of CPUs. `N` must be at
|
||||
@@ -309,9 +300,8 @@ The reference below is also the program's specification.
|
||||
of directories on other filesystems, as the same flag does for `du` and
|
||||
`rsync`. By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
||||
excludes. Those are the system, temporary and cache directories the walk
|
||||
otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules
|
||||
for the walk").
|
||||
excludes. Those are the system and temporary directories the walk otherwise
|
||||
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
||||
below it. `PATH` may be a directory or a file. The flag may be given more than
|
||||
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||
@@ -563,7 +553,7 @@ walk practical on large or busy storage pools.
|
||||
`scan` prints a line on stderr naming the path and its filesystem type:
|
||||
|
||||
```
|
||||
scan: left out /dev/shm (tmpfs, not a persistent filesystem)
|
||||
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
|
||||
```
|
||||
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
@@ -573,7 +563,7 @@ walk practical on large or busy storage pools.
|
||||
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||
|
||||
```
|
||||
scan: left out /srv/cache (--exclude)
|
||||
scan: skipped /srv/cache (--exclude)
|
||||
```
|
||||
|
||||
The path is not counted as skipped. The scan keeps the records at and below
|
||||
@@ -581,17 +571,12 @@ walk practical on large or busy storage pools.
|
||||
`--no-default-excludes` does not change this rule.
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold the kernel's view of the running system, temporary
|
||||
files, and caches. A cache here is data that programs and package managers
|
||||
download or build again on their own. The default excludes depend on the
|
||||
system, and `sfdupes scan --help` lists those of the system it runs on. In the
|
||||
lists, `*` stands for any one directory name, so `/home/*/.cache` matches
|
||||
`/home/alice/.cache`. On Linux they are:
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
the system it runs on. On Linux they are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
|
||||
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
|
||||
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
```
|
||||
|
||||
On macOS they are:
|
||||
@@ -599,21 +584,13 @@ walk practical on large or busy storage pools.
|
||||
```
|
||||
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
||||
/System/Volumes/VM, /System/Volumes/Data
|
||||
/private/var/folders, /Library/Caches
|
||||
/private/var/root/Library/Caches, /private/var/root/.cache
|
||||
/private/var/root/.npm, /private/var/root/.cargo/registry
|
||||
/private/var/root/go/pkg/mod
|
||||
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
|
||||
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
|
||||
```
|
||||
|
||||
`/private/var/folders` holds each user's temporary files and caches on
|
||||
macOS. Root's home on macOS is `/var/root`, which the walk reaches as
|
||||
`/private/var/root`. A walk of `/` on macOS reaches everything in
|
||||
`/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
|
||||
macOS uses to show the data volume inside the system volume. Walking
|
||||
`/System/Volumes/Data` as well would record each of those files twice. Any
|
||||
other system has no default excludes.
|
||||
A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
|
||||
through firmlinks. Firmlinks are the links macOS uses to show the data
|
||||
volume inside the system volume. Walking `/System/Volumes/Data` as well
|
||||
would record each of those files twice. Any other system has no default
|
||||
excludes.
|
||||
|
||||
An operand at or inside a default exclude is walked, so
|
||||
`sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
|
||||
@@ -622,7 +599,7 @@ walk practical on large or busy storage pools.
|
||||
not enter, `scan` prints a line on stderr naming it:
|
||||
|
||||
```
|
||||
scan: left out /var/tmp (default exclude)
|
||||
scan: skipped /var/tmp (default exclude)
|
||||
```
|
||||
|
||||
The directory is not counted as skipped. `scan` prints the filesystem line
|
||||
@@ -682,18 +659,6 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),
|
||||
|
||||
`removed` counts deleted records, which are not part of the files seen.
|
||||
|
||||
The warning for a skipped path names the path once and gives the reason:
|
||||
|
||||
```
|
||||
scan: skipped /srv/a/closed (permission denied)
|
||||
scan: skipped /srv/a/pipe (a FIFO)
|
||||
```
|
||||
|
||||
The summary counts each path named in such a warning. A line for a path left out
|
||||
on purpose starts with `scan: left out` instead, and is not counted. A search of
|
||||
the log for `scan: skipped` therefore finds the paths the summary counts. When a
|
||||
file with several hard links cannot be read, each of its paths gets a warning.
|
||||
|
||||
#### Stopping a scan
|
||||
|
||||
`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the
|
||||
@@ -721,12 +686,11 @@ and `scan` leaves it ignored.
|
||||
|
||||
### `report` mode
|
||||
|
||||
`report` reads every record from the database. It takes optional `PATH`
|
||||
operands. With none, it uses every record. With one or more, it uses only the
|
||||
records at or below them (see "Operands of `report` and `trees`"). `report`
|
||||
never opens a scanned file. It checks that each path in a group still exists,
|
||||
and looks at no other path. It never looks up an operand. The check is `lstat`,
|
||||
the system call that looks up a path without following a symlink:
|
||||
`report` reads every record from the database. It takes no positional arguments.
|
||||
A positional argument is anything on the command line that is not a flag, such
|
||||
as the paths `scan` takes. `report` never opens a scanned file. It checks that
|
||||
each path in a group still exists, and looks at no other path. The check is
|
||||
`lstat`, the system call that looks up a path without following a symlink:
|
||||
|
||||
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||
exist.
|
||||
@@ -758,27 +722,6 @@ with one signature come together, and so do the records of one size. So `report`
|
||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||
smaller size, it writes them, ordered by `first` path.
|
||||
|
||||
#### Operands of `report` and `trees`
|
||||
|
||||
`report` and `trees` take the same `PATH` operands:
|
||||
|
||||
- A record is used when its path is an operand or lies below one. `/srv/photos`
|
||||
covers `/srv/photos/x` but not `/srv/photos2/x`.
|
||||
- The records used are grouped, ordered and totalled as they are without
|
||||
operands. A copy outside the operands is not reported. With copies `/a/f`,
|
||||
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
|
||||
row, `/srv/photos/x` and `/srv/photos/y`.
|
||||
- A relative operand is joined to the working directory as `pwd -P` prints it.
|
||||
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
|
||||
elements and repeated or trailing slashes are dropped. Each `..` drops the
|
||||
element before it.
|
||||
- No operand is looked up on the filesystem. A symlink in an operand is not
|
||||
followed, and an operand need not exist. The records hold real paths (see
|
||||
"Operands" under "`scan` mode"), so an operand that passes through a symlink
|
||||
matches no record.
|
||||
- An operand that matches no record is not an error; it adds nothing to the
|
||||
report.
|
||||
|
||||
#### Report output format
|
||||
|
||||
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
||||
@@ -806,8 +749,7 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate groups;
|
||||
- the dupe files, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
@@ -818,12 +760,11 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
|
||||
### `trees` mode
|
||||
|
||||
`trees` reads the same database as `report` and takes the same `PATH` operands
|
||||
(see "Operands of `report` and `trees`"). It reports whole duplicate directory
|
||||
trees. Two directories are duplicate trees when they hold the same relative
|
||||
paths, and each path holds a file with the same signature in both. It rebuilds
|
||||
the directory tree from the paths in the records, split on `/`. Like `report`,
|
||||
it never opens a scanned file and never looks up an operand. It checks only that
|
||||
`trees` reads the same database as `report` and takes no positional arguments.
|
||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||
when they hold the same relative paths, and each path holds a file with the same
|
||||
signature in both. It rebuilds the directory tree from the paths in the records,
|
||||
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
||||
the directory at the top of each copy still exists, as `report` checks a path.
|
||||
|
||||
Definitions:
|
||||
@@ -851,11 +792,6 @@ Processing:
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- With operands, `trees` builds the tree from the records it uses. It reports
|
||||
only directories at or below an operand. Any other directory lies above an
|
||||
operand and can hold files that are not used, so its digest does not describe
|
||||
it. Such a directory is never reported and never implies a group of its
|
||||
children.
|
||||
- A directory that is gone is dropped from its group. A group left with fewer
|
||||
than two directories is not reported.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
@@ -909,12 +845,10 @@ directory's path is `/`.
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate-tree groups;
|
||||
- the dupe trees, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows, leaving out a row
|
||||
whose `dupe` lies inside another row's `dupe`.
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
|
||||
```
|
||||
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
||||
@@ -923,10 +857,8 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
||||
### `dwim` mode
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments. A positional
|
||||
argument is anything on the command line that is not a flag, such as the paths
|
||||
`scan` takes. Like `scan`, it should run as root, so that it can read every
|
||||
file.
|
||||
duplicate trees in one command. It takes no positional arguments. Like `scan`,
|
||||
it should run as root, so that it can read every file.
|
||||
|
||||
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
|
||||
|
||||
@@ -1005,22 +937,7 @@ All five subcommands work on one SQLite database file.
|
||||
#### Location
|
||||
|
||||
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
||||
empty. Otherwise it is at its default path, which depends on the user:
|
||||
|
||||
- For root, it is `/var/lib/sfdupes/db.sqlite`.
|
||||
- For any other user, it is `sfdupes/db.sqlite` in the XDG state directory. That
|
||||
directory is `$XDG_STATE_HOME` when that is set and not empty. Otherwise it is
|
||||
`~/.local/state` on Linux and `~/Library/Application Support` on macOS. An
|
||||
`XDG_STATE_HOME` that is not an absolute path is a fatal error, exit 1, for
|
||||
every subcommand. The error names `XDG_STATE_HOME`.
|
||||
|
||||
`scan` creates each missing directory on the way to the database with mode 755.
|
||||
At the default path of a user other than root, it uses mode 700 instead, as the
|
||||
XDG Base Directory specification asks. Other users cannot enter a directory with
|
||||
mode 700, so they cannot read a database in it.
|
||||
|
||||
All five subcommands use the same default, so a user's `report` reads the
|
||||
database that user's `scan` wrote.
|
||||
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
||||
|
||||
Each subcommand prints one line on stderr naming the database it uses, before
|
||||
anything else and whether or not stderr is a terminal. The line gives the path
|
||||
@@ -1060,9 +977,9 @@ Only one `scan` at a time can hold the lock.
|
||||
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
||||
lock is taken with the `flock(2)` system call, a request a program makes to
|
||||
the kernel, and only one process can hold it. The lock file's name is the
|
||||
database's path, with symlinks resolved, plus `.lock`. For root, by default,
|
||||
that is `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the
|
||||
same lock file.
|
||||
database's path, with symlinks resolved, plus `.lock`. By default that is
|
||||
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
||||
lock file.
|
||||
- `scan` takes the lock before it walks the filesystem or opens the database.
|
||||
Just before that, it creates the database file, empty, if it is missing. So
|
||||
the lock file is named the same way before the first scan as after. A symlink
|
||||
@@ -1351,8 +1268,7 @@ showing the count it stopped at.
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- an empty `--exclude` value, as in `--exclude ""`;
|
||||
- an empty `PATH` for `report` or `trees`, as in `sfdupes report ""`;
|
||||
- `info` or `dwim` with a positional argument.
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
the tracker yet is filed as an issue first.
|
||||
- Branch from `next`.
|
||||
- Do the work, with tests, in small focused commits.
|
||||
- Delete Next Step from `TODO.md`, in the same commit as the work.
|
||||
- Record the step at the top of Completed Steps, in the same commit as the work.
|
||||
- Push the branch and open a PR against `next`. Its title ends with
|
||||
` (closes #N)`.
|
||||
- An independent review gates each merge to `next`. Every finding is addressed
|
||||
@@ -15,7 +15,8 @@
|
||||
|
||||
- Pre-1.0.
|
||||
- The tracker is authoritative for the pre-1.0 backlog. The open issues under
|
||||
the `1.0.0` milestone are what remains before the tag.
|
||||
the `1.0.0` milestone are what remains before the tag. This file records
|
||||
history and process, not the queue.
|
||||
|
||||
# Next Step
|
||||
|
||||
@@ -23,6 +24,777 @@
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
|
||||
here, because a copy drifts as soon as the tracker moves.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- On a terminal, the line below the `walk` display shows the path of the last
|
||||
file the walk found, at each draw. Its control characters are written out, as
|
||||
`\t` or `\x1b`, so that the terminal does not act on them. The line is erased
|
||||
when the walk ends, and the next phase is drawn on it (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/160)
|
||||
|
||||
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
|
||||
`--exclude`, with the same meaning. Both commands register and check them with
|
||||
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
|
||||
|
||||
- The help of `sfdupes` and of `scan` names the database, its default path and
|
||||
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
|
||||
order of the rows and the escapes. The help of `info` lists its lines. The
|
||||
help of `dwim` says that it scans `/`, should run as root and keeps the scan
|
||||
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
|
||||
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||
enters nor records an excluded path, and keeps the records below it. An
|
||||
operand at or below one is a fatal error (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/163)
|
||||
|
||||
- `report` and `trees` leave out a copy that was deleted after the scan. Each
|
||||
checks only the copy itself with `lstat`: the file, or the directory at the
|
||||
top of a tree. A group left with one copy is not reported. A missing `scan`
|
||||
operand is still a fatal error (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/173)
|
||||
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
||||
`README.md` shows under "Subcommands" (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||
counts toward a tree's digest (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||
path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/157)
|
||||
|
||||
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
|
||||
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
|
||||
prints the report again (2026-11-06,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/154)
|
||||
|
||||
- `scan` resolves each operand to its real path, symlinks followed, before the
|
||||
walk. A symlink operand is scanned at its target instead of skipped. With
|
||||
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
||||
walks `sub` on its own instead of dropping it and deleting its records
|
||||
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
|
||||
|
||||
- `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS
|
||||
counts a new file's blocks only seconds after the write. The count can land in
|
||||
under a second. Both now say up to seconds after the write, as the comment on
|
||||
`diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/152)
|
||||
|
||||
- The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at
|
||||
random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks,
|
||||
and one ZFS commit can take a file through several sizes. The test that needs
|
||||
the size to change waits for it, for at most 30 seconds. Each test reads the
|
||||
size before and after its scan. The rewrite test rescans until the two reads
|
||||
match, then compares exactly. The other two fail only on a `disk_size` outside
|
||||
the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
|
||||
|
||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
||||
sentence. Every completed step is kept, with its date and its issue or branch.
|
||||
The policy-compliance checklist of 2026-07-22 is folded into its step
|
||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
||||
it. It leads with what the program is and how to use it. A reference section
|
||||
covers every setting and every subcommand. Development and testing come last.
|
||||
Every statement was checked against the code on `next`. Five comments in
|
||||
`internal/sfdupes/cli_test.go` cite "Settings", where their rules now live.
|
||||
One comment each in these files cites what the README now says:
|
||||
- `internal/sfdupes/scan_test.go`;
|
||||
- `internal/sfdupes/trees_test.go`;
|
||||
- `internal/sfdupes/exclude_test.go`.
|
||||
|
||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- Each record holds `disk_size`, the bytes the file takes on disk. That is its
|
||||
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
|
||||
makes. `trees` prints a `disk_size` column after `size`. It is the sum over
|
||||
the `dupe` copy's files. On a compressed or sparse file `disk_size` differs
|
||||
from `size`. A rescan updates the `disk_size` of an unchanged file whose size
|
||||
on disk changed, without reading the file (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||
|
||||
- The database keeps the number of records and the sum of their sizes in the one
|
||||
row of a `totals` table. Three triggers on `files` keep that row current.
|
||||
`info` reads that row. So do `scan`'s `load` phase and the `read` phases of
|
||||
`report` and `trees`, for their totals. No count reads every record any more.
|
||||
The triggers make each write to `files` slower. `scan` keeps SQLite's
|
||||
temporary data in memory. That wins back part of that time (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/146)
|
||||
|
||||
- an operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. `scan` warns about it, counts it as
|
||||
skipped and does not walk it (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/142)
|
||||
|
||||
- A file or directory removed while the scan runs is not an error, in the walk
|
||||
or in the hash phase. The scan prints no line for it and does not count it as
|
||||
skipped. When the first of several hard links to a file is gone, the hash
|
||||
phase reads the next one instead. On a terminal every progress display starts
|
||||
with the spinner, the hash phase's included. A phase with an exact total ends
|
||||
its line with its percent and a 10-column bar at the terminal's right edge.
|
||||
The bar no longer moves as the line's length changes. A line too wide for the
|
||||
terminal now loses its rate before its bar. The time remaining is labelled
|
||||
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
|
||||
|
||||
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
||||
bytes of a SHA-256. That is half the size of the hex text they held. The
|
||||
hashes stay raw from the hash phase through the database to `report` and
|
||||
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
||||
|
||||
- The hash phase gives three hashes to every file that the walk records. `head`
|
||||
and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
|
||||
SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
|
||||
100 MiB up. Gone are:
|
||||
- the 10 MiB and 50 MiB thresholds;
|
||||
- the size census, which counted the sizes of every walked file and of every
|
||||
record outside the operands. A file was hashed only when the census showed
|
||||
another file of its size;
|
||||
- the records without hashes;
|
||||
- the content phase.
|
||||
|
||||
A scan has four phases, and a rescan reads only new and changed files.
|
||||
`info` no longer counts the records with `head` and `tail` hashes, nor those
|
||||
with a content hash, since every record has all three (2026-10-25,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||
|
||||
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
|
||||
again from `sneak/prompts` `next` at `0653579`. Two steps now come before
|
||||
`ARG CACHEBUST`:
|
||||
- the module download in the `lint`, `test` and `builder` stages of the
|
||||
`Dockerfile`;
|
||||
- the package install in `builder`.
|
||||
|
||||
Docker keeps them until one of these changes:
|
||||
- the base image;
|
||||
- the install line;
|
||||
- `go.mod` or `go.sum`.
|
||||
|
||||
Every step that checks or compiles the code still comes after
|
||||
`ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache.
|
||||
README says the same (2026-10-18,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||
|
||||
- The release rule was set. Until 1.0, each tip of `main` that passes the build
|
||||
gets the next `v0.0.N` tag.
|
||||
`go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it.
|
||||
`go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||
the binary. That is the tag, or the pseudo-version of a `go install` build. It
|
||||
prints `dev` only when neither is known. README "Install" documented:
|
||||
- the rule;
|
||||
- what `@main` and `v0.0.1` give;
|
||||
- how to check an installed binary.
|
||||
|
||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116)
|
||||
|
||||
- `scan` creates the database's directory where the database path leads. The
|
||||
directory is the path with its last element cut off as text, so a `..` in it
|
||||
stays. A `..` after a symlinked directory therefore goes through the symlink,
|
||||
as it does when the file is opened. With
|
||||
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
|
||||
`/data/x`, it now creates `/data/new` and the database in it. It used to
|
||||
create `/srv/new` and then fail (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
||||
|
||||
- `scan` never records its own database. Once the database and its `.lock` file
|
||||
are open, it notes the device and inode of each of these:
|
||||
- the database file;
|
||||
- its `-wal` and `-shm` files;
|
||||
- the `.lock` file.
|
||||
|
||||
The walk passes over any file with the device and inode of one of them,
|
||||
however it reaches the file. It prints no line for such a file and counts
|
||||
nothing. A record an earlier scan left for one of them is deleted
|
||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
||||
|
||||
- `scan` never records or reads anything on a filesystem that is not persistent,
|
||||
such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or
|
||||
keeps in memory. `scan` matches them by device ID, however the walk or the
|
||||
content phase reaches a path. As the scan starts, it lists the mounted
|
||||
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
||||
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
||||
not enter the default excludes either. So `sfdupes scan /` skips the kernel's
|
||||
view of the running system and the temporary directories. On Linux those are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
```
|
||||
|
||||
On macOS they are:
|
||||
|
||||
```
|
||||
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
||||
/System/Volumes/VM, /System/Volumes/Data
|
||||
```
|
||||
|
||||
A Windows list exists in the code but is not used. An operand at or inside a
|
||||
default exclude is still walked. The scan prints a line for each path it
|
||||
leaves out, without counting it as skipped. The records there count as
|
||||
outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
|
||||
the default excludes only, and `scan --help` lists them (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||
|
||||
- A progress display's rate covers the items completed over the last 5 seconds.
|
||||
When stderr is not a terminal, it covers the items since the line before. It
|
||||
falls to 0 while nothing completes. A burst after a wait no longer reads as a
|
||||
steady rate of over 1,000 files a second. The time remaining still comes from
|
||||
the phase's average. Once no item has completed for 10 seconds, the line shows
|
||||
`stalled` and where the phase waits in place of the rate, as
|
||||
`stalled disk/db`. The places are:
|
||||
- `disk`;
|
||||
- `db`;
|
||||
- `out`, for stdout;
|
||||
- `cpu`.
|
||||
|
||||
A phase that may wait in two places names both. Each phase's word is set in
|
||||
the code that defines the phase. A line too wide for the terminal loses its
|
||||
bar first, then its rate or those words. Four parts stay whole:
|
||||
- the phase;
|
||||
- the progress;
|
||||
- the elapsed time;
|
||||
- the ETA, the time remaining.
|
||||
|
||||
The content phase had one display for its checks and one for its reads. Both
|
||||
lost those words on an 80-column terminal, at counts like those of README's
|
||||
example. That happened for most elapsed times and ETAs in minutes. Every
|
||||
other phase kept them whenever the elapsed time and the ETA were both in
|
||||
minutes. The content phase writes its last partial batch, the records not
|
||||
yet written, under a display of its own (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
|
||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value:
|
||||
- the path and where it came from;
|
||||
- the sizes of the database file and of its `-wal` file, when there is one;
|
||||
- when the database file was last changed;
|
||||
- the schema version;
|
||||
- the number of records;
|
||||
- how many records have `head` and `tail` hashes;
|
||||
- how many records have a content hash;
|
||||
- the sizes of all records added up.
|
||||
|
||||
It opens the database as `report` does and shows two phases on stderr,
|
||||
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
|
||||
one read-only transaction, a read of the database at one moment. The counts
|
||||
come from one query (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||
|
||||
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
||||
`[2/2] read`, with the progress display `scan` and `trees` use. The display
|
||||
runs from the moment `report` starts until the summary line. `open` says it is
|
||||
opening the database, also while it waits on a busy one. `read` counts the
|
||||
records, then reads them against that count in signature order. SQLite takes
|
||||
that order from the `files_signature` index without sorting, so the first
|
||||
record comes at once. `report` forms the duplicate groups itself and holds
|
||||
only those of the size it is reading. It writes them ordered by first path
|
||||
before it reads the next size. stdout and the summary line are unchanged, and
|
||||
SQLite no longer needs a temporary file to sort in (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
||||
|
||||
- `trees` shows its work on stderr in four phases, from `[1/4] open` to
|
||||
`[4/4] write`, each with the progress display `scan` uses. The display runs
|
||||
from the moment `trees` starts until the summary line. `open` says it is
|
||||
opening the database, also while it waits on a busy one. `read` counts the
|
||||
records, then reads them against that count. `group` counts the directories it
|
||||
groups, and `write` the groups it writes. A display's rate names what its
|
||||
phase counts:
|
||||
- files;
|
||||
- records;
|
||||
- directories;
|
||||
- groups.
|
||||
|
||||
stdout and the summary line are unchanged. When stdout and stderr are the
|
||||
same file, as one terminal is, no line of stdout shares a line with the
|
||||
display. When they are not, the display keeps being drawn while stdout waits
|
||||
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||
|
||||
- Each `scan` phase's progress display names the phase and its place among the
|
||||
five, as `[2/5] walk`. It runs from the moment that the phase starts. It
|
||||
shows:
|
||||
- the progress;
|
||||
- the elapsed time;
|
||||
- the ETA, the time remaining.
|
||||
|
||||
`load` counts the records first. The walk's total is estimated from the
|
||||
records under the operands. The content phase takes its total from SQLite's
|
||||
grouping alone. Until the first record is returned, it says it is searching
|
||||
the database. On a terminal a phase with an exact total draws its line
|
||||
followed by the bar of the progress-bar library. Any other phase draws a
|
||||
spinner followed by the line. Every display is drawn again several times a
|
||||
second. No line reaches the terminal's last column, however narrow the
|
||||
terminal is, even after it is made narrower during a phase. A line too wide
|
||||
leaves out its rate, then its bar. The rest stays whole. The line is cut
|
||||
short only when the rest alone does not fit, as below about 70 columns late
|
||||
in a large scan. When stderr is not a terminal, the line is printed every 5
|
||||
seconds whether or not an item completed (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
|
||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||
they use, as an absolute path. The line says whether that path is the default
|
||||
or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README
|
||||
"Database" says when `scan` writes each file (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
||||
|
||||
- The canonical files and model scripts were copied again from `sneak/prompts`
|
||||
`next` at `e3f79c4`:
|
||||
- `golangci-lint` v2.14.0;
|
||||
- lint and test are phases of the `Dockerfile` that write no image, and
|
||||
`make test` runs the suite under the race detector. Gone are
|
||||
`Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`;
|
||||
- every `docker build` in `script/` passes a new `CACHEBUST` value. Every
|
||||
stage of the `Dockerfile` declares it first, so every `RUN` step runs on
|
||||
every build. The steps that compile keep Go's build cache, and the tests
|
||||
run with `-count=1`;
|
||||
- `prettier` runs on the host, from the `node` and `yarn` that
|
||||
`script/bootstrap` installs. The `prettier` and `markdown` stages are
|
||||
gone, and the build stage installs `git` and `make` itself;
|
||||
- a new push cancels the workflow's older run on the same branch, and a run
|
||||
stops after 20 minutes;
|
||||
- a coding agent's settings file is deleted.
|
||||
|
||||
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
||||
`ARG CACHEBUST`:
|
||||
- the module download in `lint`, `test` and the build stage;
|
||||
- the install of `git` and `make` in the build stage.
|
||||
|
||||
Docker keeps them between builds. Every step that checks or compiles the
|
||||
code still runs on every build (2026-10-15,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95)
|
||||
|
||||
- With `-x`, an operand under another operand is no longer dropped when a
|
||||
directory between the two, the inner one included, is on a different
|
||||
filesystem than the outer one. The inner operand is then walked on its own.
|
||||
For a regular-file operand the check starts at its directory. The records of
|
||||
files gone from below it are deleted (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
||||
|
||||
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
||||
under its operand, instead of deleting them. It counts their sizes in the size
|
||||
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
||||
|
||||
- The hash and content phases open each file without blocking and check that it
|
||||
is still a regular file. A file that is now a FIFO or device node is warned
|
||||
about and skipped instead of hanging the scan (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/100)
|
||||
|
||||
- The scan lock file is named from the database's path with symlinks resolved,
|
||||
after `scan` creates the database file if it is missing. A scan through a
|
||||
symlink to the database therefore fails while another scan holds the lock
|
||||
(2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101)
|
||||
|
||||
- On a terminal, a warning for the last file of a phase no longer erases the
|
||||
phase's finished bar. The bar is drawn again below the warning (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/102)
|
||||
|
||||
- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus
|
||||
`mtime_nsec`, and compares it at that resolution. A same-size rewrite within
|
||||
the same second is hashed again (2026-10-07,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/12)
|
||||
|
||||
- The narration was cut from `TODO.md` Completed Steps and from the comments in
|
||||
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now
|
||||
branches from and merges to `next` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/49)
|
||||
|
||||
- `make test-race` ran the test suite under the race detector in a cgo-enabled
|
||||
container, outside `make check`. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
|
||||
under the race detector, in the `Dockerfile`'s Debian-based `test` phase.
|
||||
`make test-race` is gone (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
||||
|
||||
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
||||
rather than serve the gates, the lint and test steps, from cache. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
|
||||
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
|
||||
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39)
|
||||
|
||||
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the
|
||||
Gitea workflow checks it. All Markdown was reformatted (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||
|
||||
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
||||
|
||||
- Tests cover:
|
||||
- a missing database;
|
||||
- `scan` keeping stdout empty;
|
||||
- its skip warning;
|
||||
- the `report` and `trees` summary lines;
|
||||
- every subcommand going through `runE`.
|
||||
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||
|
||||
- `.golangci.yml` was replaced with the current canonical copy, which uses
|
||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||
|
||||
- A test fails when either `hashWorker` cancellation check in
|
||||
`internal/sfdupes/scan.go` is removed (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||
|
||||
- A database path holding `?`, `#` or `%` opens exactly the file it names
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
||||
|
||||
- `scan` rejects `--workers` below 1 as a usage error instead of running
|
||||
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
|
||||
|
||||
- A test fails when either walk cancellation check in `internal/sfdupes/scan.go`
|
||||
is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||
|
||||
- A test checks that `scan` refuses a database with another schema version
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||
|
||||
- Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected,
|
||||
and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles`
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||
|
||||
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/17)
|
||||
|
||||
- `scan` creates the schema in one transaction. A version-0 database with a
|
||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||
|
||||
- README documents install, Docker, a daily `cron` scan and how to read and
|
||||
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||
|
||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
|
||||
nothing changes owner. The build stage only compiles, as root, and the tests
|
||||
run as `nobody` in the `test` phase (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/43)
|
||||
|
||||
- `--version` prints `sfdupes VERSION` to stdout. README documents it and
|
||||
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
|
||||
|
||||
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed,
|
||||
deletes nothing more, and exits 1 (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/5)
|
||||
|
||||
- `report` and `trees` stream the records instead of holding them all in memory.
|
||||
The schema gains the `files_signature` index (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
||||
|
||||
- Progress prints at once on a non-terminal and uses a real terminal test.
|
||||
Warnings print through a spinner instead of racing its redraw (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||
|
||||
- These operands are warned about and skipped, and the records beneath them are
|
||||
kept:
|
||||
- symlinks;
|
||||
- sockets;
|
||||
- FIFOs;
|
||||
- device nodes;
|
||||
- `.zfs` directories.
|
||||
|
||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9)
|
||||
|
||||
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
||||
second `scan` fails at once with exit 1 (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/53)
|
||||
|
||||
- Tests cover stdout write failures in `report` and `trees`. README states that
|
||||
`| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null`
|
||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
|
||||
|
||||
- `report` and `trees` open the database read-only, and `scan` leaves it out of
|
||||
WAL mode, so reading needs only read access (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
||||
|
||||
- Four characters are escaped in `report`, `trees` and warning paths:
|
||||
- tab;
|
||||
- newline;
|
||||
- carriage return;
|
||||
- backslash.
|
||||
|
||||
The root directory's path is `/` (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||
|
||||
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
||||
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
||||
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
||||
fails if the context carries `.git` and the version is still empty, `dev` or
|
||||
`unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`),
|
||||
so it stamps the same value as `make build` (2026-10-02, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||
|
||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||
size, and README "Duplicate detection" documented every size threshold. A file
|
||||
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
|
||||
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
|
||||
phase. The content phase ran after the update phase. It read a file for
|
||||
`content` only when its size, `head` and `tail` matched another record's, from
|
||||
this scan or an earlier one. `content` was the whole file below 50 MiB, and
|
||||
gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
|
||||
changed since its record was written. `report` and `trees` left out any record
|
||||
without a `content` hash. The `content` column is part of the version 1
|
||||
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
|
||||
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
|
||||
|
||||
- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
|
||||
`.dockerignore` (2026-09-21, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||
|
||||
- The lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||
`Dockerfile.lint` were fixed. Both pins became the policy
|
||||
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||
without the false `(Debian-based)` note or the tag. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
||||
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone,
|
||||
and so is `script/verify-lint-image-pin`, which compared the two pins
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
|
||||
|
||||
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
|
||||
owner ruling the linter is never installed on a host, and
|
||||
`golangci-lint config verify` runs before `golangci-lint run`.
|
||||
`script/bootstrap` stopped installing or pinning the linter, and
|
||||
`script/verify-linter-pin` was retired. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
|
||||
`Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the
|
||||
build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`.
|
||||
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
|
||||
an image build may run `docker`, so the phase calls `golangci-lint` directly
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
|
||||
|
||||
- The Docker build stage's prerequisites were installed by running
|
||||
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
||||
`script/verify-linter-pin` failed the build unless the linter copied from the
|
||||
lint stage was the version that `script/bootstrap` pinned. Builds then took up
|
||||
to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
|
||||
filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
||||
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
|
||||
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/42)
|
||||
|
||||
- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and
|
||||
`script/docker` cannot report a pass they did not earn. The `Dockerfile`
|
||||
copies the tree before its gates. On an unchanged tree Docker served them from
|
||||
cache, and the build exited 0 without running them. The fix was a
|
||||
`CHECK_EPOCH` build argument. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
|
||||
passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile`
|
||||
declares it before every step that checks or compiles the code. Run as root,
|
||||
the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through
|
||||
the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody`
|
||||
(2026-08-09, branch `cibuild-cache-bust`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/32)
|
||||
|
||||
- `script/bootstrap` checked the installed `golangci-lint` version instead of
|
||||
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
|
||||
`go install` module reference derived from it. Any installed version that was
|
||||
not the pin was reinstalled, whether:
|
||||
- older;
|
||||
- newer;
|
||||
- absent;
|
||||
- unparseable.
|
||||
|
||||
`go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the
|
||||
first `golangci-lint` on `PATH`. So bootstrap re-read the effective version
|
||||
after installing. On a mismatch it printed both paths and both versions and
|
||||
exited non-zero. It did not reorder `PATH` or delete anyone's binary. The
|
||||
`--version` call kept its stderr and was bounded by `timeout(1)` where that
|
||||
exists. `git`, `make` and `go` kept presence-only checks. The version check
|
||||
was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again
|
||||
to a no-op. Stub runs of the script under `dash` covered:
|
||||
- a thirteen-input version-parse matrix;
|
||||
- a shadowed install, which must exit non-zero;
|
||||
- an install destination not on `PATH`;
|
||||
- `GOBIN` set;
|
||||
- a wedged binary, which must hit the timeout.
|
||||
|
||||
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
||||
hiding findings on `main`. Superseded by
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer
|
||||
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||
|
||||
- The hash worker pool unwinds on the error path. The pool is an owned,
|
||||
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
||||
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
|
||||
defers `pool.stop()`, which cancels and then drains `results` until the last
|
||||
goroutine has exited. Draining is what frees a worker parked on a send. `ctx`
|
||||
is passed from `cmd.Context()` as the first parameter, through:
|
||||
- `runScan`;
|
||||
- `syncScan`;
|
||||
- both worker pools;
|
||||
- the whole database layer.
|
||||
|
||||
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
|
||||
walk. A cancelled walk yields a partial size census, and every file that it
|
||||
never reached looks vanished to the update phase. That phase's own `BeginTx`
|
||||
also fails on the cancelled context before deleting anything. The guard is
|
||||
still the barrier once an interrupted scan may commit what it has. Tests
|
||||
drive `run(scan)` against a database whose insert trigger aborts. They
|
||||
assert that the scan fails instead of hanging, and that the goroutine count
|
||||
returns to its pre-scan baseline. Other tests cancel a scan part-way through
|
||||
the walk, deterministically, by counting the scan's own checks of
|
||||
`ctx.Done()`. They assert that it stops at the guard with every record
|
||||
intact. It then holds a partial size census and a still-populated record
|
||||
index. Direct tests cover the remaining cancellation branches of both pools.
|
||||
They test:
|
||||
- `sendEvent`;
|
||||
- the walk workers;
|
||||
- `dispatchDirs`;
|
||||
- `feedHashJobs`;
|
||||
- `hashWorker`;
|
||||
- `hashPhase`.
|
||||
|
||||
(2026-08-09, branch `hash-pool-cleanup`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/6)
|
||||
|
||||
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
|
||||
are gone. So the deferred `db.Close()` runs when a subcommand fails, and with
|
||||
it the SQLite WAL checkpoint. These return errors instead:
|
||||
- `runScan`;
|
||||
- `runReport`;
|
||||
- `runTrees`;
|
||||
- `loadRecords`;
|
||||
- `resolveRoots`.
|
||||
|
||||
The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a
|
||||
`fatalError`, anything a subcommand returned, to exit 1, and cobra's own
|
||||
argument and flag errors to exit 2. So a runtime failure is never reported
|
||||
as a usage error, nor printed with the usage text.
|
||||
`internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the
|
||||
exit codes from README "Error handling" and the stdout/stderr split. It also
|
||||
checks that a fatal error after the database is open leaves no `-wal` or
|
||||
`-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch
|
||||
`db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
|
||||
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
|
||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
||||
replaced with the canonical file. Four thresholds now live under
|
||||
`linters.settings`, as the v2 schema requires, so they are applied:
|
||||
- `lll`;
|
||||
- `funlen`;
|
||||
- `cyclop`;
|
||||
- `dupl`.
|
||||
|
||||
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||
|
||||
- `Makefile` targets were converted to scripts-to-rule-them-all `script/`
|
||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
||||
- `bootstrap`, `setup` and `projectname`;
|
||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
||||
|
||||
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||
|
||||
- The binary is the default `make` target. Plain `make` builds `sfdupes`, where
|
||||
it used to run `check` plus `build`. `make build` remains as an alias
|
||||
(2026-07-24, branch `make-default-target`)
|
||||
|
||||
- The phases became scan-wide, with concurrent operands and batched updates. All
|
||||
operands seed the shared walk pool, and every pass runs once over the whole
|
||||
scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
|
||||
and update cycles and their stderr announcements are gone. The update pass
|
||||
writes in batches, each in its own transaction. The filesystem is
|
||||
authoritative and the database follows it, so one scan's changes need not land
|
||||
all at once (2026-07-24, branch `scan-wide-phases`)
|
||||
|
||||
- The stat pass was split back out of the walk. Phases are strictly sequential
|
||||
again, per operand, with parallelism only inside each phase:
|
||||
- walk;
|
||||
- stat;
|
||||
- hash;
|
||||
- update.
|
||||
|
||||
The walk enumerates paths with per-directory workers, and the stat pass
|
||||
lstats them with per-file workers. That restores the exact total and ETA of
|
||||
the stat bar (2026-07-24, branch `parallel-phases`)
|
||||
|
||||
- Each operand is announced on stderr before its passes. With per-operand walk,
|
||||
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
|
||||
totals that looked like the whole run's (2026-07-24, branch
|
||||
`scan-operand-progress`)
|
||||
|
||||
- The walk became parallel. The walk pass was a single goroutine and took hours
|
||||
at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
|
||||
million files. It is now a walk by a pool of per-directory workers that
|
||||
records size and mtime as it goes. That folds away the separate stat pass and
|
||||
halves the metadata reads. Each `PATH` operand commits in its own transaction,
|
||||
so an interrupted scan keeps completed operands (2026-07-24, branch
|
||||
`parallel-walk`)
|
||||
|
||||
- `scan` gained a persistent database. It maintains an SQLite database that
|
||||
survives between runs, with the absolute path as each record's key. The driver
|
||||
is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
|
||||
new or changed files, by mtime and size. It deletes the records of files
|
||||
vanished from under the scanned operands and leaves records outside them
|
||||
untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
|
||||
database, with no positional arguments, instead of a scan stream. The database
|
||||
is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
|
||||
journaling plus a single-transaction update keep a report run during a scan
|
||||
safe (2026-07-24, branch `persistent-database`)
|
||||
|
||||
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
|
||||
was tagged, and `main` and the tags were pushed (2026-07-23)
|
||||
|
||||
- The `scan` command line was reworked. Required `PATH...` operands, parsed by
|
||||
cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
|
||||
flag follows the GNU convention and stops at filesystem boundaries, which the
|
||||
walk crosses by default (2026-07-23, branch `scan-required-paths`)
|
||||
|
||||
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
||||
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
||||
2026-07-22 found these gaps, and the branch closed them:
|
||||
- `.gitignore`, with the standard patterns for secrets and for OS, editor
|
||||
and Go files. The compiled `sfdupes` binary and `files.dat` had been
|
||||
sitting untracked in the tree;
|
||||
- `.editorconfig`;
|
||||
- `LICENSE` and the README License section. MIT was assumed from house
|
||||
convention, for the owner to confirm;
|
||||
- `REPO_POLICIES.md` in the repo root;
|
||||
- the canonical `.golangci.yml`, with its 150 findings fixed, so that
|
||||
`make lint` is clean;
|
||||
- the `Makefile` targets the policy requires:
|
||||
- `test` and `lint`;
|
||||
- `fmt` and `fmt-check`;
|
||||
- `docker` and `hooks`.
|
||||
|
||||
`check` depended on `build`, which wrote the binary. `make check` must
|
||||
not modify files, so `check` no longer depends on `build`;
|
||||
|
||||
- a test suite, which the policy required with a 30-second timeout and the
|
||||
conditional `-v` rerun. At 64% coverage, it covers:
|
||||
- parsing and grouping;
|
||||
- digests and suppression;
|
||||
- hashing and the scan pipeline;
|
||||
- a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had:
|
||||
- hash-pinned images;
|
||||
- a lint stage that failed fast;
|
||||
- a build stage that ran `make check`;
|
||||
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
|
||||
checkout action pinned by commit SHA;
|
||||
- the README sections the policy requires. The first is a Description whose
|
||||
first line names the project, its purpose and category, its license and
|
||||
its author. Then come Getting Started and Rationale, and at the end TODO,
|
||||
License and Author;
|
||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||
- the pre-commit hook, installed with `make hooks` once that target existed.
|
||||
|
||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||
|
||||
- `git init` was run, with a README-only first commit. The code baseline was
|
||||
committed on `main` (2026-07-22)
|
||||
|
||||
- The `scan`, `report` and `trees` subcommands were implemented (pre-git
|
||||
history)
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Possible later features, out of scope per README, are a byte-for-byte check of
|
||||
|
||||
+2
-4
@@ -5,10 +5,8 @@
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures that the other subcommands read. The
|
||||
// database is SFDUPES_DATABASE, or by default /var/lib/sfdupes/db.sqlite
|
||||
// for root and sfdupes/db.sqlite in the XDG state directory for any
|
||||
// other user.
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
|
||||
@@ -3,7 +3,6 @@ module sneak.berlin/go/sfdupes
|
||||
go 1.25.7
|
||||
|
||||
require (
|
||||
github.com/adrg/xdg v0.5.3
|
||||
github.com/rivo/uniseg v0.4.7
|
||||
github.com/schollz/progressbar/v3 v3.19.1
|
||||
github.com/spf13/cobra v1.10.2
|
||||
|
||||
@@ -1,5 +1,3 @@
|
||||
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
|
||||
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
|
||||
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
||||
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||
|
||||
+14
-42
@@ -34,14 +34,10 @@ const (
|
||||
|
||||
// databaseHelp is the paragraph of the root and scan help that names
|
||||
// the database (README "Location").
|
||||
const databaseHelp = `scan and dwim store what they find in a database file. report,
|
||||
trees and info read it. The file is $SFDUPES_DATABASE when that is set
|
||||
and not empty. Otherwise root uses /var/lib/sfdupes/db.sqlite. Any other
|
||||
user uses sfdupes/db.sqlite in the XDG state directory: $XDG_STATE_HOME,
|
||||
or ~/.local/state when that is unset or empty
|
||||
(~/Library/Application Support on macOS). An XDG_STATE_HOME that is
|
||||
not an absolute path is an error.
|
||||
`
|
||||
const databaseHelp = "scan and dwim store what they find in a database " +
|
||||
"file. report, trees\nand info read it. The file is $" + databaseEnv +
|
||||
" when that is set and\nnot empty, and otherwise " +
|
||||
defaultDatabasePath + ", where creating it\nneeds root.\n"
|
||||
|
||||
// escapesHelp is the paragraph of the report and trees help on how
|
||||
// paths are escaped (README "Report output format").
|
||||
@@ -53,13 +49,6 @@ const escapesHelp = `Four characters in a path are escaped, so each row stays on
|
||||
\r a carriage return
|
||||
`
|
||||
|
||||
// operandsHelp is the paragraph of the report and trees help on their
|
||||
// PATH operands (README "Operands of `report` and `trees`").
|
||||
const operandsHelp = `With PATH operands, only the records at or below them are used. A
|
||||
relative PATH is joined to the working directory. No PATH is looked up
|
||||
on the filesystem, so a symlink in one is not followed.
|
||||
`
|
||||
|
||||
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
|
||||
// summary in the help of their subcommands.
|
||||
const (
|
||||
@@ -72,7 +61,7 @@ row per duplicate file. The columns are:
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp + "\n" + operandsHelp
|
||||
` + escapesHelp
|
||||
|
||||
treesHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||
row per duplicate directory tree. The columns are:
|
||||
@@ -85,7 +74,7 @@ row per duplicate directory tree. The columns are:
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp + "\n" + operandsHelp
|
||||
` + escapesHelp
|
||||
|
||||
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
|
||||
value, separated by a tab:
|
||||
@@ -119,11 +108,6 @@ var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||
// would name the working directory.
|
||||
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||
|
||||
// errEmptyOperand is the usage error for an empty PATH operand of
|
||||
// report or trees, as from "$DIR" with DIR unset. Taken as a path, it
|
||||
// would name the working directory.
|
||||
var errEmptyOperand = errors.New("PATH must not be empty")
|
||||
|
||||
// Version is the build version, injected at link time via -ldflags
|
||||
// (see the Makefile); empty when the linker set none, as for a binary
|
||||
// built by go install. buildVersion turns it into what --version prints.
|
||||
@@ -245,20 +229,20 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
"print the version to stdout")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport + " [PATH...]",
|
||||
Use: cmdReport,
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Args: rejectEmptyOperand,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runReport(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees + " [PATH...]",
|
||||
Use: cmdTrees,
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Args: rejectEmptyOperand,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runTrees(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
@@ -403,18 +387,6 @@ func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||
return errEmptyExcludeValue
|
||||
}
|
||||
|
||||
// rejectEmptyOperand rejects an empty PATH operand of report or trees.
|
||||
// That is a usage error reported in one line, as for --workers below 1.
|
||||
func rejectEmptyOperand(cmd *cobra.Command, args []string) error {
|
||||
if !slices.Contains(args, "") {
|
||||
return nil
|
||||
}
|
||||
|
||||
cmd.SilenceUsage = true
|
||||
|
||||
return errEmptyOperand
|
||||
}
|
||||
|
||||
// runE adapts a subcommand implementation, or the version print, to
|
||||
// cobra's RunE. Cobra prints the error and the command's usage text for
|
||||
// every error RunE returns, but a subcommand that ran and failed has no
|
||||
|
||||
+33
-147
@@ -15,7 +15,6 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/adrg/xdg"
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
@@ -560,17 +559,13 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
// line.
|
||||
t.Setenv(databaseEnv, "")
|
||||
|
||||
dbPath, err := defaultDatabasePath(os.Geteuid())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Whether scan calls the database new depends on that machine too.
|
||||
scanWant := "scan: database " + dbPath + " (default)\n"
|
||||
scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
|
||||
|
||||
_, err = os.Stat(dbPath)
|
||||
_, err := os.Stat(defaultDatabasePath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
scanWant = "scan: new database " + dbPath + " (default)\n"
|
||||
scanWant = "scan: new database " + defaultDatabasePath +
|
||||
" (default)\n"
|
||||
}
|
||||
|
||||
missing := filepath.Join(t.TempDir(), "nope")
|
||||
@@ -588,23 +583,23 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
},
|
||||
cmdReport: {
|
||||
run: func() error {
|
||||
return runReport(cancelledContext(t), nil, io.Discard)
|
||||
return runReport(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "report: database " + dbPath + " (default)\n" +
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||
},
|
||||
cmdTrees: {
|
||||
run: func() error {
|
||||
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||
return runTrees(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "trees: database " + dbPath + " (default)\n" +
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||
},
|
||||
cmdInfo: {
|
||||
run: func() error {
|
||||
return runInfo(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "info: database " + dbPath + " (default)\n" +
|
||||
want: "info: database " + defaultDatabasePath + " (default)\n" +
|
||||
infoOpenLine + "\n" + infoOpenLine + "\n",
|
||||
},
|
||||
}
|
||||
@@ -631,6 +626,8 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
}{
|
||||
"no subcommand": {[]string{}, usageMarker},
|
||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
@@ -721,19 +718,6 @@ func TestRunHelp(t *testing.T) {
|
||||
t.Errorf("run(%v) stdout = %q, want nothing (data only)",
|
||||
args, got)
|
||||
}
|
||||
|
||||
// README §Location: the help of sfdupes and of scan says where
|
||||
// the database is, and that XDG_STATE_HOME must be an absolute
|
||||
// path.
|
||||
for _, want := range []string{
|
||||
"scan and dwim store", databaseEnv, rootDatabasePath,
|
||||
stateHomeEnv, "~/.local/state", "not an absolute path is an error",
|
||||
} {
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("run(%v) stderr = %q, want it to name %q",
|
||||
args, stderr.String(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The scan help lists --exclude with its argument.
|
||||
@@ -776,37 +760,36 @@ func TestRunScanUsageLine(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHelpDescribesOutput(t *testing.T) {
|
||||
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Settings: the help of report, trees, info and dwim says
|
||||
// what each prints. The help of report and trees also explains
|
||||
// their PATH operands. TestRunHelp checks the database paragraph
|
||||
// of the root and scan help.
|
||||
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||
operands := []string{
|
||||
"With PATH operands", "at or below them", "working directory",
|
||||
"No PATH is looked up", "not followed",
|
||||
// README §Settings: the help names the database, and says what
|
||||
// report, trees, info and dwim print.
|
||||
database := []string{
|
||||
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
|
||||
}
|
||||
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||
|
||||
// The subcommand whose help is printed.
|
||||
// The subcommand whose help is printed; none for the root.
|
||||
cases := []struct {
|
||||
subcommand []string
|
||||
want []string
|
||||
}{
|
||||
{nil, database},
|
||||
{[]string{cmdScan}, database},
|
||||
{
|
||||
[]string{cmdReport},
|
||||
slices.Concat([]string{
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n size ",
|
||||
"largest first",
|
||||
}, escapes, operands),
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdTrees},
|
||||
slices.Concat([]string{
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n files ",
|
||||
"\n size ", "\n disk_size ", "largest first",
|
||||
}, escapes, operands),
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdInfo},
|
||||
@@ -1013,7 +996,8 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||
|
||||
stderr := scanOK(t, dir)
|
||||
|
||||
warning := "scan: skipped " + unreadable + " (permission denied)\n"
|
||||
warning := "hash " + unreadable + ": open " + unreadable +
|
||||
": permission denied\n"
|
||||
if !strings.Contains(stderr, warning) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||
}
|
||||
@@ -1285,7 +1269,7 @@ func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
||||
|
||||
stderr := scanOK(t, link)
|
||||
|
||||
warning := "scan: skipped " + fifo + " (a FIFO)\n"
|
||||
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
||||
if !strings.Contains(stderr, warning) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||
}
|
||||
@@ -1320,7 +1304,7 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
||||
|
||||
stderr := scanOK(t, operand)
|
||||
|
||||
warning := "scan: skipped " + operand + " (a " + kind + ")\n"
|
||||
warning := "walk " + operand + ": skipping " + kind + " operand\n"
|
||||
if !strings.Contains(stderr, warning) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||
}
|
||||
@@ -1354,7 +1338,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
||||
writeFile(t, hourly, "f", pattern(1, 10))
|
||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
||||
|
||||
warning := "scan: skipped " + hourly + " (inside a .zfs directory)\n"
|
||||
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
|
||||
|
||||
for _, tc := range []struct {
|
||||
operands []string
|
||||
@@ -1406,7 +1390,7 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
||||
|
||||
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
||||
warning := "scan: skipped " + daily + " (inside a .zfs directory)\n"
|
||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 1 skipped\n"
|
||||
|
||||
@@ -1625,100 +1609,6 @@ func TestRunDatabasePathUsedAsGiven(t *testing.T) {
|
||||
})
|
||||
}
|
||||
|
||||
func TestRunNonRootUsesStateDirectory(t *testing.T) {
|
||||
// README §Location: without SFDUPES_DATABASE, a user other than
|
||||
// root keeps the database in the XDG state directory. scan creates
|
||||
// the missing directories with mode 0700, and report and trees read
|
||||
// the same file.
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root uses " + rootDatabasePath)
|
||||
}
|
||||
|
||||
t.Cleanup(xdg.Reload)
|
||||
|
||||
state := filepath.Join(t.TempDir(), "state")
|
||||
t.Setenv(stateHomeEnv, state)
|
||||
t.Setenv(databaseEnv, "")
|
||||
xdg.Reload()
|
||||
|
||||
assertRunsUseDatabase(t, filepath.Join(state, "sfdupes"), "db.sqlite")
|
||||
|
||||
for _, dir := range []string{state, filepath.Join(state, "sfdupes")} {
|
||||
fi, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got := fi.Mode().Perm(); got != stateDirPerm {
|
||||
t.Errorf("%s mode = %#o, want %#o", dir, got, stateDirPerm)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunNonRootDatabaseEnvDirectoryMode(t *testing.T) {
|
||||
// README §Location: for SFDUPES_DATABASE, scan creates each missing
|
||||
// directory with mode 755, for a user other than root too.
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("needs a user other than root")
|
||||
}
|
||||
|
||||
top := filepath.Join(t.TempDir(), "missing")
|
||||
sub := filepath.Join(top, "sub")
|
||||
t.Setenv(databaseEnv, filepath.Join(sub, "db.sqlite"))
|
||||
|
||||
scanOK(t, t.TempDir())
|
||||
|
||||
for _, dir := range []string{top, sub} {
|
||||
fi, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got := fi.Mode().Perm(); got != dbDirPerm {
|
||||
t.Errorf("%s mode = %#o, want %#o", dir, got, dbDirPerm)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRelativeStateHomeFails(t *testing.T) {
|
||||
// README §Location: without SFDUPES_DATABASE, a user other than
|
||||
// root whose XDG_STATE_HOME is not an absolute path gets an error
|
||||
// naming it from every subcommand, which exits 1 before it does
|
||||
// anything else.
|
||||
if os.Geteuid() == 0 {
|
||||
t.Skip("root does not use " + stateHomeEnv)
|
||||
}
|
||||
|
||||
t.Setenv(stateHomeEnv, "relative/state")
|
||||
t.Setenv(databaseEnv, "")
|
||||
|
||||
want := "sfdupes: " + stateHomeEnv +
|
||||
" must be an absolute path, got \"relative/state\"\n"
|
||||
|
||||
// A scan that got past the check would stop at this missing
|
||||
// operand, before it creates anything.
|
||||
missing := filepath.Join(t.TempDir(), "missing")
|
||||
|
||||
for _, args := range [][]string{
|
||||
{cmdScan, missing}, {cmdReport}, {cmdTrees}, {cmdInfo},
|
||||
} {
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitFatal || stderr.String() != want {
|
||||
t.Errorf("run(%v) = %d with stderr %q, want %d and %q",
|
||||
args, code, stderr.String(), exitFatal, want)
|
||||
}
|
||||
}
|
||||
|
||||
// dwim scans /, so runDwim gets the missing operand instead.
|
||||
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, io.Discard)
|
||||
if !errors.Is(err, errStateHomeNotAbsolute) {
|
||||
t.Errorf("runDwim: error = %v, want %v", err,
|
||||
errStateHomeNotAbsolute)
|
||||
}
|
||||
}
|
||||
|
||||
// holdScanLock takes the lock on the database at path, as a running
|
||||
// scan does, and holds it until the test ends. It fails the test when
|
||||
// the lock is already held.
|
||||
@@ -1949,13 +1839,9 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
||||
scanFixture(t)
|
||||
|
||||
cases := map[string]func(context.Context, io.Writer) error{
|
||||
cmdReport: func(ctx context.Context, w io.Writer) error {
|
||||
return runReport(ctx, nil, w)
|
||||
},
|
||||
cmdTrees: func(ctx context.Context, w io.Writer) error {
|
||||
return runTrees(ctx, nil, w)
|
||||
},
|
||||
cmdInfo: runInfo,
|
||||
cmdReport: runReport,
|
||||
cmdTrees: runTrees,
|
||||
cmdInfo: runInfo,
|
||||
}
|
||||
|
||||
for name, fn := range cases {
|
||||
|
||||
+15
-68
@@ -15,40 +15,28 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/adrg/xdg"
|
||||
"golang.org/x/sys/unix"
|
||||
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
||||
// disabled.
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// rootDatabasePath is the database root uses when SFDUPES_DATABASE is
|
||||
// not set.
|
||||
const rootDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
||||
// defaultDatabasePath is where the persistent scan database lives when
|
||||
// SFDUPES_DATABASE is not set.
|
||||
const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
||||
|
||||
// databaseEnv is the environment variable that overrides the database
|
||||
// path.
|
||||
const databaseEnv = "SFDUPES_DATABASE"
|
||||
|
||||
// stateHomeEnv is the environment variable that names the XDG state
|
||||
// directory.
|
||||
const stateHomeEnv = "XDG_STATE_HOME"
|
||||
|
||||
// schemaVersion is the database schema version this build reads and
|
||||
// writes, stored in PRAGMA user_version.
|
||||
const schemaVersion = 1
|
||||
|
||||
// dbDirPerm is the mode for a directory scan creates for root's
|
||||
// default database or for SFDUPES_DATABASE.
|
||||
// dbDirPerm is the mode for a database parent directory created by
|
||||
// scan.
|
||||
const dbDirPerm = 0o755
|
||||
|
||||
// stateDirPerm is the mode for a directory scan creates for the
|
||||
// database at a user's default path in the XDG state directory. It is
|
||||
// the mode the XDG Base Directory specification asks for, and it keeps
|
||||
// other users from reading the database, which lists every scanned
|
||||
// path.
|
||||
const stateDirPerm = 0o700
|
||||
|
||||
// dbFilePerm is the mode for a database file created by scan, the mode
|
||||
// SQLite gives a database file it creates.
|
||||
const dbFilePerm = 0o644
|
||||
@@ -156,48 +144,15 @@ var errScanRunning = errors.New("another scan is running")
|
||||
// 32 bytes of a SHA-256.
|
||||
var errHashSize = errors.New("hash is not 32 bytes")
|
||||
|
||||
// errStateHomeNotAbsolute reports an XDG_STATE_HOME that is set to a
|
||||
// path that is not absolute.
|
||||
var errStateHomeNotAbsolute = errors.New(
|
||||
stateHomeEnv + " must be an absolute path")
|
||||
|
||||
// defaultDatabasePath is the database used when SFDUPES_DATABASE is not
|
||||
// set: rootDatabasePath for root (euid 0), and db.sqlite in the
|
||||
// directory sfdupes under the XDG state directory for any other user.
|
||||
// xdg reads XDG_STATE_HOME and HOME once, when the program starts. For
|
||||
// any other user, an XDG_STATE_HOME that is set and not an absolute
|
||||
// path is an error, where xdg would use its default instead. The path
|
||||
// is joined as text, not cleaned, for the reason absolutePath gives.
|
||||
func defaultDatabasePath(euid int) (string, error) {
|
||||
if euid == 0 {
|
||||
return rootDatabasePath, nil
|
||||
}
|
||||
|
||||
stateHome := os.Getenv(stateHomeEnv)
|
||||
if stateHome != "" && !filepath.IsAbs(stateHome) {
|
||||
return "", fmt.Errorf("%w, got %q", errStateHomeNotAbsolute,
|
||||
stateHome)
|
||||
}
|
||||
|
||||
stateDir := strings.TrimSuffix(xdg.StateHome, "/")
|
||||
|
||||
return stateDir + "/sfdupes/db.sqlite", nil
|
||||
}
|
||||
|
||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||
// set and non-empty, defaultDatabasePath otherwise. It returns the
|
||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||
// path, made absolute by absolutePath, and where the path came from,
|
||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||
// prints first on stderr.
|
||||
func databasePath() (string, string, error) {
|
||||
path, source := os.Getenv(databaseEnv), "from "+databaseEnv
|
||||
if path == "" {
|
||||
def, err := defaultDatabasePath(os.Geteuid())
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
|
||||
path, source = def, "default"
|
||||
path, source := defaultDatabasePath, "default"
|
||||
if p := os.Getenv(databaseEnv); p != "" {
|
||||
path, source = p, "from "+databaseEnv
|
||||
}
|
||||
|
||||
abs, err := absolutePath(path)
|
||||
@@ -233,23 +188,15 @@ func absolutePath(path string) (string, error) {
|
||||
|
||||
// createDatabaseDirectory creates the directory the database at path
|
||||
// goes in, and any directory above it, if missing. path is absolute, as
|
||||
// databasePath returns it. Each directory gets mode dbDirPerm, or
|
||||
// stateDirPerm when databasePath uses a user's default path in the XDG
|
||||
// state directory: the user is not root and SFDUPES_DATABASE is unset
|
||||
// or empty. The directory is path with its last element cut off as
|
||||
// text, by filepath.Split. filepath.Dir would clean it, dropping each
|
||||
// ".." together with the directory name before it; when that name is a
|
||||
// symlink, the cleaned path names another directory than the one the
|
||||
// system reaches when it opens path.
|
||||
// databasePath returns it. The directory is path with its last element
|
||||
// cut off as text, by filepath.Split. filepath.Dir would clean it,
|
||||
// dropping each ".." together with the directory name before it; when
|
||||
// that name is a symlink, the cleaned path names another directory than
|
||||
// the one the system reaches when it opens path.
|
||||
func createDatabaseDirectory(path string) error {
|
||||
dir, _ := filepath.Split(path)
|
||||
|
||||
perm := os.FileMode(dbDirPerm)
|
||||
if os.Geteuid() != 0 && os.Getenv(databaseEnv) == "" {
|
||||
perm = stateDirPerm
|
||||
}
|
||||
|
||||
err := os.MkdirAll(dir, perm)
|
||||
err := os.MkdirAll(dir, dbDirPerm)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create database directory: %w", err)
|
||||
}
|
||||
|
||||
@@ -9,13 +9,10 @@ import (
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/adrg/xdg"
|
||||
)
|
||||
|
||||
// testDBPath returns a database path inside a fresh temp dir, with
|
||||
@@ -59,16 +56,11 @@ func TestDatabasePath(t *testing.T) {
|
||||
|
||||
t.Chdir(dir)
|
||||
|
||||
defaultPath, err := defaultDatabasePath(os.Geteuid())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
fromEnv := "from " + databaseEnv
|
||||
cases := []struct {
|
||||
env, path, source string
|
||||
}{
|
||||
{"", defaultPath, "default"},
|
||||
{"", defaultDatabasePath, "default"},
|
||||
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
|
||||
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
|
||||
fromEnv},
|
||||
@@ -97,48 +89,6 @@ func TestDatabasePath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultDatabasePath(t *testing.T) {
|
||||
// README §Location: root's database is /var/lib/sfdupes/db.sqlite.
|
||||
// Any other user's is db.sqlite in sfdupes under XDG_STATE_HOME, or
|
||||
// under ~/.local/state when XDG_STATE_HOME is unset or empty. For
|
||||
// that user, an XDG_STATE_HOME that is not an absolute path is an
|
||||
// error.
|
||||
if runtime.GOOS != goosLinux {
|
||||
t.Skip("macOS has another default state directory")
|
||||
}
|
||||
|
||||
t.Cleanup(xdg.Reload)
|
||||
t.Setenv("HOME", "/home/someone")
|
||||
|
||||
cases := []struct {
|
||||
euid int
|
||||
stateHome string
|
||||
want string
|
||||
wantErr error
|
||||
}{
|
||||
{0, "", rootDatabasePath, nil},
|
||||
{0, "/elsewhere/state", rootDatabasePath, nil},
|
||||
{0, "relative/state", rootDatabasePath, nil},
|
||||
{1000, "", "/home/someone/.local/state/sfdupes/db.sqlite", nil},
|
||||
{1000, "relative/state", "", errStateHomeNotAbsolute},
|
||||
{1000, "~/state", "", errStateHomeNotAbsolute},
|
||||
{1000, "/elsewhere/state", "/elsewhere/state/sfdupes/db.sqlite", nil},
|
||||
{1000, "/elsewhere/state/", "/elsewhere/state/sfdupes/db.sqlite", nil},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Setenv(stateHomeEnv, tc.stateHome)
|
||||
xdg.Reload()
|
||||
|
||||
got, err := defaultDatabasePath(tc.euid)
|
||||
if got != tc.want || !errors.Is(err, tc.wantErr) {
|
||||
t.Errorf("defaultDatabasePath(%d) with %s=%q = %q, %v; want %q, %v",
|
||||
tc.euid, stateHomeEnv, tc.stateHome, got, err, tc.want,
|
||||
tc.wantErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenScanDatabaseCreates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ func runDwim(
|
||||
return err
|
||||
}
|
||||
|
||||
err = runTrees(ctx, nil, stdout)
|
||||
err = runTrees(ctx, stdout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -121,9 +121,9 @@ func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
|
||||
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
||||
}
|
||||
|
||||
leftOut := "scan: left out " + excluded + " (--exclude)\n"
|
||||
if got := stderr(); !strings.Contains(got, leftOut) {
|
||||
t.Errorf("stderr = %q, want it to contain %q", got, leftOut)
|
||||
skipped := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if got := stderr(); !strings.Contains(got, skipped) {
|
||||
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -72,31 +72,14 @@ type walkRules struct {
|
||||
func defaultExcludes(goos string) []string {
|
||||
switch goos {
|
||||
case goosLinux:
|
||||
// The kernel's view of the running system, temporary files, and
|
||||
// caches that programs and package managers fill again on their
|
||||
// own: the system's, root's and those in each home in /home.
|
||||
return []string{
|
||||
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
|
||||
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
|
||||
"/root/go/pkg/mod",
|
||||
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
|
||||
"/home/*/go/pkg/mod",
|
||||
}
|
||||
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
|
||||
case goosDarwin:
|
||||
// A walk of / reaches everything in /System/Volumes/Data through
|
||||
// the firmlinks, so walking it as well would record each of
|
||||
// those files twice. /private/var/folders holds each user's
|
||||
// temporary files and caches. The walk reaches root's home,
|
||||
// /var/root, as /private/var/root.
|
||||
// those files twice.
|
||||
return []string{
|
||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
||||
"/private/var/folders", "/Library/Caches",
|
||||
"/private/var/root/Library/Caches", "/private/var/root/.cache",
|
||||
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
|
||||
"/private/var/root/go/pkg/mod",
|
||||
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
|
||||
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
|
||||
}
|
||||
case goosWindows:
|
||||
// Relative to the root of the drive an operand lies on. The
|
||||
@@ -203,7 +186,7 @@ func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: left out %s (--exclude)", path)
|
||||
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
||||
}
|
||||
|
||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||
@@ -233,7 +216,7 @@ func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: left out %s (%s, not a persistent filesystem)",
|
||||
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
|
||||
path, fsType)
|
||||
}
|
||||
|
||||
|
||||
@@ -24,22 +24,10 @@ func TestDefaultExcludes(t *testing.T) {
|
||||
// on any other system. The Windows list exists in the code only: the
|
||||
// binary does not build for Windows, so the README leaves it out.
|
||||
cases := map[string][]string{
|
||||
goosLinux: {
|
||||
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
|
||||
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
|
||||
"/root/go/pkg/mod",
|
||||
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
|
||||
"/home/*/go/pkg/mod",
|
||||
},
|
||||
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
|
||||
goosDarwin: {
|
||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
||||
"/private/var/folders", "/Library/Caches",
|
||||
"/private/var/root/Library/Caches", "/private/var/root/.cache",
|
||||
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
|
||||
"/private/var/root/go/pkg/mod",
|
||||
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
|
||||
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
|
||||
},
|
||||
goosWindows: {
|
||||
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
|
||||
@@ -75,29 +63,6 @@ func TestIsDefaultExclude(t *testing.T) {
|
||||
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// The Linux list matches the cache directories in /root and in each
|
||||
// home in /home, but nothing below them and no .cache directory
|
||||
// elsewhere.
|
||||
linux := map[string]bool{
|
||||
"/home/u/.cache": true,
|
||||
"/home/u/go/pkg/mod": true,
|
||||
"/root/.cache": true,
|
||||
"/var/cache": true,
|
||||
"/home/u/.cache/pip": false,
|
||||
"/home/u/src/.cache": false,
|
||||
"/srv/home/u/.cache": false,
|
||||
"/home/u/.cargo/bin": false,
|
||||
"/home/u/go/pkg/mod/cache": false,
|
||||
}
|
||||
|
||||
for path, want := range linux {
|
||||
got := isDefaultExclude(path, defaultExcludes(goosLinux))
|
||||
if got != want {
|
||||
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
|
||||
got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadMountInfo(t *testing.T) {
|
||||
@@ -242,7 +207,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
||||
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
|
||||
}
|
||||
|
||||
line := "\nscan: left out " + excluded + " (default exclude)\n"
|
||||
line := "\nscan: skipped " + excluded + " (default exclude)\n"
|
||||
if !strings.Contains(stderr(), line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
||||
}
|
||||
@@ -311,7 +276,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
||||
|
||||
recordByPath(t, dbRecords(t, db), f)
|
||||
|
||||
if strings.Contains(stderr(), "scan: left out") {
|
||||
if strings.Contains(stderr(), "scan: skipped") {
|
||||
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
|
||||
}
|
||||
}
|
||||
@@ -325,15 +290,15 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
|
||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
||||
t.Skip(dir + " is not inside a default exclude")
|
||||
}
|
||||
|
||||
f := writeFile(t, dir, "sub/f", pattern(1, 10))
|
||||
|
||||
stderr := scanOK(t, dir)
|
||||
if strings.Contains(stderr, "scan: left out") {
|
||||
t.Errorf("stderr = %q, want no line saying a path was left out",
|
||||
if strings.Contains(stderr, "scan: skipped") {
|
||||
t.Errorf("stderr = %q, want no line saying a path was skipped",
|
||||
stderr)
|
||||
}
|
||||
|
||||
@@ -377,7 +342,7 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
line := "scan: left out /var/tmp (default exclude)\n"
|
||||
line := "scan: skipped /var/tmp (default exclude)\n"
|
||||
|
||||
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
|
||||
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
|
||||
@@ -418,7 +383,7 @@ func TestRunScanExcludeNested(t *testing.T) {
|
||||
|
||||
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||
|
||||
line := "scan: left out " + excluded + " (--exclude)\n"
|
||||
line := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
@@ -480,8 +445,8 @@ func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
|
||||
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
|
||||
excludeFlag, dotDot, dir)
|
||||
|
||||
for _, leftOut := range []string{filepath.Dir(inDir), file, hardLink} {
|
||||
line := "scan: left out " + leftOut + " (--exclude)\n"
|
||||
for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
|
||||
line := "scan: skipped " + skipped + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
@@ -615,7 +580,7 @@ func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
|
||||
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
line := "scan: left out " + shm + " (" + fsType +
|
||||
line := "scan: skipped " + shm + " (" + fsType +
|
||||
", not a persistent filesystem)\n"
|
||||
summary := "scan: 0 files seen"
|
||||
|
||||
@@ -669,7 +634,7 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
|
||||
t.Errorf("%s recorded", unseen)
|
||||
}
|
||||
|
||||
line := "scan: left out " + shm + " (" + fsType +
|
||||
line := "scan: skipped " + shm + " (" + fsType +
|
||||
", not a persistent filesystem)\n"
|
||||
if !strings.Contains(stderr(), line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
||||
@@ -702,7 +667,7 @@ func TestWalkLeavesOutNonPersistent(t *testing.T) {
|
||||
|
||||
leftOut = append(leftOut, ev.leftOut)
|
||||
|
||||
want := "scan: left out " + ev.leftOut + " (" + fsType +
|
||||
want := "scan: skipped " + ev.leftOut + " (" + fsType +
|
||||
", not a persistent filesystem)"
|
||||
if ev.warn != want {
|
||||
t.Errorf("line = %q, want %q", ev.warn, want)
|
||||
@@ -746,20 +711,6 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
|
||||
return dir, fsType, walkRules{nonPersistent: nonPersistent}
|
||||
}
|
||||
|
||||
// insideDefaultExclude reports whether path, which is absolute and
|
||||
// clean, or a directory above it is one of excludes.
|
||||
func insideDefaultExclude(path string, excludes []string) bool {
|
||||
for dir := path; ; dir = filepath.Dir(dir) {
|
||||
if isDefaultExclude(dir, excludes) {
|
||||
return true
|
||||
}
|
||||
|
||||
if dir == "/" {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// assertRecorded checks that the database at dbPath holds a record for
|
||||
// path when want is set, and none when it is not.
|
||||
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
|
||||
|
||||
+14
-70
@@ -9,7 +9,6 @@ import (
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"syscall"
|
||||
@@ -45,12 +44,10 @@ func reportPhase(name string) string {
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout, from the records at or below the
|
||||
// operands in args, or from every record when args is empty. Of the
|
||||
// scanned filesystem it only checks, with pathExists, that each path in
|
||||
// a duplicate group is still there; it never looks up an operand. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
// duplicates report as TSV on stdout. Of the scanned filesystem it
|
||||
// only checks, with pathExists, that each path in a duplicate group is
|
||||
// still there. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (read the number of
|
||||
@@ -58,12 +55,7 @@ func reportPhase(name string) string {
|
||||
// order, writing each size's duplicate groups to stdout once that size
|
||||
// is read; see readReport). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
operands, err := cleanOperands(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -95,8 +87,7 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists,
|
||||
prog)
|
||||
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -117,14 +108,12 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. Only the records that underOperand finds under operands join a
|
||||
// group, and a path for which exists returns false is left out of its
|
||||
// group. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
// read. A path for which exists returns false is left out of its group.
|
||||
// stdout may be the terminal prog is drawn on, so the lines go through
|
||||
// prog's writer. It returns what it wrote, with the totals for the
|
||||
// summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
operands []string, stdout io.Writer, exists func(path string) bool,
|
||||
prog *progress,
|
||||
stdout io.Writer, exists func(path string) bool, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out, exists: exists}
|
||||
@@ -139,12 +128,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||
prog.increment()
|
||||
|
||||
report.records++
|
||||
|
||||
if !underOperand(r.path, operands) {
|
||||
return nil
|
||||
}
|
||||
|
||||
writeErr = report.add(r)
|
||||
|
||||
return writeErr
|
||||
@@ -176,7 +159,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
// records of one size, so it holds only the duplicate groups of the
|
||||
// size being read, and writes them, ordered by first path, once a
|
||||
// record of a smaller size comes. It counts what the summary line
|
||||
// reports; readReport counts the records read.
|
||||
// reports.
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
@@ -196,6 +179,8 @@ type reportWriter struct {
|
||||
// its paths; any other ends them, and one of a smaller size first
|
||||
// writes the groups of the size before.
|
||||
func (w *reportWriter) add(r scanRec) error {
|
||||
w.records++
|
||||
|
||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||
w.paths = append(w.paths, r.path)
|
||||
@@ -288,47 +273,6 @@ func pathExists(path string) bool {
|
||||
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
||||
}
|
||||
|
||||
// cleanOperands returns the operands of report or trees, each made
|
||||
// absolute by absolutePath and then cleaned. The records hold real
|
||||
// paths, with no "." or ".." element and no repeated or trailing
|
||||
// slash, so an operand is cleaned to match them. No operand is looked
|
||||
// up on the filesystem. None is empty: rejectEmptyOperand refuses an
|
||||
// empty operand on the command line.
|
||||
func cleanOperands(args []string) ([]string, error) {
|
||||
operands := make([]string, 0, len(args))
|
||||
|
||||
for _, a := range args {
|
||||
abs, err := absolutePath(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
|
||||
}
|
||||
|
||||
operands = append(operands, filepath.Clean(abs))
|
||||
}
|
||||
|
||||
return operands, nil
|
||||
}
|
||||
|
||||
// underOperand reports whether path is one of operands or lies below
|
||||
// one of them. Each operand is clean and absolute, as cleanOperands
|
||||
// returns it. With no operands, a report covers the whole database, so
|
||||
// underOperand is true for every path.
|
||||
func underOperand(path string, operands []string) bool {
|
||||
if len(operands) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
for _, o := range operands {
|
||||
// Only the root directory's clean path ends in "/".
|
||||
dir := strings.TrimSuffix(o, "/") + "/"
|
||||
if path == o || strings.HasPrefix(path, dir) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
|
||||
@@ -102,8 +102,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := readReport(t.Context(), db, "", nil, &out, everyPathExists,
|
||||
nil)
|
||||
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -149,136 +148,6 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
||||
return dupeGroups(t, db)
|
||||
}
|
||||
|
||||
// TestRunReportOperands runs report with operands on three copies of
|
||||
// one file, two of them under srv/photos, and a fourth copy in
|
||||
// srv/photos2, all in a new directory. Only the records at or below the
|
||||
// operands form the groups and the totals, however each operand is
|
||||
// written.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestRunReportOperands(t *testing.T) {
|
||||
h := testHash("h")
|
||||
|
||||
recs := make([]scanRec, 0, 4)
|
||||
for _, p := range []string{
|
||||
"/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z",
|
||||
} {
|
||||
recs = append(recs,
|
||||
scanRec{size: 7, head: h, tail: h, content: h, path: p})
|
||||
}
|
||||
|
||||
dir, recs := recsOnDisk(t, recs)
|
||||
a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
header := "first\tdupe\tsize\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t7\n"
|
||||
}
|
||||
photos := header + row(x, y)
|
||||
|
||||
cases := map[string]struct {
|
||||
operands []string
|
||||
want string
|
||||
summary string
|
||||
}{
|
||||
"directory": {[]string{dir + "/srv/photos"}, photos,
|
||||
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
|
||||
"trailing slash": {[]string{dir + "/srv/photos/"}, photos, ""},
|
||||
"dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""},
|
||||
"files": {[]string{a, z}, header + row(a, z), ""},
|
||||
"overlapping": {[]string{dir + "/srv", dir + "/srv/photos"},
|
||||
header + row(x, y) + row(x, z), ""},
|
||||
"root": {[]string{"/"},
|
||||
header + row(a, x) + row(a, y) + row(a, z), ""},
|
||||
"no record": {[]string{dir + "/srv/photo"}, header,
|
||||
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdReport}, tc.operands...),
|
||||
&stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
|
||||
tc.operands, code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
|
||||
// The records read are all four, whatever the operands.
|
||||
summary := "report: 4 records read, " + tc.summary + "\n"
|
||||
if got := stderr(); tc.summary != "" &&
|
||||
!strings.HasSuffix(got, summary) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, summary)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
//nolint:paralleltest // t.Chdir changes the process-wide working directory
|
||||
func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Chdir(dir)
|
||||
|
||||
wd, err := absolutePath(".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("cleanOperands = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunReportsRejectEmptyOperand checks that an empty PATH operand of
|
||||
// report or trees, alone or after another operand, is a usage error
|
||||
// reported in one line before the database is opened (README "Error
|
||||
// handling and exit codes").
|
||||
func TestRunReportsRejectEmptyOperand(t *testing.T) {
|
||||
cases := map[string][]string{
|
||||
"report": {cmdReport, ""},
|
||||
"trees": {cmdTrees, ""},
|
||||
"after an operand": {cmdTrees, t.TempDir(), ""},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
// No database exists there, so opening one would be a fatal
|
||||
// error, exit 1.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitUsage {
|
||||
t.Errorf("run(%q) = %d, want %d", args, code, exitUsage)
|
||||
}
|
||||
|
||||
want := "Error: PATH must not be empty\n"
|
||||
if stdout.String() != "" || stderr.String() != want {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
|
||||
stdout.String(), stderr.String(), want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportEscapesPaths(t *testing.T) {
|
||||
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
@@ -326,7 +195,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{now: time.Now}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, failingWriter{},
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{},
|
||||
everyPathExists, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
@@ -423,8 +292,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, os.Stderr,
|
||||
everyPathExists, p)
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -646,7 +514,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
||||
return true
|
||||
}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil)
|
||||
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -680,8 +548,7 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
||||
|
||||
exists := func(path string) bool { return path != "/a/gone" }
|
||||
|
||||
report, err := readReport(t.Context(), db, "", nil, io.Discard, exists,
|
||||
nil)
|
||||
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -714,7 +581,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
report, err := readReport(t.Context(), db, "", nil, &out, exists, nil)
|
||||
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
+12
-28
@@ -664,8 +664,8 @@ func isDatabaseFile(f fileRec, rules walkRules) bool {
|
||||
// inode run, committing completed records to the database in batches
|
||||
// as results arrive, so a long scan persists its progress as it goes
|
||||
// (an interrupted scan resumes cheaply: the next run skips everything
|
||||
// already recorded). Each path of a run that fails to hash is warned
|
||||
// about and counted as skipped; stale records for them, if any, are deleted
|
||||
// already recorded). A run that fails to hash is warned about and
|
||||
// counted as skipped; stale records for its paths, if any, are deleted
|
||||
// by the update phase. A run whose file no longer exists gets no record
|
||||
// and no warning, and is not counted as skipped. The total counts actual
|
||||
// reads, so the display's time remaining is meaningful.
|
||||
@@ -706,12 +706,9 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
|
||||
}
|
||||
|
||||
if r.err != nil {
|
||||
// Every path in the run is counted, so every path gets a line.
|
||||
for _, rec := range r.run {
|
||||
s.st.skipped++
|
||||
s.st.skipped += len(r.run)
|
||||
|
||||
prog.warnf("%s", skippedLine(rec.path, r.err))
|
||||
}
|
||||
prog.warnf("hash %s: %v", r.run[0].path, r.err)
|
||||
|
||||
continue
|
||||
}
|
||||
@@ -918,7 +915,8 @@ func operandWarning(root string, fi fs.FileInfo) string {
|
||||
// The walk never enters a .zfs directory, so it starts from nothing
|
||||
// inside one either.
|
||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
||||
return fmt.Sprintf("scan: skipped %s (inside a .zfs directory)", root)
|
||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
||||
root)
|
||||
}
|
||||
|
||||
var kind string
|
||||
@@ -944,21 +942,7 @@ func operandWarning(root string, fi fs.FileInfo) string {
|
||||
kind = "non-regular file"
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: skipped %s (a %s)", root, kind)
|
||||
}
|
||||
|
||||
// skippedLine returns the warning for a path the scan skips because of
|
||||
// err. The reason is the error without its path, which the line already
|
||||
// names.
|
||||
func skippedLine(path string, err error) string {
|
||||
reason := err.Error()
|
||||
|
||||
var pathErr *fs.PathError
|
||||
if errors.As(err, &pathErr) {
|
||||
reason = pathErr.Err.Error()
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: skipped %s (%s)", path, reason)
|
||||
return fmt.Sprintf("walk %s: skipping %s operand", root, kind)
|
||||
}
|
||||
|
||||
// seedRoot turns one PATH operand into the walk's starting state: a
|
||||
@@ -981,7 +965,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
||||
|
||||
if err != nil {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
warn: skippedLine(root, err),
|
||||
warn: fmt.Sprintf("walk %s: %v", root, err),
|
||||
fail: true,
|
||||
})
|
||||
|
||||
@@ -1110,7 +1094,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
||||
|
||||
if err != nil {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
warn: skippedLine(job.path, err),
|
||||
warn: fmt.Sprintf("walk %s: %v", job.path, err),
|
||||
fail: true,
|
||||
})
|
||||
|
||||
@@ -1169,7 +1153,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
|
||||
if err != nil {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
warn: skippedLine(p, err),
|
||||
warn: fmt.Sprintf("stat %s: %v", p, err),
|
||||
fail: true,
|
||||
})
|
||||
|
||||
@@ -1230,7 +1214,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
|
||||
if err != nil {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
warn: skippedLine(p, err),
|
||||
warn: fmt.Sprintf("walk %s: %v", p, err),
|
||||
fail: true,
|
||||
})
|
||||
|
||||
@@ -1251,7 +1235,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
leftOut: p,
|
||||
warn: fmt.Sprintf("scan: left out %s (default exclude)", p),
|
||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
||||
})
|
||||
|
||||
return dirJob{}, false
|
||||
|
||||
@@ -752,9 +752,8 @@ func TestWalkPassesOverRemovedPaths(t *testing.T) {
|
||||
|
||||
// TestScanWarnsAboutUnreadablePaths checks that a permission error is
|
||||
// still warned about and counted as skipped, unlike a removed path. The
|
||||
// walk reaches an unreadable directory. It also reaches a file and a
|
||||
// subdirectory in a directory that can be read but not searched. Another
|
||||
// such file is an operand.
|
||||
// walk reaches an unreadable directory, and a file in a directory that
|
||||
// can be read but not searched. Another such file is an operand.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
||||
@@ -766,7 +765,6 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
tree := filepath.Join(dir, "tree")
|
||||
walked := writeFile(t, tree, "unsearchable/f", []byte("x"))
|
||||
subdir := filepath.Join(filepath.Dir(walked), "d")
|
||||
operand := writeFile(t, dir, "unsearchable/f", []byte("y"))
|
||||
unreadable := filepath.Join(tree, "unreadable")
|
||||
|
||||
@@ -775,11 +773,6 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
err = os.Mkdir(subdir, 0o750)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
modes := map[string]fs.FileMode{
|
||||
unreadable: 0,
|
||||
filepath.Dir(walked): 0o400,
|
||||
@@ -800,16 +793,15 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
||||
}
|
||||
|
||||
st := syncTree(t, openTestDB(t), tree, operand)
|
||||
if st != (scanStats{skipped: 4}) {
|
||||
t.Errorf("stats = %+v, want 4 skipped and nothing else", st)
|
||||
if st != (scanStats{skipped: 3}) {
|
||||
t.Errorf("stats = %+v, want 3 skipped and nothing else", st)
|
||||
}
|
||||
|
||||
got := stderr()
|
||||
for _, warning := range []string{
|
||||
"scan: skipped " + unreadable + " (permission denied)\n",
|
||||
"scan: skipped " + walked + " (permission denied)\n",
|
||||
"scan: skipped " + subdir + " (permission denied)\n",
|
||||
"scan: skipped " + operand + " (permission denied)\n",
|
||||
"walk " + unreadable + ": open " + unreadable + ": permission denied\n",
|
||||
"stat " + walked + ": lstat " + walked + ": permission denied\n",
|
||||
"walk " + operand + ": lstat " + operand + ": permission denied\n",
|
||||
} {
|
||||
if !strings.Contains(got, warning) {
|
||||
t.Errorf("stderr = %q, want %q", got, warning)
|
||||
@@ -1949,19 +1941,14 @@ func TestScanHardlinksReadOnce(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestScanHardlinkRunFailsTogether makes a file with two hard links
|
||||
// unreadable. Both paths are counted as skipped, and each gets its own
|
||||
// warning.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestScanHardlinkRunFailsTogether(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
a := writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||
b := filepath.Join(dir, "b.bin")
|
||||
|
||||
err := os.Link(a, b)
|
||||
err := os.Link(a, filepath.Join(dir, "b.bin"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -1977,15 +1964,6 @@ func TestScanHardlinkRunFailsTogether(t *testing.T) {
|
||||
if st.skipped != 2 || st.added != 0 {
|
||||
t.Fatalf("stats = %+v, want both hardlink paths skipped", st)
|
||||
}
|
||||
|
||||
got := stderr()
|
||||
|
||||
for _, p := range []string{a, b} {
|
||||
warning := "scan: skipped " + p + " (permission denied)\n"
|
||||
if !strings.Contains(got, warning) {
|
||||
t.Errorf("stderr = %q, want %q", got, warning)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestHashPhaseSkipsFileReplacedByFIFO replaces a walked regular file
|
||||
@@ -2023,7 +2001,7 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
|
||||
s.st.skipped, len(s.batch))
|
||||
}
|
||||
|
||||
warning := "scan: skipped " + p + " (not a regular file)\n"
|
||||
warning := "hash " + p + ": not a regular file\n"
|
||||
if got := stderr(); strings.Count(got, warning) != 1 {
|
||||
t.Errorf("stderr = %q, want %q once", got, warning)
|
||||
}
|
||||
|
||||
+26
-87
@@ -37,14 +37,12 @@ func treesPhase(name string) string {
|
||||
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the paths of the records at or below the operands in args (of every
|
||||
// record when args is empty), computes a Merkle-style digest per
|
||||
// directory, and prints maximal duplicate-tree groups as TSV on stdout.
|
||||
// Of the scanned filesystem it only checks, with pathExists, that the
|
||||
// root of each duplicate tree is still there, never the files below it;
|
||||
// it never looks up an operand. It names the database on stderr before
|
||||
// it opens it. Any database problem, including a missing database, is
|
||||
// fatal.
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
|
||||
// filesystem it only checks, with pathExists, that the root of each
|
||||
// duplicate tree is still there, never the files below it. It names the
|
||||
// database on stderr before it opens it. Any database problem,
|
||||
// including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in four phases, each under a progress display of its
|
||||
// own on stderr: open (open the database), read (read the number of
|
||||
@@ -52,12 +50,7 @@ func treesPhase(name string) string {
|
||||
// hierarchy), group (group the directories by digest) and write (write
|
||||
// the groups to stdout). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
operands, err := cleanOperands(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -86,7 +79,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, operands, prog)
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -108,7 +101,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||
stalledOn: "out",
|
||||
})
|
||||
dupeTrees, err := writeTreeGroups(stdout, dupes, prog)
|
||||
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -119,23 +112,16 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
||||
"%s reclaimable\n",
|
||||
records, len(dupes), dupeTrees, humanBytes(reclaimableBytes(dupes)))
|
||||
records, len(dupes), dupeTrees, humanBytes(reclaimable))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readTree gives prog the number of records in db as its total, then
|
||||
// reads every record in path order, counting each on prog. It adds the
|
||||
// records that underOperand finds under operands to the directory
|
||||
// hierarchy. It returns the number of records read, the super-root and
|
||||
// every directory at or below one of operands.
|
||||
//
|
||||
// Any other directory lies above an operand and can hold files that are
|
||||
// not used, so its digest does not describe it. It is left out of the
|
||||
// directories returned. A directory whose parent is left out gets the
|
||||
// super-root as its parent, so suppressed never judges its group by
|
||||
// that parent's digest.
|
||||
func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
// reads every record in path order into the directory hierarchy,
|
||||
// counting each on prog. It returns the number of records read, the
|
||||
// super-root and every directory.
|
||||
func readTree(ctx context.Context, db *sql.DB,
|
||||
prog *progress,
|
||||
) (int, *treeNode, []*treeNode, error) {
|
||||
records := 0
|
||||
@@ -145,10 +131,7 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
records++
|
||||
|
||||
prog.increment()
|
||||
|
||||
if underOperand(r.path, operands) {
|
||||
tree.add(r)
|
||||
}
|
||||
tree.add(r)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
@@ -156,40 +139,29 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
|
||||
super, allDirs := tree.finish()
|
||||
|
||||
dirs := allDirs[:0]
|
||||
|
||||
for _, d := range allDirs {
|
||||
if !underOperand(d.path, operands) {
|
||||
continue
|
||||
}
|
||||
|
||||
if !underOperand(d.parent.path, operands) {
|
||||
d.parent = super
|
||||
}
|
||||
|
||||
dirs = append(dirs, d)
|
||||
}
|
||||
|
||||
return records, super, dirs, nil
|
||||
return records, super, allDirs, nil
|
||||
}
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
// a row for every member of each group but its first, counting each
|
||||
// group on prog once its rows are written. The disk_size column is the
|
||||
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
||||
// go through prog's writer. It returns the number of rows (dupe trees).
|
||||
// go through prog's writer. It returns the number of rows (dupe trees)
|
||||
// and the bytes they could reclaim.
|
||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
prog *progress,
|
||||
) (int, error) {
|
||||
) (int, int64, error) {
|
||||
out := prog.writer(stdout)
|
||||
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
dupeTrees := 0
|
||||
|
||||
var reclaimable int64
|
||||
|
||||
for _, g := range dupes {
|
||||
first := g[0]
|
||||
for _, n := range g[1:] {
|
||||
@@ -197,10 +169,11 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
escapePath(first.path), escapePath(n.path),
|
||||
first.fileCount, first.totalSize, n.diskSize)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
dupeTrees++
|
||||
reclaimable += first.totalSize
|
||||
}
|
||||
|
||||
prog.increment()
|
||||
@@ -208,44 +181,10 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
|
||||
err = out.Flush()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
return dupeTrees, nil
|
||||
}
|
||||
|
||||
// reclaimableBytes returns the sum of the size column over the rows
|
||||
// that writeTreeGroups writes for dupes, leaving out a row whose dupe
|
||||
// lies inside another row's dupe.
|
||||
func reclaimableBytes(dupes [][]*treeNode) int64 {
|
||||
isDupe := make(map[*treeNode]bool)
|
||||
|
||||
for _, g := range dupes {
|
||||
for _, n := range g[1:] {
|
||||
isDupe[n] = true
|
||||
}
|
||||
}
|
||||
|
||||
var total int64
|
||||
|
||||
for n := range isDupe {
|
||||
if !insideDupe(n, isDupe) {
|
||||
total += n.totalSize
|
||||
}
|
||||
}
|
||||
|
||||
return total
|
||||
}
|
||||
|
||||
// insideDupe reports whether a directory above n is in isDupe.
|
||||
func insideDupe(n *treeNode, isDupe map[*treeNode]bool) bool {
|
||||
for p := n.parent; p != nil; p = p.parent {
|
||||
if isDupe[p] {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
return dupeTrees, reclaimable, nil
|
||||
}
|
||||
|
||||
// treeBuilder reconstructs the directory hierarchy from records added
|
||||
|
||||
@@ -214,57 +214,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesOperands runs trees with operands on records in a new
|
||||
// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y
|
||||
// also hold files that differ. Given x/a and y/a, trees reads x and y
|
||||
// as holding one copy each of a, yet neither is reported: only the
|
||||
// directories at or below an operand are.
|
||||
func TestRunTreesOperands(t *testing.T) {
|
||||
f, g, k := testHash("f"), testHash("g"), testHash("k")
|
||||
dir, recs := recsOnDisk(t, []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
|
||||
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
|
||||
})
|
||||
b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a"
|
||||
|
||||
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t1\t10\t0\n"
|
||||
}
|
||||
|
||||
cases := map[string]struct {
|
||||
operands []string
|
||||
want string
|
||||
}{
|
||||
"none": {nil, header + row(b, xa) + row(b, ya)},
|
||||
"two copies": {[]string{xa, ya}, header + row(xa, ya)},
|
||||
"parents": {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)},
|
||||
"one copy": {[]string{b}, header},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdTrees}, tc.operands...),
|
||||
&stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
|
||||
tc.operands, code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
@@ -364,7 +313,7 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err := writeTreeGroups(os.Stderr, dupes, p)
|
||||
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -638,64 +587,3 @@ func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
|
||||
t.Errorf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReclaimableBytesCountsNestedDupeOnce checks that a dupe tree
|
||||
// inside another dupe tree adds nothing to the reclaimable bytes. /B/x
|
||||
// lies in /B, so the total adds the sizes of /B and /C/x, 1500 + 1000.
|
||||
func TestReclaimableBytesCountsNestedDupeOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
fh, ft, fc := testHash("fh"), testHash("ft"), testHash("fc")
|
||||
gh, gt, gc := testHash("gh"), testHash("gt"), testHash("gc")
|
||||
recs := []scanRec{
|
||||
{size: 1000, head: fh, tail: ft, content: fc, path: "/A/x/f"},
|
||||
{size: 500, head: gh, tail: gt, content: gc, path: "/A/y/g"},
|
||||
{size: 1000, head: fh, tail: ft, content: fc, path: "/B/x/f"},
|
||||
{size: 500, head: gh, tail: gt, content: gc, path: "/B/y/g"},
|
||||
{size: 1000, head: fh, tail: ft, content: fc, path: "/C/x/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
|
||||
wantGroups := [][]string{{"/A", "/B"}, {"/A/x", "/B/x", "/C/x"}}
|
||||
if got := groupPaths(dupes); !slices.EqualFunc(got, wantGroups,
|
||||
slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, wantGroups)
|
||||
}
|
||||
|
||||
if got := reclaimableBytes(dupes); got != 2500 {
|
||||
t.Errorf("reclaimableBytes = %d, want 2500", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReclaimableBytesCountsDupeInsideGrandparentDupeOnce checks that a
|
||||
// dupe tree adds nothing when a dupe tree above its parent holds it.
|
||||
// /B/m/x lies in /B, and /B/m is no dupe, since the group of /A and /B
|
||||
// implies that of /A/m and /B/m. The total adds the sizes of /B and
|
||||
// /C/x, 1300 + 1000.
|
||||
func TestReclaimableBytesCountsDupeInsideGrandparentDupeOnce(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f, w := testHash("f"), testHash("w")
|
||||
recs := []scanRec{
|
||||
{size: 1000, head: f, tail: f, content: f, path: "/A/m/x/f"},
|
||||
{size: 300, head: w, tail: w, content: w, path: "/A/m/w"},
|
||||
{size: 1000, head: f, tail: f, content: f, path: "/B/m/x/f"},
|
||||
{size: 300, head: w, tail: w, content: w, path: "/B/m/w"},
|
||||
{size: 1000, head: f, tail: f, content: f, path: "/C/x/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
|
||||
wantGroups := [][]string{{"/A", "/B"}, {"/A/m/x", "/B/m/x", "/C/x"}}
|
||||
if got := groupPaths(dupes); !slices.EqualFunc(got, wantGroups,
|
||||
slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, wantGroups)
|
||||
}
|
||||
|
||||
if got := reclaimableBytes(dupes); got != 2300 {
|
||||
t.Errorf("reclaimableBytes = %d, want 2300", got)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user