Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
1477c57024 |
No files matched your search
@@ -22,8 +22,9 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||
in one file. The database is kept between runs. A rescan reads only new and
|
||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||
system's job scheduler. `report`, `trees` and `info` work from the database and
|
||||
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
||||
first look at a new machine with one command.
|
||||
can run at any time. `report` and `trees` leave out a copy that was deleted
|
||||
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
|
||||
look at a new machine with one command.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -114,20 +115,15 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||
directory it cannot read, and about a new or changed file it cannot read. It
|
||||
skips the path and deletes the records at or below it (see "Rules for the
|
||||
walk").
|
||||
|
||||
An unmounted disk leaves its mount point empty, so `scan` deletes every record
|
||||
under it, and the next scan reads every file on the disk again. Test the mount
|
||||
point of each disk at or below `/srv` with its own `mountpoint -q`, joined with
|
||||
`&&`, and test `/srv` only when it is one. As the file `/etc/cron.d/sfdupes`:
|
||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
||||
|
||||
```
|
||||
30 3 * * * root if mountpoint -q /srv; then /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log; else echo "a disk is not mounted; scan of /srv skipped"; fi
|
||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||||
```
|
||||
|
||||
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
|
||||
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` directly
|
||||
before `/usr/local/bin/sfdupes`.
|
||||
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the
|
||||
command on the same line.
|
||||
- `scan` writes nothing to stdout. Its stderr, appended here to
|
||||
`/var/log/sfdupes.log`, holds:
|
||||
- one line naming the database;
|
||||
@@ -259,14 +255,15 @@ In order:
|
||||
- `report` prints the file-level duplicate report from the database.
|
||||
- `trees` prints the duplicate-tree report from the database.
|
||||
- `info` prints the database's details.
|
||||
- `dwim` scans `/`, then prints the duplicate-tree report.
|
||||
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
||||
report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
|
||||
sfdupes dwim
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -296,34 +293,22 @@ The reference below is also the program's specification.
|
||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
||||
it. The path names the file exactly, whatever characters it holds. A relative
|
||||
path is relative to the working directory.
|
||||
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
||||
while it lists the trees. It also sets how many files `scan` reads at once
|
||||
while it hashes them. The default is the number of CPUs. `N` must be at
|
||||
least 1. A smaller value is a usage error, reported in one line on stderr with
|
||||
exit 2 before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out
|
||||
of directories on other filesystems, as the same flag does for `du` and
|
||||
`rsync`. By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
||||
excludes. Those are the system and temporary directories the walk otherwise
|
||||
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
||||
below it. `PATH` may be a directory or a file. The flag may be given more than
|
||||
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||
database files (see "Rules for the walk"). So the walk leaves `PATH` out
|
||||
whatever path leads to it. That includes a hard link to a file given as
|
||||
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
|
||||
`--exclude ""`, is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned. A `PATH` that does not exist, or whose real path
|
||||
`scan` cannot find, is a fatal error. So is an operand at or below one (see
|
||||
"Operands").
|
||||
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
|
||||
lists the trees. It also sets how many files `scan` reads at once while it
|
||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
||||
smaller value is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
||||
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
||||
By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
||||
exits 0. It writes nothing to stderr.
|
||||
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
||||
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
|
||||
names the database. The help of `report`, `trees`, `info` and `dwim` says what
|
||||
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
|
||||
default excludes of the system they run on, and the filesystem types the walk
|
||||
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
|
||||
default excludes of the system it runs on, and the filesystem types the walk
|
||||
never records anything on.
|
||||
|
||||
### Duplicate detection
|
||||
@@ -377,10 +362,6 @@ default. An operand may be a directory or a regular file.
|
||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||
target does not exist. `scan` then stops before it walks anything, and exits
|
||||
with code 1.
|
||||
- An operand at or below an `--exclude` path is a fatal error too. The message
|
||||
names the operand and the `--exclude` path. `scan` compares the operand and
|
||||
each directory above it with the `--exclude` paths by device ID and inode
|
||||
number, not by name.
|
||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||
@@ -456,16 +437,15 @@ operands:
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
one of these reasons:
|
||||
- it is an `--exclude` path;
|
||||
- it is a default exclude;
|
||||
- it is not on a persistent filesystem;
|
||||
- with `-x`, it is a directory on another filesystem than its operand.
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may itself lie at or inside such a path, other
|
||||
than an `--exclude` path (see "Operands"). Its walk verifies the records
|
||||
below it, so they are deleted when their files are gone.
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
|
||||
@@ -564,17 +544,6 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- The walk never enters or records an `--exclude` path, nor anything below it.
|
||||
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||
|
||||
```
|
||||
scan: skipped /srv/cache (--exclude)
|
||||
```
|
||||
|
||||
The path is not counted as skipped. The scan keeps the records at and below
|
||||
it, as described under "What a scan does to the database".
|
||||
`--no-default-excludes` does not change this rule.
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
@@ -649,8 +618,7 @@ from a thread of its own and never blocks the workers.
|
||||
it repeats another or lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as an `--exclude` path, as a default exclude or as not on a
|
||||
persistent filesystem;
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
- the summary line;
|
||||
- a warning when a report still has the database open as the scan ends (see
|
||||
"Concurrent reads").
|
||||
@@ -862,25 +830,14 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
||||
### `dwim` mode
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments. Like `scan`,
|
||||
it should run as root, so that it can read every file.
|
||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||
its own. Like `scan`, it should run as root, so that it can read every file.
|
||||
|
||||
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
|
||||
|
||||
- `--workers N`;
|
||||
- `-x`, `--one-file-system`;
|
||||
- `--no-default-excludes`;
|
||||
- `--exclude PATH`, which may be given more than once.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
|
||||
walk skips the default excludes and crosses into other filesystems. The
|
||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||
what `sfdupes trees` runs. The largest trees come first.
|
||||
|
||||
```
|
||||
sfdupes dwim -x --exclude /srv/cache
|
||||
```
|
||||
|
||||
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||
`scan` prints, then what `trees` prints, then one last line:
|
||||
|
||||
@@ -1242,9 +1199,6 @@ showing the count it stopped at.
|
||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||
stopped by either during its scan. A fatal error is one of these:
|
||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||
- an `--exclude` path that does not exist, or whose real path `scan` cannot
|
||||
find;
|
||||
- an operand at or below an `--exclude` path;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
@@ -1264,7 +1218,6 @@ showing the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- an empty `--exclude` value, as in `--exclude ""`;
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
the tracker yet is filed as an issue first.
|
||||
- Branch from `next`.
|
||||
- Do the work, with tests, in small focused commits.
|
||||
- Delete Next Step from `TODO.md`, in the same commit as the work.
|
||||
- Record the step at the top of Completed Steps, in the same commit as the work.
|
||||
- Push the branch and open a PR against `next`. Its title ends with
|
||||
` (closes #N)`.
|
||||
- An independent review gates each merge to `next`. Every finding is addressed
|
||||
@@ -15,7 +15,8 @@
|
||||
|
||||
- Pre-1.0.
|
||||
- The tracker is authoritative for the pre-1.0 backlog. The open issues under
|
||||
the `1.0.0` milestone are what remains before the tag.
|
||||
the `1.0.0` milestone are what remains before the tag. This file records
|
||||
history and process, not the queue.
|
||||
|
||||
# Next Step
|
||||
|
||||
@@ -23,6 +24,757 @@
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
|
||||
here, because a copy drifts as soon as the tracker moves.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- `report` and `trees` leave out a copy that was deleted after the scan. Each
|
||||
checks only the copy itself with `lstat`: the file, or the directory at the
|
||||
top of a tree. A group left with one copy is not reported. A missing `scan`
|
||||
operand is still a fatal error (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/173)
|
||||
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
||||
`README.md` shows under "Subcommands" (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||
counts toward a tree's digest (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||
path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/157)
|
||||
|
||||
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
|
||||
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
|
||||
prints the report again (2026-11-06,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/154)
|
||||
|
||||
- `scan` resolves each operand to its real path, symlinks followed, before the
|
||||
walk. A symlink operand is scanned at its target instead of skipped. With
|
||||
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
||||
walks `sub` on its own instead of dropping it and deleting its records
|
||||
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
|
||||
|
||||
- `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS
|
||||
counts a new file's blocks only seconds after the write. The count can land in
|
||||
under a second. Both now say up to seconds after the write, as the comment on
|
||||
`diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/152)
|
||||
|
||||
- The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at
|
||||
random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks,
|
||||
and one ZFS commit can take a file through several sizes. The test that needs
|
||||
the size to change waits for it, for at most 30 seconds. Each test reads the
|
||||
size before and after its scan. The rewrite test rescans until the two reads
|
||||
match, then compares exactly. The other two fail only on a `disk_size` outside
|
||||
the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
|
||||
|
||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
||||
sentence. Every completed step is kept, with its date and its issue or branch.
|
||||
The policy-compliance checklist of 2026-07-22 is folded into its step
|
||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
||||
it. It leads with what the program is and how to use it. A reference section
|
||||
covers every setting and every subcommand. Development and testing come last.
|
||||
Every statement was checked against the code on `next`. Five comments in
|
||||
`internal/sfdupes/cli_test.go` cite "Settings", where their rules now live.
|
||||
One comment each in these files cites what the README now says:
|
||||
- `internal/sfdupes/scan_test.go`;
|
||||
- `internal/sfdupes/trees_test.go`;
|
||||
- `internal/sfdupes/exclude_test.go`.
|
||||
|
||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- Each record holds `disk_size`, the bytes the file takes on disk. That is its
|
||||
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
|
||||
makes. `trees` prints a `disk_size` column after `size`. It is the sum over
|
||||
the `dupe` copy's files. On a compressed or sparse file `disk_size` differs
|
||||
from `size`. A rescan updates the `disk_size` of an unchanged file whose size
|
||||
on disk changed, without reading the file (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||
|
||||
- The database keeps the number of records and the sum of their sizes in the one
|
||||
row of a `totals` table. Three triggers on `files` keep that row current.
|
||||
`info` reads that row. So do `scan`'s `load` phase and the `read` phases of
|
||||
`report` and `trees`, for their totals. No count reads every record any more.
|
||||
The triggers make each write to `files` slower. `scan` keeps SQLite's
|
||||
temporary data in memory. That wins back part of that time (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/146)
|
||||
|
||||
- an operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. `scan` warns about it, counts it as
|
||||
skipped and does not walk it (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/142)
|
||||
|
||||
- A file or directory removed while the scan runs is not an error, in the walk
|
||||
or in the hash phase. The scan prints no line for it and does not count it as
|
||||
skipped. When the first of several hard links to a file is gone, the hash
|
||||
phase reads the next one instead. On a terminal every progress display starts
|
||||
with the spinner, the hash phase's included. A phase with an exact total ends
|
||||
its line with its percent and a 10-column bar at the terminal's right edge.
|
||||
The bar no longer moves as the line's length changes. A line too wide for the
|
||||
terminal now loses its rate before its bar. The time remaining is labelled
|
||||
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
|
||||
|
||||
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
||||
bytes of a SHA-256. That is half the size of the hex text they held. The
|
||||
hashes stay raw from the hash phase through the database to `report` and
|
||||
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
||||
|
||||
- The hash phase gives three hashes to every file that the walk records. `head`
|
||||
and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
|
||||
SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
|
||||
100 MiB up. Gone are:
|
||||
- the 10 MiB and 50 MiB thresholds;
|
||||
- the size census, which counted the sizes of every walked file and of every
|
||||
record outside the operands. A file was hashed only when the census showed
|
||||
another file of its size;
|
||||
- the records without hashes;
|
||||
- the content phase.
|
||||
|
||||
A scan has four phases, and a rescan reads only new and changed files.
|
||||
`info` no longer counts the records with `head` and `tail` hashes, nor those
|
||||
with a content hash, since every record has all three (2026-10-25,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||
|
||||
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
|
||||
again from `sneak/prompts` `next` at `0653579`. Two steps now come before
|
||||
`ARG CACHEBUST`:
|
||||
- the module download in the `lint`, `test` and `builder` stages of the
|
||||
`Dockerfile`;
|
||||
- the package install in `builder`.
|
||||
|
||||
Docker keeps them until one of these changes:
|
||||
- the base image;
|
||||
- the install line;
|
||||
- `go.mod` or `go.sum`.
|
||||
|
||||
Every step that checks or compiles the code still comes after
|
||||
`ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache.
|
||||
README says the same (2026-10-18,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||
|
||||
- The release rule was set. Until 1.0, each tip of `main` that passes the build
|
||||
gets the next `v0.0.N` tag.
|
||||
`go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it.
|
||||
`go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||
the binary. That is the tag, or the pseudo-version of a `go install` build. It
|
||||
prints `dev` only when neither is known. README "Install" documented:
|
||||
- the rule;
|
||||
- what `@main` and `v0.0.1` give;
|
||||
- how to check an installed binary.
|
||||
|
||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116)
|
||||
|
||||
- `scan` creates the database's directory where the database path leads. The
|
||||
directory is the path with its last element cut off as text, so a `..` in it
|
||||
stays. A `..` after a symlinked directory therefore goes through the symlink,
|
||||
as it does when the file is opened. With
|
||||
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
|
||||
`/data/x`, it now creates `/data/new` and the database in it. It used to
|
||||
create `/srv/new` and then fail (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
||||
|
||||
- `scan` never records its own database. Once the database and its `.lock` file
|
||||
are open, it notes the device and inode of each of these:
|
||||
- the database file;
|
||||
- its `-wal` and `-shm` files;
|
||||
- the `.lock` file.
|
||||
|
||||
The walk passes over any file with the device and inode of one of them,
|
||||
however it reaches the file. It prints no line for such a file and counts
|
||||
nothing. A record an earlier scan left for one of them is deleted
|
||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
||||
|
||||
- `scan` never records or reads anything on a filesystem that is not persistent,
|
||||
such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or
|
||||
keeps in memory. `scan` matches them by device ID, however the walk or the
|
||||
content phase reaches a path. As the scan starts, it lists the mounted
|
||||
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
||||
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
||||
not enter the default excludes either. So `sfdupes scan /` skips the kernel's
|
||||
view of the running system and the temporary directories. On Linux those are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
```
|
||||
|
||||
On macOS they are:
|
||||
|
||||
```
|
||||
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
||||
/System/Volumes/VM, /System/Volumes/Data
|
||||
```
|
||||
|
||||
A Windows list exists in the code but is not used. An operand at or inside a
|
||||
default exclude is still walked. The scan prints a line for each path it
|
||||
leaves out, without counting it as skipped. The records there count as
|
||||
outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
|
||||
the default excludes only, and `scan --help` lists them (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||
|
||||
- A progress display's rate covers the items completed over the last 5 seconds.
|
||||
When stderr is not a terminal, it covers the items since the line before. It
|
||||
falls to 0 while nothing completes. A burst after a wait no longer reads as a
|
||||
steady rate of over 1,000 files a second. The time remaining still comes from
|
||||
the phase's average. Once no item has completed for 10 seconds, the line shows
|
||||
`stalled` and where the phase waits in place of the rate, as
|
||||
`stalled disk/db`. The places are:
|
||||
- `disk`;
|
||||
- `db`;
|
||||
- `out`, for stdout;
|
||||
- `cpu`.
|
||||
|
||||
A phase that may wait in two places names both. Each phase's word is set in
|
||||
the code that defines the phase. A line too wide for the terminal loses its
|
||||
bar first, then its rate or those words. Four parts stay whole:
|
||||
- the phase;
|
||||
- the progress;
|
||||
- the elapsed time;
|
||||
- the ETA, the time remaining.
|
||||
|
||||
The content phase had one display for its checks and one for its reads. Both
|
||||
lost those words on an 80-column terminal, at counts like those of README's
|
||||
example. That happened for most elapsed times and ETAs in minutes. Every
|
||||
other phase kept them whenever the elapsed time and the ETA were both in
|
||||
minutes. The content phase writes its last partial batch, the records not
|
||||
yet written, under a display of its own (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
|
||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value:
|
||||
- the path and where it came from;
|
||||
- the sizes of the database file and of its `-wal` file, when there is one;
|
||||
- when the database file was last changed;
|
||||
- the schema version;
|
||||
- the number of records;
|
||||
- how many records have `head` and `tail` hashes;
|
||||
- how many records have a content hash;
|
||||
- the sizes of all records added up.
|
||||
|
||||
It opens the database as `report` does and shows two phases on stderr,
|
||||
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
|
||||
one read-only transaction, a read of the database at one moment. The counts
|
||||
come from one query (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||
|
||||
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
||||
`[2/2] read`, with the progress display `scan` and `trees` use. The display
|
||||
runs from the moment `report` starts until the summary line. `open` says it is
|
||||
opening the database, also while it waits on a busy one. `read` counts the
|
||||
records, then reads them against that count in signature order. SQLite takes
|
||||
that order from the `files_signature` index without sorting, so the first
|
||||
record comes at once. `report` forms the duplicate groups itself and holds
|
||||
only those of the size it is reading. It writes them ordered by first path
|
||||
before it reads the next size. stdout and the summary line are unchanged, and
|
||||
SQLite no longer needs a temporary file to sort in (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
||||
|
||||
- `trees` shows its work on stderr in four phases, from `[1/4] open` to
|
||||
`[4/4] write`, each with the progress display `scan` uses. The display runs
|
||||
from the moment `trees` starts until the summary line. `open` says it is
|
||||
opening the database, also while it waits on a busy one. `read` counts the
|
||||
records, then reads them against that count. `group` counts the directories it
|
||||
groups, and `write` the groups it writes. A display's rate names what its
|
||||
phase counts:
|
||||
- files;
|
||||
- records;
|
||||
- directories;
|
||||
- groups.
|
||||
|
||||
stdout and the summary line are unchanged. When stdout and stderr are the
|
||||
same file, as one terminal is, no line of stdout shares a line with the
|
||||
display. When they are not, the display keeps being drawn while stdout waits
|
||||
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||
|
||||
- Each `scan` phase's progress display names the phase and its place among the
|
||||
five, as `[2/5] walk`. It runs from the moment that the phase starts. It
|
||||
shows:
|
||||
- the progress;
|
||||
- the elapsed time;
|
||||
- the ETA, the time remaining.
|
||||
|
||||
`load` counts the records first. The walk's total is estimated from the
|
||||
records under the operands. The content phase takes its total from SQLite's
|
||||
grouping alone. Until the first record is returned, it says it is searching
|
||||
the database. On a terminal a phase with an exact total draws its line
|
||||
followed by the bar of the progress-bar library. Any other phase draws a
|
||||
spinner followed by the line. Every display is drawn again several times a
|
||||
second. No line reaches the terminal's last column, however narrow the
|
||||
terminal is, even after it is made narrower during a phase. A line too wide
|
||||
leaves out its rate, then its bar. The rest stays whole. The line is cut
|
||||
short only when the rest alone does not fit, as below about 70 columns late
|
||||
in a large scan. When stderr is not a terminal, the line is printed every 5
|
||||
seconds whether or not an item completed (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
|
||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||
they use, as an absolute path. The line says whether that path is the default
|
||||
or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README
|
||||
"Database" says when `scan` writes each file (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
||||
|
||||
- The canonical files and model scripts were copied again from `sneak/prompts`
|
||||
`next` at `e3f79c4`:
|
||||
- `golangci-lint` v2.14.0;
|
||||
- lint and test are phases of the `Dockerfile` that write no image, and
|
||||
`make test` runs the suite under the race detector. Gone are
|
||||
`Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`;
|
||||
- every `docker build` in `script/` passes a new `CACHEBUST` value. Every
|
||||
stage of the `Dockerfile` declares it first, so every `RUN` step runs on
|
||||
every build. The steps that compile keep Go's build cache, and the tests
|
||||
run with `-count=1`;
|
||||
- `prettier` runs on the host, from the `node` and `yarn` that
|
||||
`script/bootstrap` installs. The `prettier` and `markdown` stages are
|
||||
gone, and the build stage installs `git` and `make` itself;
|
||||
- a new push cancels the workflow's older run on the same branch, and a run
|
||||
stops after 20 minutes;
|
||||
- a coding agent's settings file is deleted.
|
||||
|
||||
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
||||
`ARG CACHEBUST`:
|
||||
- the module download in `lint`, `test` and the build stage;
|
||||
- the install of `git` and `make` in the build stage.
|
||||
|
||||
Docker keeps them between builds. Every step that checks or compiles the
|
||||
code still runs on every build (2026-10-15,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95)
|
||||
|
||||
- With `-x`, an operand under another operand is no longer dropped when a
|
||||
directory between the two, the inner one included, is on a different
|
||||
filesystem than the outer one. The inner operand is then walked on its own.
|
||||
For a regular-file operand the check starts at its directory. The records of
|
||||
files gone from below it are deleted (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
||||
|
||||
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
||||
under its operand, instead of deleting them. It counts their sizes in the size
|
||||
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
||||
|
||||
- The hash and content phases open each file without blocking and check that it
|
||||
is still a regular file. A file that is now a FIFO or device node is warned
|
||||
about and skipped instead of hanging the scan (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/100)
|
||||
|
||||
- The scan lock file is named from the database's path with symlinks resolved,
|
||||
after `scan` creates the database file if it is missing. A scan through a
|
||||
symlink to the database therefore fails while another scan holds the lock
|
||||
(2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101)
|
||||
|
||||
- On a terminal, a warning for the last file of a phase no longer erases the
|
||||
phase's finished bar. The bar is drawn again below the warning (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/102)
|
||||
|
||||
- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus
|
||||
`mtime_nsec`, and compares it at that resolution. A same-size rewrite within
|
||||
the same second is hashed again (2026-10-07,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/12)
|
||||
|
||||
- The narration was cut from `TODO.md` Completed Steps and from the comments in
|
||||
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now
|
||||
branches from and merges to `next` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/49)
|
||||
|
||||
- `make test-race` ran the test suite under the race detector in a cgo-enabled
|
||||
container, outside `make check`. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
|
||||
under the race detector, in the `Dockerfile`'s Debian-based `test` phase.
|
||||
`make test-race` is gone (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
||||
|
||||
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
||||
rather than serve the gates, the lint and test steps, from cache. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
|
||||
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
|
||||
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39)
|
||||
|
||||
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the
|
||||
Gitea workflow checks it. All Markdown was reformatted (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||
|
||||
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
||||
|
||||
- Tests cover:
|
||||
- a missing database;
|
||||
- `scan` keeping stdout empty;
|
||||
- its skip warning;
|
||||
- the `report` and `trees` summary lines;
|
||||
- every subcommand going through `runE`.
|
||||
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||
|
||||
- `.golangci.yml` was replaced with the current canonical copy, which uses
|
||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||
|
||||
- A test fails when either `hashWorker` cancellation check in
|
||||
`internal/sfdupes/scan.go` is removed (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||
|
||||
- A database path holding `?`, `#` or `%` opens exactly the file it names
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
||||
|
||||
- `scan` rejects `--workers` below 1 as a usage error instead of running
|
||||
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
|
||||
|
||||
- A test fails when either walk cancellation check in `internal/sfdupes/scan.go`
|
||||
is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||
|
||||
- A test checks that `scan` refuses a database with another schema version
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||
|
||||
- Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected,
|
||||
and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles`
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||
|
||||
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/17)
|
||||
|
||||
- `scan` creates the schema in one transaction. A version-0 database with a
|
||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||
|
||||
- README documents install, Docker, a daily `cron` scan and how to read and
|
||||
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||
|
||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
|
||||
nothing changes owner. The build stage only compiles, as root, and the tests
|
||||
run as `nobody` in the `test` phase (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/43)
|
||||
|
||||
- `--version` prints `sfdupes VERSION` to stdout. README documents it and
|
||||
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
|
||||
|
||||
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed,
|
||||
deletes nothing more, and exits 1 (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/5)
|
||||
|
||||
- `report` and `trees` stream the records instead of holding them all in memory.
|
||||
The schema gains the `files_signature` index (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
||||
|
||||
- Progress prints at once on a non-terminal and uses a real terminal test.
|
||||
Warnings print through a spinner instead of racing its redraw (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||
|
||||
- These operands are warned about and skipped, and the records beneath them are
|
||||
kept:
|
||||
- symlinks;
|
||||
- sockets;
|
||||
- FIFOs;
|
||||
- device nodes;
|
||||
- `.zfs` directories.
|
||||
|
||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9)
|
||||
|
||||
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
||||
second `scan` fails at once with exit 1 (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/53)
|
||||
|
||||
- Tests cover stdout write failures in `report` and `trees`. README states that
|
||||
`| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null`
|
||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
|
||||
|
||||
- `report` and `trees` open the database read-only, and `scan` leaves it out of
|
||||
WAL mode, so reading needs only read access (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
||||
|
||||
- Four characters are escaped in `report`, `trees` and warning paths:
|
||||
- tab;
|
||||
- newline;
|
||||
- carriage return;
|
||||
- backslash.
|
||||
|
||||
The root directory's path is `/` (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||
|
||||
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
||||
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
||||
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
||||
fails if the context carries `.git` and the version is still empty, `dev` or
|
||||
`unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`),
|
||||
so it stamps the same value as `make build` (2026-10-02, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||
|
||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||
size, and README "Duplicate detection" documented every size threshold. A file
|
||||
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
|
||||
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
|
||||
phase. The content phase ran after the update phase. It read a file for
|
||||
`content` only when its size, `head` and `tail` matched another record's, from
|
||||
this scan or an earlier one. `content` was the whole file below 50 MiB, and
|
||||
gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
|
||||
changed since its record was written. `report` and `trees` left out any record
|
||||
without a `content` hash. The `content` column is part of the version 1
|
||||
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
|
||||
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
|
||||
|
||||
- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
|
||||
`.dockerignore` (2026-09-21, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||
|
||||
- The lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||
`Dockerfile.lint` were fixed. Both pins became the policy
|
||||
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||
without the false `(Debian-based)` note or the tag. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
||||
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone,
|
||||
and so is `script/verify-lint-image-pin`, which compared the two pins
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
|
||||
|
||||
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
|
||||
owner ruling the linter is never installed on a host, and
|
||||
`golangci-lint config verify` runs before `golangci-lint run`.
|
||||
`script/bootstrap` stopped installing or pinning the linter, and
|
||||
`script/verify-linter-pin` was retired. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
|
||||
`Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the
|
||||
build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`.
|
||||
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
|
||||
an image build may run `docker`, so the phase calls `golangci-lint` directly
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
|
||||
|
||||
- The Docker build stage's prerequisites were installed by running
|
||||
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
||||
`script/verify-linter-pin` failed the build unless the linter copied from the
|
||||
lint stage was the version that `script/bootstrap` pinned. Builds then took up
|
||||
to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
|
||||
filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
||||
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
|
||||
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/42)
|
||||
|
||||
- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and
|
||||
`script/docker` cannot report a pass they did not earn. The `Dockerfile`
|
||||
copies the tree before its gates. On an unchanged tree Docker served them from
|
||||
cache, and the build exited 0 without running them. The fix was a
|
||||
`CHECK_EPOCH` build argument. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
|
||||
passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile`
|
||||
declares it before every step that checks or compiles the code. Run as root,
|
||||
the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through
|
||||
the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody`
|
||||
(2026-08-09, branch `cibuild-cache-bust`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/32)
|
||||
|
||||
- `script/bootstrap` checked the installed `golangci-lint` version instead of
|
||||
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
|
||||
`go install` module reference derived from it. Any installed version that was
|
||||
not the pin was reinstalled, whether:
|
||||
- older;
|
||||
- newer;
|
||||
- absent;
|
||||
- unparseable.
|
||||
|
||||
`go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the
|
||||
first `golangci-lint` on `PATH`. So bootstrap re-read the effective version
|
||||
after installing. On a mismatch it printed both paths and both versions and
|
||||
exited non-zero. It did not reorder `PATH` or delete anyone's binary. The
|
||||
`--version` call kept its stderr and was bounded by `timeout(1)` where that
|
||||
exists. `git`, `make` and `go` kept presence-only checks. The version check
|
||||
was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again
|
||||
to a no-op. Stub runs of the script under `dash` covered:
|
||||
- a thirteen-input version-parse matrix;
|
||||
- a shadowed install, which must exit non-zero;
|
||||
- an install destination not on `PATH`;
|
||||
- `GOBIN` set;
|
||||
- a wedged binary, which must hit the timeout.
|
||||
|
||||
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
||||
hiding findings on `main`. Superseded by
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer
|
||||
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||
|
||||
- The hash worker pool unwinds on the error path. The pool is an owned,
|
||||
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
||||
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
|
||||
defers `pool.stop()`, which cancels and then drains `results` until the last
|
||||
goroutine has exited. Draining is what frees a worker parked on a send. `ctx`
|
||||
is passed from `cmd.Context()` as the first parameter, through:
|
||||
- `runScan`;
|
||||
- `syncScan`;
|
||||
- both worker pools;
|
||||
- the whole database layer.
|
||||
|
||||
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
|
||||
walk. A cancelled walk yields a partial size census, and every file that it
|
||||
never reached looks vanished to the update phase. That phase's own `BeginTx`
|
||||
also fails on the cancelled context before deleting anything. The guard is
|
||||
still the barrier once an interrupted scan may commit what it has. Tests
|
||||
drive `run(scan)` against a database whose insert trigger aborts. They
|
||||
assert that the scan fails instead of hanging, and that the goroutine count
|
||||
returns to its pre-scan baseline. Other tests cancel a scan part-way through
|
||||
the walk, deterministically, by counting the scan's own checks of
|
||||
`ctx.Done()`. They assert that it stops at the guard with every record
|
||||
intact. It then holds a partial size census and a still-populated record
|
||||
index. Direct tests cover the remaining cancellation branches of both pools.
|
||||
They test:
|
||||
- `sendEvent`;
|
||||
- the walk workers;
|
||||
- `dispatchDirs`;
|
||||
- `feedHashJobs`;
|
||||
- `hashWorker`;
|
||||
- `hashPhase`.
|
||||
|
||||
(2026-08-09, branch `hash-pool-cleanup`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/6)
|
||||
|
||||
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
|
||||
are gone. So the deferred `db.Close()` runs when a subcommand fails, and with
|
||||
it the SQLite WAL checkpoint. These return errors instead:
|
||||
- `runScan`;
|
||||
- `runReport`;
|
||||
- `runTrees`;
|
||||
- `loadRecords`;
|
||||
- `resolveRoots`.
|
||||
|
||||
The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a
|
||||
`fatalError`, anything a subcommand returned, to exit 1, and cobra's own
|
||||
argument and flag errors to exit 2. So a runtime failure is never reported
|
||||
as a usage error, nor printed with the usage text.
|
||||
`internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the
|
||||
exit codes from README "Error handling" and the stdout/stderr split. It also
|
||||
checks that a fatal error after the database is open leaves no `-wal` or
|
||||
`-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch
|
||||
`db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
|
||||
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
|
||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
||||
replaced with the canonical file. Four thresholds now live under
|
||||
`linters.settings`, as the v2 schema requires, so they are applied:
|
||||
- `lll`;
|
||||
- `funlen`;
|
||||
- `cyclop`;
|
||||
- `dupl`.
|
||||
|
||||
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||
|
||||
- `Makefile` targets were converted to scripts-to-rule-them-all `script/`
|
||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
||||
- `bootstrap`, `setup` and `projectname`;
|
||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
||||
|
||||
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||
|
||||
- The binary is the default `make` target. Plain `make` builds `sfdupes`, where
|
||||
it used to run `check` plus `build`. `make build` remains as an alias
|
||||
(2026-07-24, branch `make-default-target`)
|
||||
|
||||
- The phases became scan-wide, with concurrent operands and batched updates. All
|
||||
operands seed the shared walk pool, and every pass runs once over the whole
|
||||
scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
|
||||
and update cycles and their stderr announcements are gone. The update pass
|
||||
writes in batches, each in its own transaction. The filesystem is
|
||||
authoritative and the database follows it, so one scan's changes need not land
|
||||
all at once (2026-07-24, branch `scan-wide-phases`)
|
||||
|
||||
- The stat pass was split back out of the walk. Phases are strictly sequential
|
||||
again, per operand, with parallelism only inside each phase:
|
||||
- walk;
|
||||
- stat;
|
||||
- hash;
|
||||
- update.
|
||||
|
||||
The walk enumerates paths with per-directory workers, and the stat pass
|
||||
lstats them with per-file workers. That restores the exact total and ETA of
|
||||
the stat bar (2026-07-24, branch `parallel-phases`)
|
||||
|
||||
- Each operand is announced on stderr before its passes. With per-operand walk,
|
||||
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
|
||||
totals that looked like the whole run's (2026-07-24, branch
|
||||
`scan-operand-progress`)
|
||||
|
||||
- The walk became parallel. The walk pass was a single goroutine and took hours
|
||||
at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
|
||||
million files. It is now a walk by a pool of per-directory workers that
|
||||
records size and mtime as it goes. That folds away the separate stat pass and
|
||||
halves the metadata reads. Each `PATH` operand commits in its own transaction,
|
||||
so an interrupted scan keeps completed operands (2026-07-24, branch
|
||||
`parallel-walk`)
|
||||
|
||||
- `scan` gained a persistent database. It maintains an SQLite database that
|
||||
survives between runs, with the absolute path as each record's key. The driver
|
||||
is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
|
||||
new or changed files, by mtime and size. It deletes the records of files
|
||||
vanished from under the scanned operands and leaves records outside them
|
||||
untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
|
||||
database, with no positional arguments, instead of a scan stream. The database
|
||||
is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
|
||||
journaling plus a single-transaction update keep a report run during a scan
|
||||
safe (2026-07-24, branch `persistent-database`)
|
||||
|
||||
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
|
||||
was tagged, and `main` and the tags were pushed (2026-07-23)
|
||||
|
||||
- The `scan` command line was reworked. Required `PATH...` operands, parsed by
|
||||
cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
|
||||
flag follows the GNU convention and stops at filesystem boundaries, which the
|
||||
walk crosses by default (2026-07-23, branch `scan-required-paths`)
|
||||
|
||||
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
||||
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
||||
2026-07-22 found these gaps, and the branch closed them:
|
||||
- `.gitignore`, with the standard patterns for secrets and for OS, editor
|
||||
and Go files. The compiled `sfdupes` binary and `files.dat` had been
|
||||
sitting untracked in the tree;
|
||||
- `.editorconfig`;
|
||||
- `LICENSE` and the README License section. MIT was assumed from house
|
||||
convention, for the owner to confirm;
|
||||
- `REPO_POLICIES.md` in the repo root;
|
||||
- the canonical `.golangci.yml`, with its 150 findings fixed, so that
|
||||
`make lint` is clean;
|
||||
- the `Makefile` targets the policy requires:
|
||||
- `test` and `lint`;
|
||||
- `fmt` and `fmt-check`;
|
||||
- `docker` and `hooks`.
|
||||
|
||||
`check` depended on `build`, which wrote the binary. `make check` must
|
||||
not modify files, so `check` no longer depends on `build`;
|
||||
|
||||
- a test suite, which the policy required with a 30-second timeout and the
|
||||
conditional `-v` rerun. At 64% coverage, it covers:
|
||||
- parsing and grouping;
|
||||
- digests and suppression;
|
||||
- hashing and the scan pipeline;
|
||||
- a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had:
|
||||
- hash-pinned images;
|
||||
- a lint stage that failed fast;
|
||||
- a build stage that ran `make check`;
|
||||
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
|
||||
checkout action pinned by commit SHA;
|
||||
- the README sections the policy requires. The first is a Description whose
|
||||
first line names the project, its purpose and category, its license and
|
||||
its author. Then come Getting Started and Rationale, and at the end TODO,
|
||||
License and Author;
|
||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||
- the pre-commit hook, installed with `make hooks` once that target existed.
|
||||
|
||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||
|
||||
- `git init` was run, with a README-only first commit. The code baseline was
|
||||
committed on `main` (2026-07-22)
|
||||
|
||||
- The `scan`, `report` and `trees` subcommands were implemented (pre-git
|
||||
history)
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Possible later features, out of scope per README, are a byte-for-byte check of
|
||||
|
||||
@@ -358,9 +358,8 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
dir := buildSmokeTree(t)
|
||||
|
||||
opts := scanOptions{workers: walkCancelWorkers}
|
||||
|
||||
err := runScan(t.Context(), []string{dir}, opts)
|
||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -383,7 +382,8 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
go func() {
|
||||
defer close(done)
|
||||
|
||||
err = runScan(cancelledContext(t), []string{dir}, opts)
|
||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
||||
false, false)
|
||||
}()
|
||||
|
||||
awaitReturn(t, done, "runScan")
|
||||
@@ -410,7 +410,8 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
got, before)
|
||||
}
|
||||
|
||||
err = runScan(t.Context(), []string{dir}, opts)
|
||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -436,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
||||
dir := buildWalkCancelTree(t)
|
||||
|
||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||
scanOptions{workers: walkCancelWorkers})
|
||||
walkCancelWorkers, false, false)
|
||||
if !errors.Is(err, errInterrupted) {
|
||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||
err, errInterrupted)
|
||||
|
||||
+37
-168
@@ -10,7 +10,6 @@ import (
|
||||
"os"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"slices"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
@@ -32,82 +31,15 @@ const (
|
||||
cmdDwim = "dwim"
|
||||
)
|
||||
|
||||
// databaseHelp is the paragraph of the root and scan help that names
|
||||
// the database (README "Location").
|
||||
const databaseHelp = "scan and dwim store what they find in a database " +
|
||||
"file. report, trees\nand info read it. The file is $" + databaseEnv +
|
||||
" when that is set and\nnot empty, and otherwise " +
|
||||
defaultDatabasePath + ", where creating it\nneeds root.\n"
|
||||
|
||||
// escapesHelp is the paragraph of the report and trees help on how
|
||||
// paths are escaped (README "Report output format").
|
||||
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
|
||||
|
||||
\\ a backslash
|
||||
\t a tab
|
||||
\n a newline
|
||||
\r a carriage return
|
||||
`
|
||||
|
||||
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
|
||||
// summary in the help of their subcommands.
|
||||
const (
|
||||
reportHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||
row per duplicate file. The columns are:
|
||||
|
||||
first the path that sorts first, byte by byte, in its group of copies
|
||||
dupe another copy of that file
|
||||
size the file's size in bytes
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp
|
||||
|
||||
treesHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||
row per duplicate directory tree. The columns are:
|
||||
|
||||
first the path that sorts first, byte by byte, in its group of copies
|
||||
dupe another copy of that tree
|
||||
files the number of regular files in one copy
|
||||
size their total size in bytes
|
||||
disk_size the bytes the dupe copy takes on disk
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp
|
||||
|
||||
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
|
||||
value, separated by a tab:
|
||||
|
||||
database the path, and where it came from
|
||||
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
|
||||
-wal file size in bytes, then rounded; only if a -wal file exists
|
||||
modified when the database file last changed, in UTC
|
||||
schema version the version of the database's layout
|
||||
records the number of records
|
||||
size of recorded files their total size, in bytes, then rounded
|
||||
`
|
||||
|
||||
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
|
||||
"sfdupes trees" runs. Run it as root, so that the scan can read every
|
||||
file. The scan is stored in the database, as scan stores it, so
|
||||
"sfdupes trees" prints the report again later.
|
||||
`
|
||||
)
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
// without a subcommand. That is a usage error, and the usage text
|
||||
// cobra prints for it is the whole message.
|
||||
var errNoSubcommand = errors.New("no subcommand")
|
||||
|
||||
// errWorkersBelowOne is the usage error for a --workers value below 1.
|
||||
// errWorkersBelowOne is the usage error for a scan --workers value
|
||||
// below 1.
|
||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||
|
||||
// errEmptyExcludeValue is the usage error for an empty --exclude value,
|
||||
// as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||
// would name the working directory.
|
||||
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||
|
||||
// Version is the build version, injected at link time via -ldflags
|
||||
// (see the Makefile); empty when the linker set none, as for a binary
|
||||
// built by go install. buildVersion turns it into what --version prints.
|
||||
@@ -255,116 +187,65 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
}),
|
||||
}
|
||||
|
||||
root.Long = root.Short + ".\n\n" + databaseHelp
|
||||
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
|
||||
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
|
||||
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim,
|
||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd,
|
||||
newDwimCommand(stdout))
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
||||
|
||||
return root
|
||||
}
|
||||
|
||||
// scanOptions holds the values of the flags that scan and dwim share:
|
||||
// --workers, -x, --no-default-excludes and --exclude.
|
||||
type scanOptions struct {
|
||||
workers int
|
||||
oneFS bool
|
||||
noDefaultExcludes bool
|
||||
excludePaths []string
|
||||
}
|
||||
|
||||
// scanFlagsUsage names the flags that scan and dwim share, for their
|
||||
// usage lines.
|
||||
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]..."
|
||||
|
||||
// addScanFlags adds the flags that scan and dwim share to cmd, which
|
||||
// stores their values in opts.
|
||||
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
|
||||
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
|
||||
"concurrent workers for the walk and hash phases")
|
||||
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
// A string array, not a slice, so a comma in a path is not a
|
||||
// separator.
|
||||
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
|
||||
"never enter or record `PATH` or anything below it; repeatable")
|
||||
}
|
||||
|
||||
// check rejects the flag values that are usage errors, before the
|
||||
// command runs. It is the PreRunE of scan and dwim.
|
||||
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
|
||||
err := checkScanWorkers(cmd, o.workers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return rejectEmptyExcludeValue(cmd, o.excludePaths)
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand. Its help names the
|
||||
// database, and lists the default excludes of the system it runs on and
|
||||
// the filesystem types it never records anything on.
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x and
|
||||
// --no-default-excludes flags. Its help lists the default excludes of
|
||||
// the system it runs on, and the filesystem types it never records
|
||||
// anything on.
|
||||
func newScanCommand() *cobra.Command {
|
||||
var opts scanOptions
|
||||
var (
|
||||
scanWorkers int
|
||||
scanOneFS bool
|
||||
scanNoDefaultExcludes bool
|
||||
)
|
||||
|
||||
short := "Walk trees and synchronize the scan database"
|
||||
|
||||
scanCmd := &cobra.Command{
|
||||
Use: cmdScan + " " + scanFlagsUsage + " PATH...",
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||
// The Use line already names every flag, so cobra must not
|
||||
// append its generic "[flags]" to it.
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + databaseHelp + "\n" +
|
||||
defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: opts.check,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return checkScanWorkers(cmd, scanWorkers)
|
||||
},
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
ctx, stop := interruptContext(ctx)
|
||||
defer stop()
|
||||
|
||||
return runScan(ctx, args, opts)
|
||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
||||
scanNoDefaultExcludes)
|
||||
}),
|
||||
}
|
||||
addScanFlags(scanCmd, &opts)
|
||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||
"concurrent workers for the walk and hash phases")
|
||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
|
||||
return scanCmd
|
||||
}
|
||||
|
||||
// newDwimCommand builds the dwim subcommand, which takes the scan flags
|
||||
// with the meaning they have for scan. Its help says what dwim runs,
|
||||
// then lists the default excludes and the filesystem types the scan
|
||||
// never records anything on.
|
||||
func newDwimCommand(stdout io.Writer) *cobra.Command {
|
||||
var opts scanOptions
|
||||
|
||||
short := "Scan /, then print the duplicate-tree report"
|
||||
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim + " " + scanFlagsUsage,
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + dwimHelp + "\n" +
|
||||
defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.NoArgs,
|
||||
PreRunE: opts.check,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", opts, stdout)
|
||||
}),
|
||||
}
|
||||
addScanFlags(dwimCmd, &opts)
|
||||
|
||||
return dwimCmd
|
||||
}
|
||||
|
||||
// checkScanWorkers rejects a --workers value below 1. That is a usage
|
||||
// error reported in one line: cobra prints the returned message without
|
||||
// the usage text, and run exits 2.
|
||||
// checkScanWorkers rejects a scan --workers value below 1. That is a
|
||||
// usage error reported in one line: cobra prints the returned message
|
||||
// without the usage text, and run exits 2.
|
||||
func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
if workers >= 1 {
|
||||
return nil
|
||||
@@ -375,18 +256,6 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
||||
}
|
||||
|
||||
// rejectEmptyExcludeValue rejects an empty --exclude value. That
|
||||
// is a usage error reported in one line, as for --workers below 1.
|
||||
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||
if !slices.Contains(excludePaths, "") {
|
||||
return nil
|
||||
}
|
||||
|
||||
cmd.SilenceUsage = true
|
||||
|
||||
return errEmptyExcludeValue
|
||||
}
|
||||
|
||||
// runE adapts a subcommand implementation, or the version print, to
|
||||
// cobra's RunE. Cobra prints the error and the command's usage text for
|
||||
// every error RunE returns, but a subcommand that ran and failed has no
|
||||
|
||||
@@ -22,12 +22,6 @@ import (
|
||||
// error prints and a runtime failure must not.
|
||||
const usageMarker = "Usage:"
|
||||
|
||||
// helpFlag asks for the help text.
|
||||
const helpFlag = "--help"
|
||||
|
||||
// workersFlag is the scan and dwim flag that sets the worker count.
|
||||
const workersFlag = "--workers"
|
||||
|
||||
// TestMain stops the tests at once, saying why, when their temporary
|
||||
// directory is not on a persistent filesystem: scan records nothing
|
||||
// there (README "Rules for the walk"), so most scan tests would fail
|
||||
@@ -576,8 +570,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
}{
|
||||
cmdScan: {
|
||||
run: func() error {
|
||||
return runScan(t.Context(), []string{missing},
|
||||
scanOptions{workers: 1})
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false)
|
||||
},
|
||||
want: scanWant,
|
||||
},
|
||||
@@ -668,7 +661,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
args := []string{cmdScan, workersFlag, workers, t.TempDir()}
|
||||
args := []string{cmdScan, "--workers", workers, t.TempDir()}
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitUsage {
|
||||
@@ -699,7 +692,7 @@ func TestRunHelp(t *testing.T) {
|
||||
|
||||
// README §Settings: help goes to stderr, exits 0, and leaves stdout
|
||||
// empty.
|
||||
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
|
||||
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
|
||||
|
||||
for _, args := range cases {
|
||||
var stdout, stderr bytes.Buffer
|
||||
@@ -719,16 +712,6 @@ func TestRunHelp(t *testing.T) {
|
||||
args, got)
|
||||
}
|
||||
}
|
||||
|
||||
// The scan help lists --exclude with its argument.
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run([]string{cmdScan, "-h"}, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||
t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
|
||||
excludeFlag)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanUsageLine(t *testing.T) {
|
||||
@@ -736,8 +719,7 @@ func TestRunScanUsageLine(t *testing.T) {
|
||||
|
||||
// README §Subcommands shows this usage line for scan, and the help
|
||||
// and the usage error both print it.
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...\n"
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
||||
|
||||
cases := map[string][]string{
|
||||
"help": {cmdScan, "-h"},
|
||||
@@ -760,73 +742,6 @@ func TestRunScanUsageLine(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Settings: the help names the database, and says what
|
||||
// report, trees, info and dwim print.
|
||||
database := []string{
|
||||
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
|
||||
}
|
||||
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||
|
||||
// The subcommand whose help is printed; none for the root.
|
||||
cases := []struct {
|
||||
subcommand []string
|
||||
want []string
|
||||
}{
|
||||
{nil, database},
|
||||
{[]string{cmdScan}, database},
|
||||
{
|
||||
[]string{cmdReport},
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n size ",
|
||||
"largest first",
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdTrees},
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n files ",
|
||||
"\n size ", "\n disk_size ", "largest first",
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdInfo},
|
||||
[]string{
|
||||
"\n database ", "\n database file size ",
|
||||
"\n -wal file size ", "\n modified ",
|
||||
"\n schema version ", "\n records ",
|
||||
"\n size of recorded files ",
|
||||
},
|
||||
},
|
||||
{
|
||||
[]string{cmdDwim},
|
||||
[]string{
|
||||
"sfdupes scan /", "as root", "sfdupes trees", "database",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
args := slices.Concat(tc.subcommand, []string{helpFlag})
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
|
||||
}
|
||||
|
||||
for _, want := range tc.want {
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("run(%v) help lacks %q:\n%s",
|
||||
args, want, stderr.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunVersion(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -5,20 +5,19 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||
// root" runs with the scan flags in opts, then what "sfdupes trees"
|
||||
// runs, and ends with a line on stderr saying how to print the report
|
||||
// again. The dwim command passes "/" as root. It catches SIGINT and
|
||||
// SIGTERM as scan does, but only during the scan. An error from the
|
||||
// scan, an interrupt included, ends it before the report. During the
|
||||
// report, a signal ends it at once, as it ends trees.
|
||||
func runDwim(
|
||||
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
|
||||
) error {
|
||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
||||
// with a line on stderr saying how to print the report again. The dwim
|
||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
||||
// does, but only during the scan. An error from the scan, an interrupt
|
||||
// included, ends it before the report. During the report, a signal
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, opts)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||
|
||||
stop()
|
||||
|
||||
|
||||
@@ -2,10 +2,7 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
@@ -31,7 +28,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout)
|
||||
err := runDwim(t.Context(), root, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
@@ -70,7 +67,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout)
|
||||
err := runDwim(t.Context(), missing, &stdout)
|
||||
if err == nil {
|
||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||
}
|
||||
@@ -84,127 +81,3 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
|
||||
// dwim passes its --exclude paths to the scan: an excluded copy is
|
||||
// not recorded, so its pair is not reported.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
root := realTempDir(t)
|
||||
|
||||
kept := []string{
|
||||
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||
}
|
||||
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
|
||||
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
|
||||
|
||||
err := runDwim(t.Context(), root, opts, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||
if len(lines) != 2 {
|
||||
t.Fatalf("stdout = %q, want a header and one group",
|
||||
stdout.String())
|
||||
}
|
||||
|
||||
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
|
||||
if !strings.HasPrefix(lines[1], want) {
|
||||
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
||||
}
|
||||
|
||||
skipped := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if got := stderr(); !strings.Contains(got, skipped) {
|
||||
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimTakesScanFlags(t *testing.T) {
|
||||
// The dwim command parses and checks the scan flags as scan does.
|
||||
// Each case stops before the walk of / begins, and before the
|
||||
// database is created. "dwim --exclude /" excludes its own operand.
|
||||
cases := map[string]struct {
|
||||
args []string
|
||||
code int
|
||||
want string
|
||||
}{
|
||||
"--exclude /": {
|
||||
[]string{cmdDwim, excludeFlag, "/"},
|
||||
exitFatal,
|
||||
"sfdupes: scan /: operand excluded by --exclude /\n",
|
||||
},
|
||||
"--exclude empty": {
|
||||
[]string{cmdDwim, excludeFlag, ""},
|
||||
exitUsage,
|
||||
"Error: --exclude must not be empty\n",
|
||||
},
|
||||
"--workers 0": {
|
||||
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
|
||||
exitUsage,
|
||||
"Error: --workers must be at least 1, got 0\n",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(tc.args, os.Stdout, os.Stderr)
|
||||
if code != tc.code {
|
||||
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
|
||||
}
|
||||
|
||||
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
|
||||
"ending %q", stdout(), stderr(), tc.want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimHelpListsScanFlags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README "Subcommands" shows this usage line for dwim.
|
||||
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]...\n"
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("dwim -h stderr = %q, want the usage line %q",
|
||||
stderr.String(), want)
|
||||
}
|
||||
|
||||
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
|
||||
excludeFlag)
|
||||
}
|
||||
|
||||
// README "Settings": the help lists the default excludes and the
|
||||
// filesystem types the walk never records anything on.
|
||||
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
|
||||
if !strings.Contains(stderr.String(), help) {
|
||||
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
|
||||
help)
|
||||
}
|
||||
}
|
||||
+12
-104
@@ -1,9 +1,8 @@
|
||||
package sfdupes
|
||||
|
||||
// This file holds three rules for the walk (README "Rules for the
|
||||
// walk"): the paths given with --exclude, which the scan never enters;
|
||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
||||
// the default excludes, the paths a scan's walk does not enter below an
|
||||
// operand unless scan is given --no-default-excludes; and the
|
||||
// operand unless scan is given --no-default-excludes, and the
|
||||
// filesystems that are not persistent, on which the scan never records
|
||||
// or reads anything.
|
||||
|
||||
@@ -13,7 +12,6 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -45,25 +43,17 @@ const typesPerHelpLine = 6
|
||||
// readMountInfo reads.
|
||||
var errMountInfoLine = errors.New("unexpected line")
|
||||
|
||||
// errExcludedOperand reports an operand at or below a path given with
|
||||
// --exclude.
|
||||
var errExcludedOperand = errors.New("operand excluded by --exclude")
|
||||
|
||||
// walkRules holds the rules for the walk that a scan sets up as it
|
||||
// starts. excludePaths are the paths given with --exclude, each with
|
||||
// the device ID and inode number of the file it names (see
|
||||
// excludePathInodes), which the walk neither enters nor records,
|
||||
// whatever path leads to them. defaultExcludes are the default
|
||||
// excludes, or none with --no-default-excludes. nonPersistent maps the
|
||||
// device ID of every mounted filesystem that is not persistent to its
|
||||
// type; the scan records and reads nothing on one of those, whatever
|
||||
// the flags. databaseFiles are the scan's own database files, which the
|
||||
// walk passes over (see databaseFiles in db.go).
|
||||
// starts. excludes are the default excludes, or none with
|
||||
// --no-default-excludes. nonPersistent maps the device ID of every
|
||||
// mounted filesystem that is not persistent to its type; the scan
|
||||
// records and reads nothing on one of those, whatever the flags.
|
||||
// databaseFiles are the scan's own database files, which the walk
|
||||
// passes over (see databaseFiles in db.go).
|
||||
type walkRules struct {
|
||||
excludePaths []fileRec
|
||||
defaultExcludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
excludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
}
|
||||
|
||||
// defaultExcludes returns the default excludes of the system goos
|
||||
@@ -107,88 +97,6 @@ func isDefaultExclude(path string, excludes []string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// excludePathInodes returns each of paths, the --exclude values, as its
|
||||
// real path (see realPaths) with the device ID and inode number of the
|
||||
// file there. The walk compares what it reaches with those, as it does
|
||||
// with the database files, so it also leaves out an --exclude path
|
||||
// spelled another way, such as in another letter case on macOS.
|
||||
func excludePathInodes(paths []string) ([]fileRec, error) {
|
||||
resolved, err := realPaths(paths)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
files := make([]fileRec, 0, len(resolved))
|
||||
|
||||
for _, path := range resolved {
|
||||
fi, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
dev, ino := inodeOfInfo(fi)
|
||||
files = append(files, fileRec{path: path, dev: dev, ino: ino})
|
||||
}
|
||||
|
||||
return files, nil
|
||||
}
|
||||
|
||||
// matchingExcludePath returns the path of the one of excludePaths with
|
||||
// the device ID and inode number of fi, an lstat result, or "" when
|
||||
// there is none.
|
||||
func matchingExcludePath(fi fs.FileInfo, excludePaths []fileRec) string {
|
||||
dev, ino := inodeOfInfo(fi)
|
||||
f := fileRec{dev: dev, ino: ino}
|
||||
|
||||
for _, exclude := range excludePaths {
|
||||
if sameInode(exclude, f) {
|
||||
return exclude.path
|
||||
}
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// checkOperandsNotExcluded returns an error when one of roots, the
|
||||
// operands, or a directory above it is one of excludePaths, the
|
||||
// --exclude paths, by device ID and inode number. The error names the
|
||||
// operand and the --exclude path. The roots are absolute, so the
|
||||
// directories above each one end at /. A path here that lstat cannot
|
||||
// read is an error too.
|
||||
func checkOperandsNotExcluded(roots []string, excludePaths []fileRec) error {
|
||||
for _, root := range roots {
|
||||
for path := root; ; path = filepath.Dir(path) {
|
||||
fi, err := os.Lstat(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("scan %s: %w", root, err)
|
||||
}
|
||||
|
||||
exclude := matchingExcludePath(fi, excludePaths)
|
||||
if exclude != "" {
|
||||
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
|
||||
exclude)
|
||||
}
|
||||
|
||||
if path == "/" {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// excludePathLine returns the line the scan prints for path when fi,
|
||||
// its lstat, makes it one of rules.excludePaths, and "" when it does
|
||||
// not.
|
||||
func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
|
||||
if matchingExcludePath(fi, rules.excludePaths) == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
||||
}
|
||||
|
||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||
// not persistent: filesystems that the kernel makes up or that are kept
|
||||
// in memory. devfs is the macOS filesystem of /dev; the others are
|
||||
@@ -254,7 +162,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
|
||||
return devices, lines.Err()
|
||||
}
|
||||
|
||||
// defaultExcludesHelp describes, for the scan and dwim help, the
|
||||
// defaultExcludesHelp describes, for the scan command's help, the
|
||||
// default excludes of the system goos names and, where the scan lists
|
||||
// the mounted filesystems, the types it never records anything on.
|
||||
func defaultExcludesHelp(goos string) string {
|
||||
|
||||
@@ -14,9 +14,6 @@ import (
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// excludeFlag is the scan flag that names a path the walk never enters.
|
||||
const excludeFlag = "--exclude"
|
||||
|
||||
func TestDefaultExcludes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -191,7 +188,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
||||
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
walkRules{excludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -237,7 +234,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
|
||||
st, err := syncScan(t.Context(), db, roots, 1, false,
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
walkRules{excludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -265,7 +262,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
||||
f := writeFile(t, excluded, "f", pattern(1, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
walkRules{excludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -359,217 +356,6 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
||||
assertRecorded(t, path, f, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeNested scans a tree through the command with an
|
||||
// --exclude path two levels below the operand. The walk records nothing
|
||||
// at or below it and says so. The record an earlier scan left there is
|
||||
// kept although its file is gone.
|
||||
func TestRunScanExcludeNested(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
excluded := filepath.Join(dir, "a", "b")
|
||||
kept := writeFile(t, dir, "a/f", pattern(1, 10))
|
||||
gone := writeFile(t, excluded, "gone", pattern(2, 10))
|
||||
|
||||
scanOK(t, dir)
|
||||
|
||||
err := os.Remove(gone)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
|
||||
|
||||
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||
|
||||
line := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
|
||||
assertRecorded(t, path, kept, true)
|
||||
assertRecorded(t, path, gone, true)
|
||||
assertRecorded(t, path, unseen, false)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeRepeated gives --exclude twice, once for a
|
||||
// directory and once for a file, the file through a symlink to it. The
|
||||
// walk records neither, and records the file beside them.
|
||||
func TestRunScanExcludeRepeated(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
|
||||
file := writeFile(t, dir, "skip-file", pattern(2, 10))
|
||||
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||
|
||||
link := filepath.Join(t.TempDir(), "link")
|
||||
symlink(t, file, link)
|
||||
|
||||
scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
|
||||
|
||||
assertRecorded(t, path, inDir, false)
|
||||
assertRecorded(t, path, file, false)
|
||||
assertRecorded(t, path, kept, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeReachedByAnotherPath gives --exclude a directory
|
||||
// through a symlink and a file through "..", so the walk reaches
|
||||
// neither by the path given. The walk also reaches the file through a
|
||||
// hard link. It records none of these, prints a line naming each, and
|
||||
// records the file beside them.
|
||||
func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
inDir := writeFile(t, dir, "a/cache/f", pattern(1, 10))
|
||||
file := writeFile(t, dir, "a/big", pattern(2, 10))
|
||||
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||
|
||||
hardLink := filepath.Join(dir, "big-link")
|
||||
|
||||
err := os.Link(file, hardLink)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
link := filepath.Join(t.TempDir(), "link")
|
||||
symlink(t, filepath.Join(dir, "a"), link)
|
||||
|
||||
// Joined as text, since filepath.Join would drop the "..".
|
||||
dotDot := dir + "/a/cache/../big"
|
||||
|
||||
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
|
||||
excludeFlag, dotDot, dir)
|
||||
|
||||
for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
|
||||
line := "scan: skipped " + skipped + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
}
|
||||
|
||||
assertRecorded(t, path, inDir, false)
|
||||
assertRecorded(t, path, file, false)
|
||||
assertRecorded(t, path, hardLink, false)
|
||||
assertRecorded(t, path, kept, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludedOperand checks that an operand at or below an
|
||||
// --exclude path is a fatal error that names both, and so is an
|
||||
// --exclude path that does not exist. An operand that is a hard link to
|
||||
// an --exclude file is at it. Neither scan writes a database.
|
||||
func TestRunScanExcludedOperand(t *testing.T) {
|
||||
dir := realTempDir(t)
|
||||
sub := filepath.Join(dir, "sub")
|
||||
f := writeFile(t, sub, "f", pattern(1, 10))
|
||||
|
||||
hardLink := filepath.Join(dir, "f-link")
|
||||
|
||||
err := os.Link(f, hardLink)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
cases := map[string]struct {
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
"at it": {
|
||||
[]string{excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"below it": {
|
||||
[]string{excludeFlag, dir, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
|
||||
},
|
||||
"after an operand it does not exclude": {
|
||||
[]string{dir, excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"a hard link to it": {
|
||||
[]string{excludeFlag, f, hardLink},
|
||||
"scan " + hardLink + ": operand excluded by --exclude " + f + "\n",
|
||||
},
|
||||
"exclude missing": {
|
||||
[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
|
||||
"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
|
||||
os.Stderr)
|
||||
if code != exitFatal {
|
||||
t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
|
||||
exitFatal)
|
||||
}
|
||||
|
||||
assertFatalOutput(t, stderr(), stdout())
|
||||
|
||||
if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
|
||||
t.Errorf("stderr = %q, want %q", stderr(), tc.want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
|
||||
// given as its own argument, after "=" or after another --exclude, is
|
||||
// a usage error reported in one line before anything is scanned
|
||||
// (README "Settings").
|
||||
func TestRunScanRejectsEmptyExclude(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
cases := map[string][]string{
|
||||
"separate": {excludeFlag, "", dir},
|
||||
"joined by =": {excludeFlag + "=", dir},
|
||||
"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdScan}, args...), &stdout,
|
||||
&stderr)
|
||||
if code != exitUsage {
|
||||
t.Errorf("run(scan %q) = %d, want %d", args, code,
|
||||
exitUsage)
|
||||
}
|
||||
|
||||
want := "Error: --exclude must not be empty\n"
|
||||
if stdout.String() != "" || stderr.String() != want {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
|
||||
stdout.String(), stderr.String(), want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
||||
// directory on a filesystem that is not persistent, with and without
|
||||
// --no-default-excludes: either way it is skipped, with a line saying
|
||||
|
||||
+55
-84
@@ -98,47 +98,44 @@ func scanPhase(name string) string {
|
||||
// runs, and the lock is released after it. When ctx is cancelled, as by
|
||||
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps
|
||||
// what it has hashed (see syncScan), prints how many files its walk
|
||||
// reached, and returns errInterrupted. opts.workers must be at least 1;
|
||||
// the scan and dwim commands reject anything less. Before the lock is
|
||||
// taken, the scan lists the filesystems that are not persistent, on
|
||||
// which the walk records nothing; unless opts.noDefaultExcludes is set,
|
||||
// the walk does not enter the default excludes either. The walk neither
|
||||
// enters nor records the paths in opts.excludePaths, the --exclude
|
||||
// values; an operand at or below one is a fatal error. Once the
|
||||
// database is open, the scan notes its files, which the walk passes
|
||||
// over.
|
||||
func runScan(ctx context.Context, roots []string, opts scanOptions) error {
|
||||
// reached, and returns errInterrupted. workers must be at least 1; the
|
||||
// scan command rejects anything less. Before the lock is taken, the
|
||||
// scan lists the filesystems that are not persistent, on which the walk
|
||||
// records nothing; unless noDefaultExcludes is set, the walk does not
|
||||
// enter the default excludes either. Once the database is open, the
|
||||
// scan notes its files, which the walk passes over.
|
||||
func runScan(ctx context.Context, roots []string, workers int,
|
||||
oneFS, noDefaultExcludes bool,
|
||||
) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
printDatabasePath(dbPath, source)
|
||||
// A database file missing now is created by lockScanDatabase below.
|
||||
what := "database"
|
||||
|
||||
roots, err = realPaths(roots)
|
||||
_, err = os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
|
||||
roots, err = resolveRoots(roots)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var rules walkRules
|
||||
|
||||
rules.excludePaths, err = excludePathInodes(opts.excludePaths)
|
||||
if err != nil {
|
||||
return fmt.Errorf("--exclude: %w", err)
|
||||
}
|
||||
|
||||
err = checkOperandsNotExcluded(roots, rules.excludePaths)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
rules.nonPersistent, err = nonPersistentFilesystems()
|
||||
nonPersistent, err := nonPersistentFilesystems()
|
||||
if err != nil {
|
||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
||||
}
|
||||
|
||||
if !opts.noDefaultExcludes {
|
||||
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
|
||||
rules := walkRules{nonPersistent: nonPersistent}
|
||||
if !noDefaultExcludes {
|
||||
rules.excludes = defaultExcludes(runtime.GOOS)
|
||||
}
|
||||
|
||||
lock, err := lockScanDatabase(dbPath)
|
||||
@@ -166,7 +163,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
|
||||
return err
|
||||
}
|
||||
|
||||
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules)
|
||||
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return interrupted(st.walked)
|
||||
}
|
||||
@@ -184,21 +181,6 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// printDatabasePath prints on stderr the line that names the database
|
||||
// at dbPath, with source, saying whether it is new. A database file
|
||||
// missing now is new: lockScanDatabase creates it.
|
||||
func printDatabasePath(dbPath, source string) {
|
||||
what := "database"
|
||||
|
||||
_, err := os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
}
|
||||
|
||||
// interruptContext returns a copy of ctx that the first SIGINT or
|
||||
// SIGTERM cancels; the scan command runs the scan under it. stop
|
||||
// releases the signals.
|
||||
@@ -227,25 +209,24 @@ func interrupted(walked int) error {
|
||||
return errInterrupted
|
||||
}
|
||||
|
||||
// realPaths returns the real path of each of paths, the PATH operands
|
||||
// or the --exclude values: absolute, with every symlink in it followed.
|
||||
// Database records are keyed by that path, so scan results depend
|
||||
// neither on the working directory nor on the symlinks an operand was
|
||||
// given through. A path that cannot be resolved, because it or a
|
||||
// symlink's target does not exist or cannot be read, is a fatal error
|
||||
// before any scanning.
|
||||
func realPaths(paths []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(paths))
|
||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||
// with every symlink in it followed. Database records are keyed by
|
||||
// that path, so scan results depend neither on the working directory
|
||||
// nor on the symlinks an operand was given through. An operand that
|
||||
// cannot be resolved, because it or a symlink's target does not exist
|
||||
// or cannot be read, is a fatal error before any scanning.
|
||||
func resolveRoots(roots []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(roots))
|
||||
|
||||
for _, path := range paths {
|
||||
a, err := absolutePath(path)
|
||||
for _, root := range roots {
|
||||
a, err := absolutePath(root)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
r, err := filepath.EvalSymlinks(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
resolved = append(resolved, r)
|
||||
@@ -314,7 +295,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
|
||||
}
|
||||
|
||||
for ; dir != above; dir = filepath.Dir(dir) {
|
||||
if isDefaultExclude(dir, rules.defaultExcludes) {
|
||||
if isDefaultExclude(dir, rules.excludes) {
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -437,7 +418,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||
|
||||
// walkableRoots returns the operands the walk can start from: regular
|
||||
// files, and directories not named .zfs, none of them inside a
|
||||
// directory named .zfs. The operands are real paths from realPaths,
|
||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||
// so these checks apply to what a symlink operand points at. Every
|
||||
// other operand is warned about, counted as skipped, and dropped. A
|
||||
// dropped operand is no longer a root, so the records stored beneath it
|
||||
@@ -512,11 +493,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// as verified. When its disk_size differs from the record's, it is
|
||||
// queued for the update phase, which writes the new disk_size. A record
|
||||
// at or below a path the walk left out, because -x stopped at it, it is
|
||||
// an --exclude path or a default exclude, or it is not on a persistent
|
||||
// filesystem, counts as outside the roots: it is not deleted. That does
|
||||
// not hold below a root at or inside that path, since that root's own
|
||||
// walk verifies the records below it; nor is a line printed for such a
|
||||
// root.
|
||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
||||
// outside the roots: it is not deleted. That does not hold below a root
|
||||
// at or inside that path, since that root's own walk verifies the
|
||||
// records below it; nor is a line printed for such a root.
|
||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
// The records the last scan left under the roots estimate how many
|
||||
// files the walk will find. A first scan has none, and no estimate.
|
||||
@@ -854,9 +834,9 @@ type dirJob struct {
|
||||
// file's statted record, a warning when fail is set, or a path the walk
|
||||
// left out: when otherFSDir is set, a directory -x did not enter
|
||||
// because it is on another filesystem than its operand, and when
|
||||
// leftOut is set, an --exclude path, a default exclude or a directory or
|
||||
// file on a filesystem that is not persistent, with warn holding the
|
||||
// line that says so.
|
||||
// leftOut is set, a default exclude or a directory or file on a
|
||||
// filesystem that is not persistent, with warn holding the line that
|
||||
// says so.
|
||||
type walkEvent struct {
|
||||
rec fileRec
|
||||
warn string
|
||||
@@ -907,7 +887,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
||||
// operandWarning returns the one-line warning for an operand the walk
|
||||
// does not start from, naming the path and what it is, or "" for one it
|
||||
// does: a regular file, or a directory not named .zfs, that is not
|
||||
// inside a directory named .zfs. realPaths has already followed the
|
||||
// inside a directory named .zfs. resolveRoots has already followed the
|
||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||
// after it was resolved, and is not followed.
|
||||
func operandWarning(root string, fi fs.FileInfo) string {
|
||||
@@ -1138,9 +1118,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
||||
// no longer exists, and one that has stopped being a regular file is
|
||||
// skipped. A file on a filesystem in rules that is not persistent, as
|
||||
// through a bind mount of one file, is not recorded: it gets an event
|
||||
// naming it instead, and so does a file given with --exclude, reached
|
||||
// by any path, a hard link to it included. One of the scan's own
|
||||
// database files in rules gets no event at all.
|
||||
// naming it instead. One of the scan's own database files in rules gets
|
||||
// no event at all.
|
||||
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
rules walkRules, events chan<- walkEvent,
|
||||
) {
|
||||
@@ -1164,10 +1143,6 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p, info)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
@@ -1192,11 +1167,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
// subdirJob applies the descent rules to directory p: never enter
|
||||
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
||||
// snapshot), never enter a directory on a filesystem in rules that is
|
||||
// not persistent, nor an --exclude path or a default exclude in rules,
|
||||
// and with -x never enter a directory on a different filesystem than
|
||||
// its operand. For all but .zfs it sends an event that names the
|
||||
// directory, so the scan keeps the records below it, except those below
|
||||
// an operand at or inside it.
|
||||
// not persistent, nor a default exclude in rules, and with -x never
|
||||
// enter a directory on a different filesystem than its operand. For the
|
||||
// last three it sends an event that names the directory, so the scan
|
||||
// keeps the records below it, except those below an operand at or
|
||||
// inside it.
|
||||
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
parent dirJob, oneFS bool, rules walkRules,
|
||||
events chan<- walkEvent,
|
||||
@@ -1221,17 +1196,13 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p, info)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
return dirJob{}, false
|
||||
}
|
||||
|
||||
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||
if isDefaultExclude(p, rules.excludes) {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
leftOut: p,
|
||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
||||
|
||||
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := run([]string{
|
||||
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir,
|
||||
cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
|
||||
}, io.Discard, &stderr)
|
||||
if code != exitFatal {
|
||||
t.Fatalf("run(scan) = %d, want %d; stderr: %s",
|
||||
@@ -2289,7 +2289,7 @@ func TestPruneRoots(t *testing.T) {
|
||||
|
||||
// The walk of /p does not enter /p/q when it is a default exclude, so
|
||||
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
||||
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
|
||||
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
|
||||
device)
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
||||
|
||||
Reference in new issue
Block a user