clawbot
68881f04f2
check / check (push) Canceled after 0s
The head, tail and content columns of the files table are BLOB NOT NULL and hold the raw 32-byte SHA-256, half the size of the hex text, in the table and in the files_signature index. The hash phase, the records, report and trees carry each hash as a [sha256.Size]byte and compare it with ==; nothing is hex-encoded. Reading a record whose hash column is not 32 bytes is an error naming its path. In the tests, testHash is a stand-in hash: a string's bytes padded with zeros, so fixtures keep the sort order their strings give. Judgement call: a hash of the wrong length is refused on reading, not copied silently. Model: opus-5-5
1169 lines
61 KiB
Markdown
1169 lines
61 KiB
Markdown
# sfdupes
|
|
|
|
## Description
|
|
|
|
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
|
|
quickly finds _candidate_ duplicate files, and ultimately whole duplicate
|
|
directory trees, on very large filesystems. It does this without reading every
|
|
byte of every file.
|
|
|
|
Two files are duplicates when they have the same size and the same three SHA-256
|
|
hashes. Two of the hashes cover the first and the last 64 KiB of the file. The
|
|
third, the content hash, covers the whole file when it is under 100 MiB, so a
|
|
match proves identical content. At 100 MiB and above it covers only 1 MiB at
|
|
each gigabyte, so such a file is never read in full. A match there is a strong
|
|
candidate rather than proof.
|
|
|
|
The intended use is finding duplicate downloads and duplicated directory trees
|
|
on multi-terabyte ZFS servers, where reading every byte of every file is
|
|
prohibitively expensive. `scan` keeps a persistent SQLite database of file
|
|
signatures between runs, so it can run from cron. The reports can be generated
|
|
at any time from the most recent scan.
|
|
|
|
This README is the complete and authoritative specification.
|
|
|
|
## Getting Started
|
|
|
|
```sh
|
|
make build
|
|
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
|
./sfdupes scan /srv
|
|
./sfdupes report > dupes.tsv
|
|
./sfdupes trees > dupetrees.tsv
|
|
```
|
|
|
|
`scan` walks one or more filesystem trees and maintains one database record per
|
|
regular file (path, size, mtime, head hash, tail hash, content hash). The
|
|
database persists between runs. A rescan hashes only the files that are new or
|
|
changed, and removes the records of files that no longer exist. `report` reads
|
|
the database and prints the file-level duplicates report. `trees` reads the same
|
|
database and prints the duplicate-tree report. `info` prints the database's
|
|
details: its path, the sizes of its files and the number of its records. A
|
|
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
|
|
prints a usage message and exits 2.
|
|
|
|
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
|
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
|
cron job, with the reporting commands run interactively whenever needed; their
|
|
results are as fresh as the last completed scan.
|
|
|
|
### Install
|
|
|
|
With Go installed, this builds and installs the newest release:
|
|
|
|
```sh
|
|
go install sneak.berlin/go/sfdupes@latest
|
|
```
|
|
|
|
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
|
|
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
|
|
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
|
|
`@main` installs the tip before it is tagged:
|
|
|
|
```sh
|
|
go install sneak.berlin/go/sfdupes@main
|
|
```
|
|
|
|
`v0.0.1` is an early build without a database. It does not work as this README
|
|
describes, and `go.mod` retracts it.
|
|
|
|
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
|
|
see which version an installed binary is:
|
|
|
|
```sh
|
|
sfdupes --version
|
|
go version -m "$(command -v sfdupes)"
|
|
```
|
|
|
|
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
|
|
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
|
|
or for `@main` a pseudo-version ending in the first 12 characters of the
|
|
commit's hash. A binary built from a clone or into the Docker image carries the
|
|
git tag or commit it was built from instead, and `VERSION` is `dev` only when
|
|
the binary carries no version at all. `go version -m` prints what Go recorded in
|
|
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
|
|
and earlier print `dev`, so check those with `go version -m`.
|
|
|
|
From a clone, `make build` writes the binary to `./sfdupes`:
|
|
|
|
```sh
|
|
git clone https://git.eeqj.de/sneak/sfdupes.git
|
|
cd sfdupes
|
|
make build
|
|
```
|
|
|
|
Copy the binary to `/usr/local/bin` for the cron job below.
|
|
|
|
`make docker` builds the Docker image, tagged `sfdupes`, after running the tests
|
|
and the linter (see "Build"). The image runs `sfdupes` as root with the database
|
|
at its default path, so a bind mount of `/var/lib/sfdupes` keeps the database
|
|
between runs. Mount the scanned tree at the same path inside the container as on
|
|
the host; read-only is enough. The database records paths as the container sees
|
|
them, so the reports then name the host's paths.
|
|
|
|
```sh
|
|
make docker
|
|
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
|
sfdupes scan /srv
|
|
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
|
```
|
|
|
|
### Daily scan from cron
|
|
|
|
Run `scan` as root, so that it can read every file: a path it cannot read is
|
|
skipped with a warning and loses its database record (see "Rules for the walk").
|
|
As a file `/etc/cron.d/sfdupes`:
|
|
|
|
```
|
|
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
|
```
|
|
|
|
- The database is `/var/lib/sfdupes/db.sqlite`, created with its directory by
|
|
the first scan. To keep it elsewhere, set
|
|
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
|
|
- `scan` writes nothing to stdout. Its stderr, appended here to
|
|
`/var/log/sfdupes.log`, holds these lines:
|
|
- a first line naming the database (see "Database");
|
|
- plain progress lines naming each phase and its place among the four, as
|
|
`[2/4] walk`: one as the phase starts, one every 5 seconds while it runs,
|
|
and one as it ends;
|
|
- a warning for each path it skips;
|
|
- a line for each default exclude its walk does not enter, and for each path
|
|
it leaves out because it is not on a persistent filesystem (see "Rules for
|
|
the walk");
|
|
- the summary line (see "Progress" and "`scan` mode").
|
|
|
|
The log grows with every scan; rotate it like any other.
|
|
|
|
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
|
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
|
reason among the last lines of the log; `tail` prints them, and cron mails
|
|
them to root if the host can send mail.
|
|
- A scan still running when the next one starts carries on. The new one fails at
|
|
once, and the lines cron mails include
|
|
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
|
- `report`, `trees` and `info` need only read access to the database (see
|
|
"Database"). Under the usual umask of `022` the first scan creates it readable
|
|
by every user, so an unprivileged user can run them against root's database.
|
|
|
|
### Reading the reports
|
|
|
|
Each row of `report` names two copies of one file, and each row of `trees` two
|
|
copies of one directory tree (see "Report output format" and "Trees output
|
|
format"). In a group of copies, the path that sorts first byte by byte is
|
|
`first` and every other path is a `dupe` of it. `first` says nothing about which
|
|
copy is the original or the oldest; which copy to keep is your choice.
|
|
|
|
A row is a candidate, not proof:
|
|
|
|
- The reports read only the database, so they show the files as of the last
|
|
scan; a file may have changed or gone since.
|
|
- A file of 100 MiB or more is compared only on samples of its content (see
|
|
"Duplicate detection").
|
|
- Paths that are hard links to one file are listed as duplicates, but they share
|
|
their data, so removing one frees nothing.
|
|
|
|
Compare a pair byte for byte before removing either copy. For the row
|
|
`/srv/a/big.iso`, `/srv/b/big-copy.iso`, `4294967296`:
|
|
|
|
```sh
|
|
cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
|
|
[ /srv/a/big.iso -ef /srv/b/big-copy.iso ] && echo "hard links"
|
|
```
|
|
|
|
`cmp` prints nothing and exits 0 only when every byte matches, and otherwise
|
|
reports where the files differ. The second line prints `hard links` when the two
|
|
paths are the same file, so removing either frees nothing.
|
|
|
|
A path holding a backslash, tab, newline or carriage return is escaped in the
|
|
reports (see "Report output format"). Undo the escapes before using it.
|
|
`printf '%b'` does exactly that, because every backslash in an escaped path
|
|
starts one of the four escapes. Command substitution drops trailing newlines, so
|
|
print an `x` after the path and remove it afterwards, or a path that ends in a
|
|
newline names a different file:
|
|
|
|
```sh
|
|
p="$(printf '%bx' '/srv/a/tab\tname.txt')"; p="${p%x}"
|
|
cmp "$p" /srv/b/tab-copy.txt
|
|
```
|
|
|
|
Check a `trees` row with `diff -r`, which compares the two trees file by file
|
|
and also names anything present in only one of them, such as an empty directory
|
|
or a symlink, which `trees` does not see.
|
|
|
|
## Rationale
|
|
|
|
Duplicate finders that hash every file in full do not scale to the target
|
|
environment. It holds about 10 million files and 150 TB, on disks that may be
|
|
slow or busy, such as a ZFS pool under resilver.
|
|
|
|
sfdupes reads a file under 100 MiB in full, and a larger one only at its two
|
|
ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
|
makes the comparison exact. A full read of every large file would take
|
|
unreasonably long. Sampling cuts a 150 GiB file to about 150 MiB of reads, at
|
|
the price of a comparison that is not exact.
|
|
|
|
The hashes are kept in a persistent database, so the expensive filesystem pass
|
|
is incremental. A rescan reads only the files that are new or whose size or
|
|
mtime changed. All analysis then works from the database alone.
|
|
|
|
The end goal is not individual files but whole duplicated trees, such as a
|
|
download extracted twice. An operator can consider removing such a tree as a
|
|
unit.
|
|
|
|
## Design
|
|
|
|
Goals, in order:
|
|
|
|
1. **Find whole duplicate trees, not just files.** The end goal is to identify
|
|
places where the exact same set of files and directories exists at two or
|
|
more paths (duplicate extractions, duplicate downloads, copied project
|
|
trees), so the operator can consider removing an entire subtree at once.
|
|
File-level duplicate detection is the foundation; tree-level detection is
|
|
built on top of it.
|
|
2. **Bound the I/O per file.** A file is read only when it is new or has
|
|
changed. A file under 100 MiB is then read in full, and a larger one only at
|
|
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
|
|
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
|
|
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
|
|
under resilver). Holding one small record (path, size, mtime) per file in
|
|
memory during a scan is acceptable; holding every file's hashes is not (they
|
|
stay in the database). The reporting commands do not hold every file's
|
|
hashes either: `report` reads the records in signature order and holds the
|
|
duplicate paths of one size at a time, so its memory grows with the
|
|
duplicates of the size that has the most of them, and `trees` reads the
|
|
records in path order and keeps each directory's path, digest and totals,
|
|
plus the hashes of only the files in the directories holding the record
|
|
being read, so its memory grows with the number of directories and with the
|
|
size of the largest directory.
|
|
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
|
|
maintains a persistent database, and an unchanged file is never read again
|
|
on a rescan. All analysis (`report`, `trees`) works from the database alone
|
|
and must never touch the scanned filesystem again. `scan` is designed to be
|
|
cronned; the reports run at any time against the last completed scan.
|
|
4. **Clean stream separation.** Everything on stdout is machine-readable data.
|
|
All progress, warnings, summaries, and help and usage text go to stderr.
|
|
Never mix them.
|
|
|
|
### Constraints
|
|
|
|
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
|
|
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
|
|
progress-bar library** (`github.com/schollz/progressbar/v3`),
|
|
`golang.org/x/term` to tell whether stderr is a terminal, **one SQLite
|
|
driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled), and
|
|
`golang.org/x/sys` for `flock(2)` (the scan lock, see "Database").
|
|
`github.com/spf13/viper` is permitted if configuration-file support is ever
|
|
needed, but is not currently used. No other third-party deps.
|
|
- Cross-compilation is not a concern. Builds run with cgo disabled (the
|
|
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
|
|
- Analysis modes (`report`, `trees`) must be deterministic: identical database
|
|
contents, identical output, regardless of the order in which records were
|
|
inserted.
|
|
|
|
### Subcommands
|
|
|
|
Four subcommands, all implemented:
|
|
|
|
1. `scan` — walk the filesystem and synchronize the database: one signature
|
|
record per regular file.
|
|
2. `report` — file-level duplicate report from the database.
|
|
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
|
|
from the database records, compute a Merkle-style digest per directory, and
|
|
report maximal groups of identical trees.
|
|
4. `info` — the database's details: its path, the sizes of its files, its
|
|
schema version, the number of its records and their total size.
|
|
|
|
```
|
|
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
|
sfdupes report > dupes.tsv
|
|
sfdupes trees > dupetrees.tsv
|
|
sfdupes info
|
|
sfdupes --version
|
|
sfdupes [command] --help
|
|
```
|
|
|
|
`--workers N` sets the size of each `scan` worker pool (default: the number of
|
|
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
|
|
operand's filesystem, and `--no-default-excludes` lets the walk enter the
|
|
default excludes, a list of paths; all three are described under "`scan` mode".
|
|
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
|
|
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
|
|
alone or after a subcommand, prints the help text to stderr and exits 0, writing
|
|
nothing to stdout.
|
|
|
|
### Database
|
|
|
|
All four subcommands operate on a single SQLite database file:
|
|
|
|
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
|
|
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
|
|
command-line flag. The path names the file exactly, whatever characters it
|
|
holds (`?`, `#` and `%` included); a relative path is relative to the working
|
|
directory.
|
|
- Each subcommand first prints one line on stderr naming the database it uses,
|
|
before anything else it prints (progress displays included), whether or not
|
|
stderr is a terminal. The line gives the path with the working directory put
|
|
in front of it when it is relative, and otherwise as set (a `..` in it stays,
|
|
so the line names the same file), escaped like the paths in a report (see
|
|
"Report output format"), and says where the path came from:
|
|
|
|
```
|
|
scan: new database /var/lib/sfdupes/db.sqlite (default)
|
|
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
|
```
|
|
|
|
The working directory put in front is the one `pwd -P` prints, with its
|
|
symlinks resolved. After a `cd` through a symlink to a directory, a path
|
|
starting `../` therefore names a file beside the directory the symlink leads
|
|
to, as it does for any other program. `scan` says `new database` when no
|
|
file is at that path as it starts. `report`, `trees` and `info` print the
|
|
line before they open the database, so the error for a missing database
|
|
comes after it.
|
|
|
|
- `scan` creates the database (and its parent directory) on first use. `report`,
|
|
`trees` and `info` require an existing database; a missing database file is a
|
|
fatal error (exit 1) telling the user to run `scan` first.
|
|
- Only one `scan` runs against a database at a time. For its whole run, `scan`
|
|
holds an exclusive `flock(2)` lock on a lock file beside the database, named
|
|
by appending `.lock` to the database's path with symlinks resolved
|
|
(`/var/lib/sfdupes/db.sqlite.lock` by default), taken before it walks the
|
|
filesystem or opens the database. Every path to one database, through a
|
|
symlink to the file or to a directory above it, names the same lock file. Just
|
|
before it takes the lock, `scan` creates the database file, empty, if it is
|
|
missing (through a symlink at the database path, it creates the symlink's
|
|
target), so the lock file is named the same way before the first scan sets up
|
|
the database as after. A second `scan` against the same database does not
|
|
wait: it fails at once with a one-line error naming the lock file and exits 1,
|
|
without walking anything or opening the database, and the running scan carries
|
|
on. The lock file is created on first use, open to its owner only, and left in
|
|
place: a leftover file blocks nothing, because the lock ends with the process
|
|
holding it however it ends, a fatal error or an interrupt included, and
|
|
deleting the file while a scan runs would let a second scan start. `report`,
|
|
`trees` and `info` never take the lock, so they run during a scan.
|
|
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
|
|
running a report while a cron `scan` is in progress is safe. The filesystem is
|
|
authoritative; the database is an eventually-consistent reflection of it.
|
|
Hashed records are committed in batched transactions while the scan is still
|
|
running (keeping the WAL small and letting concurrent reports observe
|
|
progress), so a report may see a scan's changes partially applied, and a scan
|
|
that dies partway leaves a valid database holding every batch committed so far
|
|
(an interrupted scan also commits the batch in progress, see "Error handling
|
|
and exit codes"); the next scan skips those records and converges toward the
|
|
filesystem.
|
|
- `scan` switches the database back to rollback-journal mode when it closes it,
|
|
so between scans the database file alone holds the whole database. Each switch
|
|
needs the database to itself: a `scan` that starts while a report is still
|
|
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
|
|
that ends while a report has the database open warns and leaves the database
|
|
in WAL mode until the next scan. `report` writes its rows while it reads the
|
|
records, so it is still reading while its output is paused (a pager, a stalled
|
|
pipe), and a `scan` started then fails after the busy timeout.
|
|
- A `scan` writes to the database and the files beside it at these points (its
|
|
phases are described under "`scan` mode"):
|
|
- As it starts, before the walk: the directory, an empty database file and
|
|
the `.lock` file, each only if missing.
|
|
- Then, still before the walk: the schema, in a new database, and the `-wal`
|
|
and `-shm` files, which stay beside the database while the scan runs.
|
|
- While it loads the existing records (`load`) and walks: no records. A
|
|
first scan's database holds none, however long its walk has run.
|
|
- During the hash phase: the first records, committed 10,000 at a time.
|
|
- In the update phase: the rest of the hashed records, and the deletions.
|
|
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
|
away, unless a report still has the database open (see above).
|
|
- `report`, `trees` and `info` open the database read-only and need only read
|
|
access to the database file, and no write access to its directory. While the
|
|
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
|
|
which SQLite creates with the database file's permissions.
|
|
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
|
|
with any other version is a fatal error. `scan` creates the schema and sets
|
|
the version in one transaction, so a first scan stopped while doing so leaves
|
|
an empty database the next scan sets up. A database at version 0 that already
|
|
has a `files` table was therefore not made by sfdupes; every subcommand
|
|
refuses it with an error telling the user to remove the file and rescan):
|
|
|
|
```sql
|
|
CREATE TABLE files (
|
|
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
|
size INTEGER NOT NULL, -- bytes, from lstat
|
|
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
|
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
|
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
|
tail BLOB NOT NULL, -- raw 32-byte SHA-256 of the last 64 KiB
|
|
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
|
|
) WITHOUT ROWID;
|
|
CREATE INDEX files_signature ON files (size, head, tail, content);
|
|
```
|
|
|
|
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
|
|
UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond:
|
|
`mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the
|
|
nanoseconds past that second. Split this way they hold any mtime a
|
|
filesystem can record, one before 1678 or after 2262 included, which a
|
|
single 64-bit count of nanoseconds cannot. They are used only for change
|
|
detection and are not part of the duplicate key. Every record holds the
|
|
three hashes that "Duplicate detection" below describes. The
|
|
`files_signature` index lets `report` read the records in signature order
|
|
without sorting the table.
|
|
|
|
### Duplicate detection
|
|
|
|
Two files are duplicates when they have the same size and the same three hashes.
|
|
`scan` computes the hashes and stores them in the database. `report` and `trees`
|
|
group the files whose size and hashes all match, whichever scan recorded them.
|
|
|
|
`scan` gives every file it records three SHA-256 hashes:
|
|
|
|
- `head`, the hash of the file's first 64 KiB.
|
|
- `tail`, the hash of the file's last 64 KiB.
|
|
- `content`, the content hash described below.
|
|
|
|
In a file of 64 KiB or less, the first and the last 64 KiB are both the whole
|
|
file, so all three hashes are the hash of the whole file.
|
|
|
|
The content hash depends on the file's size:
|
|
|
|
- Under 100 MiB, it is the SHA-256 of the whole file. Agreement on it is proof
|
|
of identical content, barring a SHA-256 collision.
|
|
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
|
|
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
|
|
GiB and so on. The samples are fed into one hash in order. A sample that would
|
|
run past the end of the file stops there.
|
|
|
|
The sampled hash is **deliberately probabilistic**. Bytes outside the samples
|
|
and outside the first and last 64 KiB are never read. Two large files that
|
|
differ only in such bytes are reported as duplicates. That is the price of never
|
|
reading a 150 GB file end to end.
|
|
|
|
Only files of the same size are compared, so their samples always lie at the
|
|
same offsets. A file under 100 MiB and one of 100 MiB or more never share a
|
|
size, so a whole-file hash is never compared with a sampled one.
|
|
|
|
`scan` reads a file only to hash it, and only when the file is new or has
|
|
changed since its record was written (see "`scan` mode").
|
|
|
|
### `scan` mode
|
|
|
|
`scan` requires one or more `PATH` operands naming the trees to scan. There is
|
|
no default path; invoking `scan` with no operand is a usage error (usage message
|
|
on stderr, exit 2). An operand may be a directory or a regular file; an operand
|
|
that does not exist is a fatal error (exit 1). Because database records persist
|
|
between runs and are keyed by absolute path, each operand is resolved to an
|
|
absolute, lexically cleaned path (symlinks are not resolved) before walking, so
|
|
results do not depend on the working directory. All operands belong to a single
|
|
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
|
pool. Overlapping operands are harmless — an operand that duplicates another or
|
|
lies under another is dropped before walking, so every file is reached exactly
|
|
once and produces one database record. An operand under another one is kept,
|
|
though, and walked on its own when any directory from it (from its directory,
|
|
for a regular file) up to but not including the operand covering it is a default
|
|
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
|
|
`-x`, is on a different filesystem than that operand, because that operand's
|
|
walk does not enter such a directory and so does not reach it.
|
|
|
|
An operand that is a symlink (never followed, not even as an operand), socket,
|
|
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
|
|
a one-line warning naming the path and what it is, counts it as skipped, and
|
|
drops it from the scanned operands before reading the database. Another operand
|
|
beneath it is still scanned. The records stored beneath it are not deleted: they
|
|
are treated like any other record outside the scanned operands. If it lies under
|
|
another operand, they are under that operand instead, and are treated like any
|
|
other record there. This is not an error: a scan whose every operand is dropped
|
|
walks nothing and exits 0. An operand that is not on a persistent filesystem
|
|
(see "Rules for the walk") is dropped the same way. The line `scan` prints for
|
|
it is not a warning, though, and it is not counted as skipped.
|
|
|
|
`scan` synchronizes the database with the filesystem state under the scanned
|
|
operands:
|
|
|
|
- A file not yet in the database is hashed and inserted.
|
|
- A file already in the database is **skipped without reading its contents**
|
|
when its lstat size equals the recorded size and its lstat mtime is not newer
|
|
than the recorded mtime. This is what makes a daily rescan cheap.
|
|
- A file whose mtime is newer than recorded, or whose size differs, is hashed
|
|
again. Change detection compares the mtime to the nanosecond, as finely as the
|
|
filesystem records it, so a same-size rewrite counts as a change whenever the
|
|
filesystem gives it a later mtime than recorded, even within the same second.
|
|
- A database record whose path lies under one of the scanned operands but was
|
|
not successfully processed this run is deleted. This removes records for
|
|
deleted files. It also removes records for paths that failed to stat or hash
|
|
this run: the database only ever contains signatures verified by the most
|
|
recent scan that covered them (a subsequent successful scan re-adds such
|
|
files).
|
|
- Database records outside the scanned operands are untouched, so disjoint trees
|
|
can be scanned on different schedules into the same database.
|
|
- A record at or below a path the walk left out counts as outside the scanned
|
|
operands, so it is not deleted. The walk leaves out a path that:
|
|
- is a default exclude;
|
|
- is not on a persistent filesystem;
|
|
- with `-x`, is a directory on another filesystem than its operand.
|
|
|
|
A filesystem mounted under an operand can therefore be scanned on its own
|
|
schedule into the same database. Records below an operand at or inside such
|
|
a path are not kept this way, since that operand's own walk verifies them.
|
|
|
|
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
|
|
inside the walk and hash phases, and batched database writes begin during the
|
|
hash phase:
|
|
|
|
1. **load** — read the existing records. Each record under the operands is kept
|
|
in memory, by path, with its size and its mtime, for the walk to compare
|
|
each file with.
|
|
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
|
with the walk worker pool: every operand seeds the shared queue, and each
|
|
worker reads one directory at a time, handing discovered subdirectories back
|
|
to the queue and running `lstat` on each regular file as it is discovered
|
|
(while the directory's metadata is still hot). Sequential directory
|
|
enumeration is metadata-latency-bound and takes hours at tens of millions of
|
|
files; per-directory parallelism is what makes the walk tractable on large
|
|
or busy pools. The walk compares each file with its record as it goes, and
|
|
an unchanged file needs nothing more. Every new or changed file is carried
|
|
to the hash phase as a (path, size, mtime) record.
|
|
3. **hash** — the worker pool hashes every carried file as described in
|
|
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
|
`content`. Zero-length files have constant hashes and are never opened.
|
|
Files are hashed in **inode order**, which minimizes seeks on spinning
|
|
disks. Paths that are hard links to the same inode are **read once**, and
|
|
all share the one result. A hard-link backup farm therefore costs one read
|
|
per inode, not per path. The phase total counts actual reads, so progress
|
|
and ETA are meaningful. Completed records are committed in batched
|
|
transactions **while hashing runs**. A scan interrupted after hours
|
|
therefore keeps everything hashed so far, and the next scan resumes cheaply,
|
|
skipping records already written.
|
|
4. **update** — commit the final partial batch of hashed records. Then delete
|
|
the records the scan did not verify: those of vanished files, and of paths
|
|
that failed to stat or hash.
|
|
|
|
Rules for the walk:
|
|
|
|
- Only regular files. Skip directories, symlinks (do not follow, including
|
|
symlink operands), sockets, FIFOs, and device nodes. An operand that is a
|
|
symlink, socket, FIFO, or device node is dropped as described in "`scan` mode"
|
|
above.
|
|
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
|
|
them would list every file once per snapshot), not even when it is an operand;
|
|
such an operand is dropped the same way.
|
|
- Never record the scan's own database. Once the database and its `.lock` file
|
|
are open, `scan` notes the device and inode of the database file, its `-wal`
|
|
and `-shm` files and its `.lock` file, and the walk passes over a file with
|
|
one of them, however it reaches it: as an operand, below one, or through a
|
|
symlinked directory. It prints no line for such a file and does not count it
|
|
as skipped. A record that an earlier scan left for one of them is deleted,
|
|
like the record of any file the scan no longer verifies. There is no
|
|
`-journal` file during the walk, because the database is in WAL mode then (see
|
|
"Database").
|
|
- Filesystem boundaries are crossed by default. With `-x` (long form
|
|
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
|
|
into a directory on a different filesystem than its `PATH` operand; each
|
|
operand is bounded by its own filesystem. The records below such a directory
|
|
count as outside the scanned operands, as described in "`scan` mode" above:
|
|
they are not deleted. That does not hold below an operand at or inside that
|
|
directory.
|
|
- Never record anything that is not on a persistent filesystem. A filesystem is
|
|
not persistent when the kernel makes it up or keeps it in memory: one of type
|
|
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
|
|
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
|
|
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
|
|
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
|
|
it names the database and before it takes the lock, `scan` lists the mounted
|
|
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
|
|
of mounted filesystems on macOS (on any other system it lists none), and notes
|
|
the device ID of each one of those types. On Linux that is the device ID the
|
|
list gives, so a filesystem that a later mount hides still gives its own. On
|
|
macOS it is read with `lstat` on the mount point, and where one filesystem is
|
|
mounted over another on the same mount point, the one mounted last counts. If
|
|
`scan` cannot list them, it fails (exit 1). The walk then never enters a
|
|
directory, and never records a file, on one of those devices, however it
|
|
reaches it: as an operand, below one, or through a bind mount, which keeps the
|
|
device ID of what it binds. No flag turns this off. For each operand,
|
|
directory or file it leaves out this way, `scan` prints a line on stderr
|
|
naming it and the type of its filesystem. It prints that line as it prints a
|
|
warning, so the line never lands inside a progress display:
|
|
|
|
```
|
|
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
|
|
```
|
|
|
|
That line is not a warning, and the path is not counted as skipped. An
|
|
earlier scan's records at or below it count as outside the scanned operands,
|
|
as described in "`scan` mode" above: they are not deleted. That does not
|
|
hold below an operand at or inside it.
|
|
|
|
- Below an operand, never descend into a default exclude: a path that holds the
|
|
kernel's view of the running system, or temporary files, so that
|
|
`sfdupes scan /` walks only the files worth comparing. The default excludes
|
|
depend on the system, and `sfdupes scan --help` lists those of the system it
|
|
runs on:
|
|
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
|
|
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
|
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
|
|
everything in `/System/Volumes/Data` through the firmlinks, so walking it
|
|
as well would record each of those files twice.
|
|
- On Windows, relative to the root of the drive an operand lies on: the
|
|
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
|
|
`hiberfil.sys`, and the directories `System Volume Information`,
|
|
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
|
|
matches one path component, as it does in any default exclude. sfdupes
|
|
does not build for Windows yet, so this list is not used.
|
|
- On any other system there are none.
|
|
|
|
`--no-default-excludes` turns them all off; what is not on a persistent
|
|
filesystem stays left out. An operand at or inside a default exclude is
|
|
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
|
|
persistent filesystem. For each default exclude its walk does not enter,
|
|
`scan` prints a line on stderr naming it, as it prints a warning:
|
|
|
|
```
|
|
scan: skipped /var/tmp (default exclude)
|
|
```
|
|
|
|
That line is not a warning, and the directory is not counted as skipped. A
|
|
default exclude that is not on a persistent filesystem, as `/proc` is, gets
|
|
the line for its filesystem instead. One that is itself an operand gets no
|
|
line, since it is walked as that operand. When the walk does not enter a
|
|
default exclude, the records below it count as outside the scanned operands,
|
|
as described in "`scan` mode" above: they are not deleted. That does not
|
|
hold below an operand at or inside it.
|
|
|
|
- On any per-path error (permission denied, file vanished between passes,
|
|
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
|
Per-file errors never abort the run; the final summary reports how many were
|
|
skipped. As specified above, a skipped path that has a database record from an
|
|
earlier scan loses that record. An unreadable directory subtree likewise loses
|
|
its records. This is accepted: the database mirrors what the latest scan could
|
|
actually verify. The exception is an operand dropped before the database was
|
|
read, whose record counts as outside the scanned operands.
|
|
|
|
Concurrency: the walk phase, which also stats files, and the hash phase each use
|
|
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
|
|
parallelizes across directories, and hashing across files. `--workers` must be
|
|
at least 1: a smaller value is a usage error, reported in one line on stderr
|
|
with exit 2 before anything is scanned. Both phases are seek-bound on spinning
|
|
disks, so raising `--workers` well past the core count can help on pools with
|
|
many spindles. The main goroutine owns the database writes and the progress
|
|
counts. Each progress display draws the counts from a goroutine of its own, and
|
|
must never block the workers.
|
|
|
|
`scan` writes nothing to stdout. The summary line on stderr reports the files
|
|
seen this run broken down by disposition, plus skips:
|
|
|
|
```
|
|
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
|
|
```
|
|
|
|
(`removed` counts deleted database records, which are not part of the files-seen
|
|
total.)
|
|
|
|
### `report` mode
|
|
|
|
`report` reads every record from the database and takes no positional arguments.
|
|
|
|
**`report` must never touch the filesystem being analyzed.** It does not stat,
|
|
open, or otherwise access any path that appears in the records; its only I/O is
|
|
reading the database and writing stdout/stderr. `report` must produce identical
|
|
output whether or not the scanned filesystem is still mounted.
|
|
|
|
Processing:
|
|
|
|
- Group the records by the key `(size, head, tail, content)`.
|
|
- Every group with two or more paths is a duplicate group.
|
|
- Within each group, sort paths lexicographically (byte order). The first path
|
|
is the group's `first`; every other path is a `dupe`.
|
|
- Order groups by size descending (biggest reclaimable space first), tie-broken
|
|
by `first` path ascending. Output must be fully deterministic for a given
|
|
database state.
|
|
|
|
`report` runs in two phases, each with its own progress display on stderr (see
|
|
"Progress"): `open` opens the database; `read` counts the records, then reads
|
|
them in signature order, largest size first. That is the order of the
|
|
`files_signature` index read backwards, so SQLite sorts nothing and returns the
|
|
first record at once. The records sharing a signature come together, and so do
|
|
the records of one size, so `report` holds only the duplicate groups of the size
|
|
it is reading, and writes them, ordered by `first` path, once it reaches a
|
|
smaller size. Its memory holds at most the duplicate paths of one size at a
|
|
time, never every record.
|
|
|
|
#### Report output format
|
|
|
|
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
|
|
group of N):
|
|
|
|
```
|
|
first dupe size
|
|
/srv/a/big.iso /srv/b/big-copy.iso 4294967296
|
|
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296
|
|
```
|
|
|
|
Paths are raw bytes and may hold any byte except NUL, so the path columns
|
|
(`first` and `dupe`) are escaped to keep every row one line of tab-separated
|
|
fields: a backslash is written as `\\`, a tab as `\t`, a newline as `\n`, and a
|
|
carriage return as `\r`. Every other byte is written unchanged, including bytes
|
|
that are not valid UTF-8. Undoing those four escapes gives back the stored path.
|
|
Grouping and ordering use the stored path, not the escaped one. The warnings
|
|
`scan` prints on stderr are escaped the same way, so each warning is one line.
|
|
|
|
Summary to stderr: records read, number of duplicate groups, number of dupe
|
|
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
|
units.
|
|
|
|
### `trees` mode
|
|
|
|
`trees` reads the same database as `report` (no positional arguments) and
|
|
reports **entire duplicate directory trees**: directories under which the exact
|
|
same set of relative paths exists with the exact same file signatures.
|
|
|
|
**`trees` must never touch the filesystem being analyzed** — the same rule as
|
|
`report`. The directory hierarchy is reconstructed purely from the paths in the
|
|
records, split on `/`.
|
|
|
|
Definitions:
|
|
|
|
- A file's **signature** is `(size, head, tail, content)` — mtime is
|
|
informational and excluded.
|
|
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
|
|
serialize the directory's child entries — for a file child, its name and
|
|
signature; for a subdirectory child, its name and that subdirectory's digest —
|
|
sort the serialized entries byte-lexicographically, and hash the
|
|
concatenation. Names are part of the digest: two trees whose files differ only
|
|
in name are _not_ duplicates.
|
|
- Two directories are **duplicate trees** when their digests are equal. Equal
|
|
digests imply equal recursive file count and equal total byte size.
|
|
|
|
Known limitation (accepted): hard-linked paths are reported as duplicates by
|
|
`report` and count toward duplicate trees — their content is genuinely identical
|
|
— even though they share storage, so removing one reclaims no space. Inode
|
|
identity is used during the scan to avoid redundant reads but is not persisted
|
|
in the database.
|
|
|
|
Known limitation (accepted): only regular files that appear in the database
|
|
define a tree. Empty directories are invisible, and a file skipped during the
|
|
scan (e.g. permission error) in one copy but not the other will make
|
|
otherwise-identical trees compare as different.
|
|
|
|
Processing:
|
|
|
|
- Build the hierarchy, compute every directory's digest, and group directories
|
|
by digest. Every group with two or more directories is a duplicate-tree group.
|
|
- **Report only maximal trees.** A group is suppressed when its members' parents
|
|
are pairwise distinct directories that all share a single digest — such a
|
|
group is wholly implied by its parents' (or a further ancestor's) group.
|
|
Groups containing sibling directories, or members whose parents differ, are
|
|
always reported.
|
|
- Within each group, sort paths lexicographically (byte order); the first path
|
|
is `first`, every other path is a `dupe`.
|
|
- Order groups by total tree size descending, tie-broken by `first` path
|
|
ascending. Output must be fully deterministic for a given input.
|
|
|
|
`trees` runs in four phases, each with its own progress display on stderr (see
|
|
"Progress"): `open` opens the database; `read` counts the records, then reads
|
|
them in path order, building the hierarchy and computing the digests as it goes;
|
|
`group` groups the directories by digest and picks the maximal groups; `write`
|
|
writes the groups to stdout.
|
|
|
|
#### Trees output format
|
|
|
|
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
|
group of N). `files` is the recursive regular-file count of one copy of the
|
|
tree; `size` is the recursive total byte size of one copy:
|
|
|
|
```
|
|
first dupe files size
|
|
/srv/a/project /srv/backup/project 3417 104857600
|
|
```
|
|
|
|
The `first` and `dupe` paths are escaped as described under "Report output
|
|
format". The root directory's path is `/`.
|
|
|
|
Summary to stderr: records read, number of duplicate-tree groups, number of dupe
|
|
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
|
units.
|
|
|
|
### `info` mode
|
|
|
|
`info` takes no positional arguments and prints the database's details on
|
|
stdout, one per line: a name, a tab, and the value. It opens the database the
|
|
way `report` does, read-only and without the scan lock, so it runs during a scan
|
|
and needs only read access (see "Database"). Like `report`, it never touches the
|
|
scanned filesystem.
|
|
|
|
```
|
|
database /var/lib/sfdupes/db.sqlite (default)
|
|
database file size 2147483648 (2.0 GiB)
|
|
-wal file size 4124152 (3.9 MiB)
|
|
modified 2026-10-16T12:02:50Z
|
|
schema version 1
|
|
records 12345678
|
|
size of recorded files 150000000000000 (136.4 TiB)
|
|
```
|
|
|
|
- `database` is the path and where it came from, as on the line every subcommand
|
|
prints first on stderr (see "Database").
|
|
- `database file size` and `-wal file size` are in bytes, then in human units.
|
|
The `-wal` line is there only when a `-wal` file is beside the database, as
|
|
while a scan runs.
|
|
- `modified` is the time the database file was last changed, in UTC. While the
|
|
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
|
- `schema version` is the database's `PRAGMA user_version`. A database with
|
|
another version than this build's is refused, as by `report`.
|
|
- `records` is the number of records. `size of recorded files` adds up the sizes
|
|
of all records, in bytes, then in human units.
|
|
|
|
`info` runs in two phases, each with its own progress display on stderr (see
|
|
"Progress"): `open` opens the database; `count` reads the schema version and
|
|
counts the records with one query, a single pass over the table, both in one
|
|
read-only transaction, so the numbers agree with each other while a scan is
|
|
committing. `info` writes its lines once the `count` display has ended, so on a
|
|
terminal they come after the displays.
|
|
|
|
### Progress
|
|
|
|
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
|
|
`info`; rendering in the style of `pv` is the model. All progress goes to
|
|
stderr.
|
|
|
|
`scan`, `report`, `trees` and `info` must always show that they are working and
|
|
not hung. Each phase gets its own display from the moment it starts, naming the
|
|
phase and its place among the command's phases and showing its progress, its
|
|
elapsed time and its estimated time remaining (ETA).
|
|
|
|
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
|
`[4/4] update`):
|
|
|
|
- `load` counts the records first, then reads them against that count. While the
|
|
count runs, the display says it is counting the records.
|
|
- `walk` has no exact total. Its estimate is the number of records the last scan
|
|
left under the operands, known once `load` is done and shown with a `~`. Once
|
|
the walk passes the estimate, or when there are no such records, as on a first
|
|
scan, the ETA is unknown.
|
|
- `hash` and `update` have exact totals. Only files that actually need hashing
|
|
appear in the hash total, so its ETA is meaningful.
|
|
|
|
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
|
`[4/4] write`, see "`trees` mode"):
|
|
|
|
- `open` has no total. While it opens the database, waiting on a busy one
|
|
included, the display says it is opening the database.
|
|
- `read` counts the records first, then reads them against that count. While the
|
|
count runs, the display says it is counting the records.
|
|
- `group` has the directories `read` found as its total, and `write` the
|
|
duplicate-tree groups.
|
|
|
|
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
|
|
|
|
- `open` has no total. While it opens the database, waiting on a busy one
|
|
included, the display says it is opening the database.
|
|
- `read` counts the records first, then reads them against that count, writing
|
|
the rows to stdout as it goes. While the count runs, the display says it is
|
|
counting the records.
|
|
|
|
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
|
|
|
- `open` has no total. While it opens the database, waiting on a busy one
|
|
included, the display says it is opening the database.
|
|
- `count` has no total either, because one query counts everything: the display
|
|
says it is counting the records until the query returns, and its ETA is
|
|
unknown.
|
|
|
|
Each display is one line holding:
|
|
|
|
- the phase and its place among the phases
|
|
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
|
has a total, the items processed, or what the phase is waiting for
|
|
- current rate (items/s), naming the items the phase counts: files, records,
|
|
directories or groups. It covers the items completed over the last 5 seconds,
|
|
or, when stderr is not a terminal, since the line before, so it falls to 0
|
|
while nothing completes, and a burst of items after a wait shows as a burst
|
|
rather than as a steady rate. Once no item has completed for 10 seconds, the
|
|
word `stalled` and where the phase waits take the rate's place (see below).
|
|
- elapsed time
|
|
- estimated time remaining, or `unknown`. It comes from the phase's average rate
|
|
since it started counting, so that it does not swing with each burst.
|
|
|
|
Example shape (exact layout is flexible, content is not):
|
|
|
|
```
|
|
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
|
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
|
|
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
|
|
```
|
|
|
|
When no item has completed for 10 seconds, as while one large file is read, a
|
|
disk is slow or the database is busy, the line says `stalled` and where the
|
|
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
|
|
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
|
|
may wait on either. `walk` says `disk`. `hash` says `disk/db`, as it reads files
|
|
and writes their records to the database between them. `load`, `update` and the
|
|
`read` phase of `trees` say `db`. `trees`' `write` says `out`, and its `group`,
|
|
which works in memory, says `cpu`. The `read` phase of `report` writes the
|
|
report while it reads the records, so it says `db/out`. A phase still waiting
|
|
for its total already says what it waits for, and says nothing more.
|
|
|
|
Additional requirements:
|
|
|
|
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
|
|
the `open` phase of `report` and `trees`, and both phases of `info`, once it
|
|
has counted what it will do) is drawn as its line followed by the progress-bar
|
|
library's bar, 10 columns wide, which shows the percent in place of the line's
|
|
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
|
library's spinner followed by its line. Each display is drawn again several
|
|
times a second, so its elapsed time keeps moving while a phase waits for its
|
|
next item.
|
|
- On a terminal no line reaches the terminal's last column, however narrow the
|
|
terminal is. A line too wide for the terminal first loses its bar, which gives
|
|
way to the spinner, then its rate, the words of a stalled phase, or what the
|
|
phase is waiting for. The phase and its place, the progress, the elapsed time
|
|
and the ETA are kept whole. On an 80-column terminal, at counts like those in
|
|
the example above, a stalled line keeps its words whenever the elapsed time
|
|
and the ETA are both in minutes. Where the words and the ETA do not both fit,
|
|
as at millions of files and times in hours, the words give way. Only when
|
|
those four alone are too wide, as on a terminal narrower than about 70 columns
|
|
hours into a scan of millions of files, is the line cut short at the
|
|
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
|
|
while a phase runs gets the same: the next draw erases the line within the new
|
|
width and fits the line to it, and the display goes on redrawing in place.
|
|
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
|
redraws: print the line the moment each phase starts, every 5 seconds while it
|
|
runs (whether or not an item has completed, and never more often), and once
|
|
more as the phase ends.
|
|
- Progress must be non-blocking with respect to the worker pool: the main
|
|
goroutine counts the items, and each display draws them from a goroutine of
|
|
its own.
|
|
- A warning printed during a phase always lands on a line of its own, never
|
|
inside the progress display. So does every line written to stdout when stdout
|
|
and stderr are the same file (the same device and inode): one terminal, as in
|
|
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
|
|
sent to. When they are not, stdout is written without waiting on the display,
|
|
and the display keeps being drawn while a slow reader of stdout holds up a
|
|
write.
|
|
- A phase that stops short of its total, as an interrupted one does, is left
|
|
showing the count it stopped at rather than filled up.
|
|
|
|
### Error handling and exit codes
|
|
|
|
- `0`: success, even if individual files were skipped with warnings.
|
|
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
|
|
already running against the same database, the database cannot be
|
|
created/opened/read/written, a missing database for `report`/`trees`/`info`,
|
|
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
|
|
below).
|
|
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
|
|
`--workers` below 1, and `report`/`trees`/`info` with any positional
|
|
argument).
|
|
|
|
A stdout write failure, such as a full disk, is reported in one line on stderr
|
|
and exits 1. Two cases never reach sfdupes as a failed write:
|
|
|
|
- When the reader of a stdout pipe exits early, as in `sfdupes report | head`,
|
|
the next write ends sfdupes with `SIGPIPE`, quietly and without a summary, the
|
|
way `cat` or `sort` end. The shell reports the signal (status 141 in most
|
|
shells), not exit 1.
|
|
- When stdout is closed outright (`sfdupes report >&-`), the Go runtime opens
|
|
`/dev/null` in its place before sfdupes starts, so the output is discarded and
|
|
the run succeeds, as with `> /dev/null`.
|
|
|
|
`scan` stops cleanly on `SIGINT` (Ctrl-C) or `SIGTERM`. Its workers stop taking
|
|
work, each finishing at most the directory listing or file it is reading; the
|
|
progress display is finished; and the records it has hashed but not yet
|
|
committed are committed, so the next scan does not hash them again. Apart from
|
|
that commit it starts no further writes or deletions: records are deleted only
|
|
after a complete walk, so those under paths an interrupted walk never reached
|
|
are kept. The database is closed and the lock released as on any other exit, the
|
|
line `scan: interrupted after N files` goes to stderr, N being the number of
|
|
files the walk reached, and the exit code is 1. The next scan skips the records
|
|
already written and converges as usual.
|
|
|
|
After the first signal `scan` stops catching them, so a second one ends it at
|
|
once, as an uncaught signal does: the records not yet committed are lost, and
|
|
the database is left valid, as when any scan dies (see "Database"). A `SIGINT`
|
|
that `scan` inherits as ignored, as a script's background job does, stays
|
|
ignored.
|
|
|
|
## Entrypoints
|
|
|
|
This repository adheres to the
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
standard: the normalized executables in `script/` are the entrypoints for the
|
|
development workflow, and the `Makefile` targets are thin shims that call them.
|
|
Every script is POSIX `sh`, resolves the repository root itself so it can be run
|
|
from any working directory, and may be invoked directly. The provided
|
|
entrypoints are:
|
|
|
|
- `script/bootstrap` — install everything needed to build and develop this
|
|
repository, idempotently, assuming nothing is present. `git`, `make`, and `go`
|
|
come from the first of nix, apt, brew, or apk found on the host, and are
|
|
presence-checked only. An installed node is used when its major version is 22;
|
|
otherwise node 22.17.0 is installed through nvm, which comes from a release
|
|
archive whose sha256 the script checks, and yarn 1.22.22 is activated under it
|
|
through corepack. With an installed node 22, a yarn already on `PATH` is used
|
|
as it is; otherwise yarn 1.22.22 is activated through corepack, or installed
|
|
with `npm` when there is no corepack. `yarn install --frozen-lockfile` then
|
|
installs the prettier that `package.json` and `yarn.lock` pin. `golangci-lint`
|
|
is never installed: it runs in Docker (see `script/lint`). `docker` is not
|
|
installed either; testing, linting and the image build need it. Ends with
|
|
`go mod download`.
|
|
- `script/setup` — make a fresh clone ready for development: runs
|
|
`script/bootstrap`, then `script/install-precommit`.
|
|
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
|
|
need the name call it, so they stay identical across repositories.
|
|
- `script/test` — run the test suite under the race detector, with a 90-second
|
|
timeout and coverage enabled, rerunning verbosely on failure so the logs show
|
|
which test failed. It builds the `Dockerfile`'s `test` phase alone and writes
|
|
no image. The race detector needs cgo and a C compiler, which the build never
|
|
uses, so the phase starts from a digest-pinned Debian `golang` image, which
|
|
has `gcc`. The tests run as `nobody`, because several of them make a file
|
|
unreadable and root reads it anyway.
|
|
- `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase
|
|
alone and writes no image: in the digest-pinned `golangci/golangci-lint`
|
|
image, the gofmt check, `golangci-lint config verify` and `golangci-lint run`
|
|
run as build steps, so a successful build is a clean lint. The linter is never
|
|
run on the host, which makes a working `docker` the one prerequisite for
|
|
linting.
|
|
- `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every
|
|
Markdown file with prettier, at the settings in `.prettierrc` (4-space
|
|
indents, prose wrapped at 80 columns). Both run on the host, prettier through
|
|
yarn at the version `yarn.lock` pins. When the node on `PATH` is not node 22,
|
|
this loads the node 22.17.0 that `script/bootstrap` installed through nvm.
|
|
- `script/fmt-check` — the read-only counterpart of `script/fmt`: prints any
|
|
unformatted file and exits non-zero instead of writing. gofmt and prettier
|
|
both run every time, and each names itself when it fails. The `Dockerfile`'s
|
|
`lint` phase runs the same gofmt check.
|
|
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
|
|
that order. Modifies nothing. The first two need `docker`, the third what
|
|
`script/bootstrap` installs.
|
|
- `script/docker` — build the Docker image, tagged with the name from
|
|
`script/projectname`, passing the output of
|
|
`git describe --tags --always --dirty` (or `unknown` when that is empty) as
|
|
the `VERSION` build argument. The `Dockerfile`'s build stage depends on its
|
|
`lint` and `test` phases, so this also runs the tests and the linter.
|
|
- `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the
|
|
image as `script/docker` does. This is what the Gitea workflow runs on push; a
|
|
successful run means every check passed. The tests and the linter run twice:
|
|
once in `script/check`, and again as stages of the image build.
|
|
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
|
|
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
|
|
`script/check`.
|
|
- `script/install-precommit` — install the git pre-commit hook, as
|
|
`.git/hooks/pre-commit`, that runs `script/precommit`.
|
|
|
|
Every `docker build` in `script/` passes a new `CACHEBUST` build argument. In
|
|
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
|
that install system packages and download the Go modules, or right after `FROM`
|
|
in a stage with neither, and every step that lints, tests or compiles comes
|
|
after it, so every `RUN` step after it runs again each time. On an unchanged
|
|
tree Docker would otherwise serve the gate steps from its cache, and the build
|
|
would pass having run no test and no linter. The layers above that line stay in
|
|
Docker's build cache until the base image, their own lines, `go.mod` or `go.sum`
|
|
change, or until that cache is cleared; only then are the packages installed and
|
|
the modules downloaded again. Go's build cache is kept between builds on the
|
|
steps that compile, so only what changed is compiled again; the tests run with
|
|
`-count=1`, so none reports a stored result. `golangci-lint run` also keeps
|
|
golangci-lint's cache, and takes a package's findings from it only when the
|
|
package's files and those of the packages it imports, `go.mod`, the linters and
|
|
their settings in `.golangci.yml`, and the golangci-lint version are all
|
|
unchanged.
|
|
|
|
## Build
|
|
|
|
The `script/` entrypoints above are where the implementations live; the
|
|
`Makefile` targets are shims onto them, except `build`, which carries the
|
|
compile recipe:
|
|
|
|
- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
|
|
the default target.
|
|
- `make bootstrap` — install the build and development dependencies.
|
|
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
|
|
- `make test` — run the test suite under the race detector, in Docker (90-second
|
|
timeout; reruns with `-v` on failure; see `script/test`); requires `docker`.
|
|
- `make lint` — run `golangci-lint` with the repo config and the gofmt check, in
|
|
Docker (see `script/lint`); requires `docker`.
|
|
- `make fmt` / `make fmt-check` — format the Go sources and the Markdown /
|
|
verify formatting without writing, on the host; requires `go` and what
|
|
`make bootstrap` installs (see `script/fmt`).
|
|
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
|
|
`docker` and what `make bootstrap` installs.
|
|
- `make docker` — build the Docker image, which runs the gates as build stages.
|
|
- `make hooks` — install the pre-commit hook.
|
|
- `make clean` — remove the binary.
|
|
|
|
### Definition of done
|
|
|
|
All of the following, run in this directory, must pass:
|
|
|
|
1. `make check` passes (tests, lint, `gofmt`, prettier).
|
|
2. `make docker` succeeds.
|
|
3. Smoke test — create a throwaway tree in a temp dir (never test against real
|
|
data):
|
|
|
|
```sh
|
|
d=$(mktemp -d)
|
|
export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
|
|
mkdir -p "$d/a" "$d/b"
|
|
head -c 2000 /dev/urandom > "$d/a/one.bin"
|
|
cp "$d/a/one.bin" "$d/b/copy.bin"
|
|
cp "$d/a/one.bin" "$d/b/copy2.bin"
|
|
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
|
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
|
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
|
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
|
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
|
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
|
head -c 3000 /dev/urandom > "$d/t1/f1"
|
|
head -c 100 /dev/urandom > "$d/t1/sub/f2"
|
|
cp "$d/t1/f1" "$d/t2/f1"
|
|
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
|
|
cp "$d/t1/f1" "$d/t3/f1"
|
|
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
|
|
./sfdupes scan "$d"
|
|
./sfdupes report
|
|
./sfdupes trees
|
|
# incremental behavior (scan a subtree; records elsewhere persist):
|
|
./sfdupes scan "$d/a" # everything unchanged, nothing hashed
|
|
printf 'z' >> "$d/a/one.bin" # modify: next scan re-hashes it
|
|
rm "$d/a/unique.bin" # delete: next scan removes its record
|
|
./sfdupes scan "$d/a" # 1 updated, 1 removed
|
|
./sfdupes report
|
|
```
|
|
|
|
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
|
|
group (two dupe rows, `first` is the lexicographically smallest path);
|
|
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
|
|
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
|
|
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
|
|
ordered by size descending.
|
|
|
|
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
|
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
|
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
|
differs by name).
|
|
|
|
Expected from the second `report` (after the modify/delete rescan):
|
|
`one.bin` has left its group (its content changed), so
|
|
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
|
|
database.
|
|
|
|
The test suite automates this scenario (see `scan_test.go`), plus a negative
|
|
check: `report` and `trees` operate on the database alone and never touch
|
|
the scanned filesystem.
|
|
|
|
## TODO
|
|
|
|
Tracked in [TODO.md](TODO.md).
|
|
|
|
## Non-goals
|
|
|
|
- No byte-for-byte compare, and no deletion or linking of duplicates. Files that
|
|
match are compared by a SHA-256 of the whole file below 100 MiB, and only by
|
|
samples at 100 MiB and over. The reports are advisory; acting on them is the
|
|
user's job.
|
|
- No persistence beyond the SQLite database described above; no export/import
|
|
formats.
|
|
- No daemon or filesystem watcher; scheduling rescans is cron's job.
|
|
|
|
## License
|
|
|
|
MIT. See [LICENSE](LICENSE).
|
|
|
|
## Author
|
|
|
|
[@sneak](https://sneak.berlin)
|