sneak
dfbfdd0601
check / check (push) Canceled after 0s
The files table gets a disk_size column: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes. trees adds it up per directory and prints it as a column after size. On a compressed or sparse file it is less than size, so two copies of one tree can differ; each row shows the dupe copy's. Judgement call: a row's disk_size is the dupe copy's, not the first's. Unchanged files keep the disk_size from their last hash: a change in allocation that leaves size and mtime alone is not picked up. Model: opus-5-5
1250 lines
65 KiB
Markdown
1250 lines
65 KiB
Markdown
# sfdupes
|
||
|
||
## Description
|
||
|
||
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
|
||
quickly finds _candidate_ duplicate files, and ultimately whole duplicate
|
||
directory trees, on very large filesystems. It does this without reading every
|
||
byte of every file.
|
||
|
||
Two files are duplicates when they have the same size and the same three SHA-256
|
||
hashes. Two of the hashes cover the first and the last 64 KiB of the file. The
|
||
third, the content hash, covers the whole file when it is under 100 MiB, so a
|
||
match proves identical content. At 100 MiB and above it covers only 1 MiB at
|
||
each gigabyte, so such a file is never read in full. A match there is a strong
|
||
candidate rather than proof.
|
||
|
||
The intended use is finding duplicate downloads and duplicated directory trees
|
||
on multi-terabyte ZFS servers, where reading every byte of every file is
|
||
prohibitively expensive. `scan` keeps a persistent SQLite database of file
|
||
signatures between runs, so it can run from cron. The reports can be generated
|
||
at any time from the most recent scan.
|
||
|
||
This README is the complete and authoritative specification.
|
||
|
||
## Getting Started
|
||
|
||
```sh
|
||
make build
|
||
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||
./sfdupes scan /srv
|
||
./sfdupes report > dupes.tsv
|
||
./sfdupes trees > dupetrees.tsv
|
||
```
|
||
|
||
`scan` walks one or more filesystem trees and maintains one database record per
|
||
regular file (path, size, size on disk, mtime, head hash, tail hash, content
|
||
hash). The database persists between runs. A rescan hashes only the files that
|
||
are new or changed, and removes the records of files that no longer exist.
|
||
`report` reads the database and prints the file-level duplicates report. `trees`
|
||
reads the same database and prints the duplicate-tree report. `info` prints the
|
||
database's details: its path, the sizes of its files and the number of its
|
||
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||
operand — prints a usage message and exits 2.
|
||
|
||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||
cron job, with the reporting commands run interactively whenever needed; their
|
||
results are as fresh as the last completed scan.
|
||
|
||
### Install
|
||
|
||
With Go installed, this builds and installs the newest release:
|
||
|
||
```sh
|
||
go install sneak.berlin/go/sfdupes@latest
|
||
```
|
||
|
||
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
|
||
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
|
||
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
|
||
`@main` installs the tip before it is tagged:
|
||
|
||
```sh
|
||
go install sneak.berlin/go/sfdupes@main
|
||
```
|
||
|
||
`v0.0.1` is an early build without a database. It does not work as this README
|
||
describes, and `go.mod` retracts it.
|
||
|
||
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
|
||
see which version an installed binary is:
|
||
|
||
```sh
|
||
sfdupes --version
|
||
go version -m "$(command -v sfdupes)"
|
||
```
|
||
|
||
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
|
||
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
|
||
or for `@main` a pseudo-version ending in the first 12 characters of the
|
||
commit's hash. A binary built from a clone or into the Docker image carries the
|
||
git tag or commit it was built from instead, and `VERSION` is `dev` only when
|
||
the binary carries no version at all. `go version -m` prints what Go recorded in
|
||
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
|
||
and earlier print `dev`, so check those with `go version -m`.
|
||
|
||
From a clone, `make build` writes the binary to `./sfdupes`:
|
||
|
||
```sh
|
||
git clone https://git.eeqj.de/sneak/sfdupes.git
|
||
cd sfdupes
|
||
make build
|
||
```
|
||
|
||
Copy the binary to `/usr/local/bin` for the cron job below.
|
||
|
||
`make docker` builds the Docker image, tagged `sfdupes`, after running the tests
|
||
and the linter (see "Build"). The image runs `sfdupes` as root with the database
|
||
at its default path, so a bind mount of `/var/lib/sfdupes` keeps the database
|
||
between runs. Mount the scanned tree at the same path inside the container as on
|
||
the host; read-only is enough. The database records paths as the container sees
|
||
them, so the reports then name the host's paths.
|
||
|
||
```sh
|
||
make docker
|
||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||
sfdupes scan /srv
|
||
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
||
```
|
||
|
||
### Daily scan from cron
|
||
|
||
Run `scan` as root, so that it can read every file: a path it cannot read is
|
||
skipped with a warning and loses its database record (see "Rules for the walk").
|
||
As a file `/etc/cron.d/sfdupes`:
|
||
|
||
```
|
||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||
```
|
||
|
||
- The database is `/var/lib/sfdupes/db.sqlite`, created with its directory by
|
||
the first scan. To keep it elsewhere, set
|
||
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
|
||
- `scan` writes nothing to stdout. Its stderr, appended here to
|
||
`/var/log/sfdupes.log`, holds these lines:
|
||
- a first line naming the database (see "Database");
|
||
- plain progress lines naming each phase and its place among the four, as
|
||
`[2/4] walk`: one as the phase starts, one every 5 seconds while it runs,
|
||
and one as it ends;
|
||
- a warning for each path it skips;
|
||
- a line for each default exclude its walk does not enter, and for each path
|
||
it leaves out because it is not on a persistent filesystem (see "Rules for
|
||
the walk");
|
||
- the summary line (see "Progress" and "`scan` mode").
|
||
|
||
The log grows with every scan; rotate it like any other.
|
||
|
||
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
||
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
||
reason among the last lines of the log; `tail` prints them, and cron mails
|
||
them to root if the host can send mail.
|
||
- A scan still running when the next one starts carries on. The new one fails at
|
||
once, and the lines cron mails include
|
||
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
||
- `report`, `trees` and `info` need only read access to the database (see
|
||
"Database"). Under the usual umask of `022` the first scan creates it readable
|
||
by every user, so an unprivileged user can run them against root's database.
|
||
|
||
### Reading the reports
|
||
|
||
Each row of `report` names two copies of one file, and each row of `trees` two
|
||
copies of one directory tree (see "Report output format" and "Trees output
|
||
format"). In a group of copies, the path that sorts first byte by byte is
|
||
`first` and every other path is a `dupe` of it. `first` says nothing about which
|
||
copy is the original or the oldest; which copy to keep is your choice.
|
||
|
||
A row is a candidate, not proof:
|
||
|
||
- The reports read only the database, so they show the files as of the last
|
||
scan; a file may have changed or gone since.
|
||
- A file of 100 MiB or more is compared only on samples of its content (see
|
||
"Duplicate detection").
|
||
- Paths that are hard links to one file are listed as duplicates, but they share
|
||
their data, so removing one frees nothing.
|
||
|
||
Compare a pair byte for byte before removing either copy. For the row
|
||
`/srv/a/big.iso`, `/srv/b/big-copy.iso`, `4294967296`:
|
||
|
||
```sh
|
||
cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
|
||
[ /srv/a/big.iso -ef /srv/b/big-copy.iso ] && echo "hard links"
|
||
```
|
||
|
||
`cmp` prints nothing and exits 0 only when every byte matches, and otherwise
|
||
reports where the files differ. The second line prints `hard links` when the two
|
||
paths are the same file, so removing either frees nothing.
|
||
|
||
A path holding a backslash, tab, newline or carriage return is escaped in the
|
||
reports (see "Report output format"). Undo the escapes before using it.
|
||
`printf '%b'` does exactly that, because every backslash in an escaped path
|
||
starts one of the four escapes. Command substitution drops trailing newlines, so
|
||
print an `x` after the path and remove it afterwards, or a path that ends in a
|
||
newline names a different file:
|
||
|
||
```sh
|
||
p="$(printf '%bx' '/srv/a/tab\tname.txt')"; p="${p%x}"
|
||
cmp "$p" /srv/b/tab-copy.txt
|
||
```
|
||
|
||
Check a `trees` row with `diff -r`, which compares the two trees file by file
|
||
and also names anything present in only one of them, such as an empty directory
|
||
or a symlink, which `trees` does not see.
|
||
|
||
## Rationale
|
||
|
||
Duplicate finders that hash every file in full do not scale to the target
|
||
environment. It holds about 10 million files and 150 TB, on disks that may be
|
||
slow or busy, such as a ZFS pool under resilver.
|
||
|
||
sfdupes reads a file under 100 MiB in full, and a larger one only at its two
|
||
ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
||
makes the comparison exact. A full read of every large file would take
|
||
unreasonably long. Sampling cuts a 150 GiB file to about 150 MiB of reads, at
|
||
the price of a comparison that is not exact.
|
||
|
||
The hashes are kept in a persistent database, so the expensive filesystem pass
|
||
is incremental. A rescan reads only the files that are new or whose size or
|
||
mtime changed. All analysis then works from the database alone.
|
||
|
||
The end goal is not individual files but whole duplicated trees, such as a
|
||
download extracted twice. An operator can consider removing such a tree as a
|
||
unit.
|
||
|
||
## Design
|
||
|
||
Goals, in order:
|
||
|
||
1. **Find whole duplicate trees, not just files.** The end goal is to identify
|
||
places where the exact same set of files and directories exists at two or
|
||
more paths (duplicate extractions, duplicate downloads, copied project
|
||
trees), so the operator can consider removing an entire subtree at once.
|
||
File-level duplicate detection is the foundation; tree-level detection is
|
||
built on top of it.
|
||
2. **Bound the I/O per file.** A file is read only when it is new or has
|
||
changed. A file under 100 MiB is then read in full, and a larger one only at
|
||
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
|
||
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
|
||
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
|
||
under resilver). Holding one small record (path, size, mtime) per file in
|
||
memory during a scan is acceptable; holding every file's hashes is not (they
|
||
stay in the database). The reporting commands do not hold every file's
|
||
hashes either: `report` reads the records in signature order and holds the
|
||
duplicate paths of one size at a time, so its memory grows with the
|
||
duplicates of the size that has the most of them, and `trees` reads the
|
||
records in path order and keeps each directory's path, digest and totals,
|
||
plus the hashes of only the files in the directories holding the record
|
||
being read, so its memory grows with the number of directories and with the
|
||
size of the largest directory.
|
||
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
|
||
maintains a persistent database, and an unchanged file is never read again
|
||
on a rescan. All analysis (`report`, `trees`) works from the database alone
|
||
and must never touch the scanned filesystem again. `scan` is designed to be
|
||
cronned; the reports run at any time against the last completed scan.
|
||
4. **Clean stream separation.** Everything on stdout is machine-readable data.
|
||
All progress, warnings, summaries, and help and usage text go to stderr.
|
||
Never mix them.
|
||
|
||
### Constraints
|
||
|
||
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
|
||
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
|
||
progress-bar library** (`github.com/schollz/progressbar/v3`),
|
||
`golang.org/x/term` to tell whether stderr is a terminal, **one SQLite
|
||
driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled), and
|
||
`golang.org/x/sys` for `flock(2)` (the scan lock, see "Database").
|
||
`github.com/spf13/viper` is permitted if configuration-file support is ever
|
||
needed, but is not currently used. No other third-party deps.
|
||
- Cross-compilation is not a concern. Builds run with cgo disabled (the
|
||
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
|
||
- Analysis modes (`report`, `trees`) must be deterministic: identical database
|
||
contents, identical output, regardless of the order in which records were
|
||
inserted.
|
||
|
||
### Subcommands
|
||
|
||
Four subcommands, all implemented:
|
||
|
||
1. `scan` — walk the filesystem and synchronize the database: one signature
|
||
record per regular file.
|
||
2. `report` — file-level duplicate report from the database.
|
||
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
|
||
from the database records, compute a Merkle-style digest per directory, and
|
||
report maximal groups of identical trees.
|
||
4. `info` — the database's details: its path, the sizes of its files, its
|
||
schema version, the number of its records and their total size.
|
||
|
||
```
|
||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||
sfdupes report > dupes.tsv
|
||
sfdupes trees > dupetrees.tsv
|
||
sfdupes info
|
||
sfdupes --version
|
||
sfdupes [command] --help
|
||
```
|
||
|
||
`--workers N` sets the size of each `scan` worker pool (default: the number of
|
||
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
|
||
operand's filesystem, and `--no-default-excludes` lets the walk enter the
|
||
default excludes, a list of paths; all three are described under "`scan` mode".
|
||
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
|
||
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
|
||
alone or after a subcommand, prints the help text to stderr and exits 0, writing
|
||
nothing to stdout.
|
||
|
||
### Database
|
||
|
||
All four subcommands operate on a single SQLite database file:
|
||
|
||
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
|
||
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
|
||
command-line flag. The path names the file exactly, whatever characters it
|
||
holds (`?`, `#` and `%` included); a relative path is relative to the working
|
||
directory.
|
||
- Each subcommand first prints one line on stderr naming the database it uses,
|
||
before anything else it prints (progress displays included), whether or not
|
||
stderr is a terminal. The line gives the path with the working directory put
|
||
in front of it when it is relative, and otherwise as set (a `..` in it stays,
|
||
so the line names the same file), escaped like the paths in a report (see
|
||
"Report output format"), and says where the path came from:
|
||
|
||
```
|
||
scan: new database /var/lib/sfdupes/db.sqlite (default)
|
||
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
||
```
|
||
|
||
The working directory put in front is the one `pwd -P` prints, with its
|
||
symlinks resolved. After a `cd` through a symlink to a directory, a path
|
||
starting `../` therefore names a file beside the directory the symlink leads
|
||
to, as it does for any other program. `scan` says `new database` when no
|
||
file is at that path as it starts. `report`, `trees` and `info` print the
|
||
line before they open the database, so the error for a missing database
|
||
comes after it.
|
||
|
||
- `scan` creates the database (and its parent directory) on first use. `report`,
|
||
`trees` and `info` require an existing database; a missing database file is a
|
||
fatal error (exit 1) telling the user to run `scan` first.
|
||
- Only one `scan` runs against a database at a time. For its whole run, `scan`
|
||
holds an exclusive `flock(2)` lock on a lock file beside the database, named
|
||
by appending `.lock` to the database's path with symlinks resolved
|
||
(`/var/lib/sfdupes/db.sqlite.lock` by default), taken before it walks the
|
||
filesystem or opens the database. Every path to one database, through a
|
||
symlink to the file or to a directory above it, names the same lock file. Just
|
||
before it takes the lock, `scan` creates the database file, empty, if it is
|
||
missing (through a symlink at the database path, it creates the symlink's
|
||
target), so the lock file is named the same way before the first scan sets up
|
||
the database as after. A second `scan` against the same database does not
|
||
wait: it fails at once with a one-line error naming the lock file and exits 1,
|
||
without walking anything or opening the database, and the running scan carries
|
||
on. The lock file is created on first use, open to its owner only, and left in
|
||
place: a leftover file blocks nothing, because the lock ends with the process
|
||
holding it however it ends, a fatal error or an interrupt included, and
|
||
deleting the file while a scan runs would let a second scan start. `report`,
|
||
`trees` and `info` never take the lock, so they run during a scan.
|
||
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
|
||
running a report while a cron `scan` is in progress is safe. The filesystem is
|
||
authoritative; the database is an eventually-consistent reflection of it.
|
||
Hashed records are committed in batched transactions while the scan is still
|
||
running (keeping the WAL small and letting concurrent reports observe
|
||
progress), so a report may see a scan's changes partially applied, and a scan
|
||
that dies partway leaves a valid database holding every batch committed so far
|
||
(an interrupted scan also commits the batch in progress, see "Error handling
|
||
and exit codes"); the next scan skips those records and converges toward the
|
||
filesystem.
|
||
- `scan` switches the database back to rollback-journal mode when it closes it,
|
||
so between scans the database file alone holds the whole database. Each switch
|
||
needs the database to itself: a `scan` that starts while a report is still
|
||
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
|
||
that ends while a report has the database open warns and leaves the database
|
||
in WAL mode until the next scan. `report` writes its rows while it reads the
|
||
records, so it is still reading while its output is paused (a pager, a stalled
|
||
pipe), and a `scan` started then fails after the busy timeout.
|
||
- A `scan` writes to the database and the files beside it at these points (its
|
||
phases are described under "`scan` mode"):
|
||
- As it starts, before the walk: the directory, an empty database file and
|
||
the `.lock` file, each only if missing.
|
||
- Then, still before the walk: the schema, in a new database, and the `-wal`
|
||
and `-shm` files, which stay beside the database while the scan runs.
|
||
- While it loads the existing records (`load`) and walks: no records. A
|
||
first scan's database holds none, however long its walk has run.
|
||
- During the hash phase: the first records, committed 10,000 at a time.
|
||
- In the update phase: the rest of the hashed records, and the deletions.
|
||
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
||
away, unless a report still has the database open (see above).
|
||
- `report`, `trees` and `info` open the database read-only and need only read
|
||
access to the database file, and no write access to its directory. While the
|
||
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
|
||
which SQLite creates with the database file's permissions.
|
||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
|
||
with any other version is a fatal error. `scan` creates the schema and sets
|
||
the version in one transaction, so a first scan stopped while doing so leaves
|
||
an empty database the next scan sets up. A database at version 0 that already
|
||
has a `files` table was therefore not made by sfdupes; every subcommand
|
||
refuses it with an error telling the user to remove the file and rescan):
|
||
|
||
```sql
|
||
CREATE TABLE files (
|
||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||
size INTEGER NOT NULL, -- bytes, from lstat
|
||
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
|
||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||
tail BLOB NOT NULL, -- raw 32-byte SHA-256 of the last 64 KiB
|
||
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
|
||
) WITHOUT ROWID;
|
||
CREATE INDEX files_signature ON files (size, head, tail, content);
|
||
CREATE TABLE totals (
|
||
records INTEGER NOT NULL, -- number of records in files
|
||
size INTEGER NOT NULL -- sizes of all records in files added up
|
||
);
|
||
INSERT INTO totals (records, size) VALUES (0, 0);
|
||
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
||
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
||
END;
|
||
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
||
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
||
END;
|
||
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
||
UPDATE totals SET size = size + NEW.size - OLD.size;
|
||
END;
|
||
```
|
||
|
||
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
|
||
UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond:
|
||
`mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the
|
||
nanoseconds past that second. Split this way they hold any mtime a
|
||
filesystem can record, one before 1678 or after 2262 included, which a
|
||
single 64-bit count of nanoseconds cannot. They are used only for change
|
||
detection and are not part of the duplicate key. Every record holds the
|
||
three hashes that "Duplicate detection" below describes. The
|
||
`files_signature` index lets `report` read the records in signature order
|
||
without sorting the table.
|
||
|
||
`totals` holds one row: the number of records in `files` and the sum of
|
||
their sizes. Three triggers on `files` keep that row current. Each runs in
|
||
the transaction of the change to `files` that fires it, so the row and the
|
||
records always agree. `info` reads both numbers from this row. `scan`,
|
||
`report` and `trees` read the number of records from it as the total of
|
||
their progress displays. Counting the records in `files` would read every
|
||
record, which takes minutes on a database of tens of millions. The triggers
|
||
make each write to `files` slower, so a first scan of many small files takes
|
||
about two fifths longer. On large files, where reading them takes most of
|
||
the time, the difference is small.
|
||
|
||
### Duplicate detection
|
||
|
||
Two files are duplicates when they have the same size and the same three hashes.
|
||
`scan` computes the hashes and stores them in the database. `report` and `trees`
|
||
group the files whose size and hashes all match, whichever scan recorded them.
|
||
|
||
`scan` gives every file it records three SHA-256 hashes:
|
||
|
||
- `head`, the hash of the file's first 64 KiB.
|
||
- `tail`, the hash of the file's last 64 KiB.
|
||
- `content`, the content hash described below.
|
||
|
||
In a file of 64 KiB or less, the first and the last 64 KiB are both the whole
|
||
file, so all three hashes are the hash of the whole file.
|
||
|
||
The content hash depends on the file's size:
|
||
|
||
- Under 100 MiB, it is the SHA-256 of the whole file. Agreement on it is proof
|
||
of identical content, barring a SHA-256 collision.
|
||
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
|
||
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
|
||
GiB and so on. The samples are fed into one hash in order. A sample that would
|
||
run past the end of the file stops there.
|
||
|
||
The sampled hash is **deliberately probabilistic**. Bytes outside the samples
|
||
and outside the first and last 64 KiB are never read. Two large files that
|
||
differ only in such bytes are reported as duplicates. That is the price of never
|
||
reading a 150 GB file end to end.
|
||
|
||
Only files of the same size are compared, so their samples always lie at the
|
||
same offsets. A file under 100 MiB and one of 100 MiB or more never share a
|
||
size, so a whole-file hash is never compared with a sampled one.
|
||
|
||
`scan` reads a file only to hash it, and only when the file is new or has
|
||
changed since its record was written (see "`scan` mode").
|
||
|
||
### `scan` mode
|
||
|
||
`scan` requires one or more `PATH` operands naming the trees to scan. There is
|
||
no default path; invoking `scan` with no operand is a usage error (usage message
|
||
on stderr, exit 2). An operand may be a directory or a regular file; an operand
|
||
that does not exist is a fatal error (exit 1). Because database records persist
|
||
between runs and are keyed by absolute path, each operand is resolved to an
|
||
absolute, lexically cleaned path (symlinks are not resolved) before walking, so
|
||
results do not depend on the working directory. All operands belong to a single
|
||
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
||
pool. Overlapping operands are harmless — an operand that duplicates another or
|
||
lies under another is dropped before walking, so every file is reached exactly
|
||
once and produces one database record. An operand under another one is kept,
|
||
though, and walked on its own when any directory from it (from its directory,
|
||
for a regular file) up to but not including the operand covering it is a default
|
||
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
|
||
`-x`, is on a different filesystem than that operand, because that operand's
|
||
walk does not enter such a directory and so does not reach it.
|
||
|
||
An operand is not scanned when it is:
|
||
|
||
- a symlink, which `scan` never follows, not even as an operand;
|
||
- a socket, FIFO or device node;
|
||
- at or inside a directory named `.zfs`.
|
||
|
||
An operand reached through a symlink into a `.zfs` directory is not scanned
|
||
either.
|
||
|
||
This holds even when the operand lies under another operand. `scan` prints a
|
||
one-line warning naming the path and what it is, counts it as skipped, and drops
|
||
it from the scanned operands before reading the database. An operand beneath a
|
||
symlink operand is still scanned. The records stored beneath the dropped operand
|
||
are not deleted: they are treated like any other record outside the scanned
|
||
operands. If it lies under another operand, they are under that operand instead,
|
||
and are treated like any other record there. This is not an error: a scan whose
|
||
every operand is dropped walks nothing and exits 0. An operand that is not on a
|
||
persistent filesystem (see "Rules for the walk") is dropped the same way. The
|
||
line `scan` prints for it is not a warning, though, and it is not counted as
|
||
skipped.
|
||
|
||
`scan` synchronizes the database with the filesystem state under the scanned
|
||
operands:
|
||
|
||
- A file not yet in the database is hashed and inserted.
|
||
- A file already in the database is **skipped without reading its contents**
|
||
when its lstat size equals the recorded size and its lstat mtime is not newer
|
||
than the recorded mtime. This is what makes a daily rescan cheap.
|
||
- A file whose mtime is newer than recorded, or whose size differs, is hashed
|
||
again. Change detection compares the mtime to the nanosecond, as finely as the
|
||
filesystem records it, so a same-size rewrite counts as a change whenever the
|
||
filesystem gives it a later mtime than recorded, even within the same second.
|
||
- A database record whose path lies under one of the scanned operands but was
|
||
not successfully processed this run is deleted. This removes records for
|
||
deleted files. It also removes records for paths that failed to stat or hash
|
||
this run: the database only ever contains signatures verified by the most
|
||
recent scan that covered them (a subsequent successful scan re-adds such
|
||
files).
|
||
- Database records outside the scanned operands are untouched, so disjoint trees
|
||
can be scanned on different schedules into the same database.
|
||
- A record at or below a path the walk left out counts as outside the scanned
|
||
operands, so it is not deleted. The walk leaves out a path that:
|
||
- is a default exclude;
|
||
- is not on a persistent filesystem;
|
||
- with `-x`, is a directory on another filesystem than its operand.
|
||
|
||
A filesystem mounted under an operand can therefore be scanned on its own
|
||
schedule into the same database. Records below an operand at or inside such
|
||
a path are not kept this way, since that operand's own walk verifies them.
|
||
|
||
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
|
||
inside the walk and hash phases, and batched database writes begin during the
|
||
hash phase:
|
||
|
||
1. **load** — read the existing records. Each record under the operands is kept
|
||
in memory, by path, with its size and its mtime, for the walk to compare
|
||
each file with.
|
||
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||
with the walk worker pool: every operand seeds the shared queue, and each
|
||
worker reads one directory at a time, handing discovered subdirectories back
|
||
to the queue and running `lstat` on each regular file as it is discovered
|
||
(while the directory's metadata is still hot). Sequential directory
|
||
enumeration is metadata-latency-bound and takes hours at tens of millions of
|
||
files; per-directory parallelism is what makes the walk tractable on large
|
||
or busy pools. The walk compares each file with its record as it goes, and
|
||
an unchanged file needs nothing more. Every new or changed file is carried
|
||
to the hash phase as a (path, size, size on disk, mtime) record.
|
||
3. **hash** — the worker pool hashes every carried file as described in
|
||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||
`content`. Zero-length files have constant hashes and are never opened.
|
||
Files are hashed in **inode order**, which minimizes seeks on spinning
|
||
disks. Paths that are hard links to the same inode are **read once**, and
|
||
all share the one result. A hard-link backup farm therefore costs one read
|
||
per inode, not per path. The phase total counts actual reads, so its
|
||
progress and its time remaining are meaningful. Completed records are
|
||
committed in batched transactions **while hashing runs**. A scan interrupted
|
||
after hours therefore keeps everything hashed so far, and the next scan
|
||
resumes cheaply, skipping records already written.
|
||
4. **update** — commit the final partial batch of hashed records. Then delete
|
||
the records the scan did not verify: those of vanished files, and of paths
|
||
that failed to stat or hash.
|
||
|
||
Rules for the walk:
|
||
|
||
- Only regular files. Skip directories, symlinks (do not follow, including
|
||
symlink operands), sockets, FIFOs, and device nodes. An operand that is a
|
||
symlink, socket, FIFO, or device node is dropped as described in "`scan` mode"
|
||
above.
|
||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
|
||
them would list every file once per snapshot), not even when it is an operand;
|
||
such an operand is dropped the same way.
|
||
- Never record the scan's own database. Once the database and its `.lock` file
|
||
are open, `scan` notes the device and inode of the database file, its `-wal`
|
||
and `-shm` files and its `.lock` file, and the walk passes over a file with
|
||
one of them, however it reaches it: as an operand, below one, or through a
|
||
symlinked directory. It prints no line for such a file and does not count it
|
||
as skipped. A record that an earlier scan left for one of them is deleted,
|
||
like the record of any file the scan no longer verifies. There is no
|
||
`-journal` file during the walk, because the database is in WAL mode then (see
|
||
"Database").
|
||
- Filesystem boundaries are crossed by default. With `-x` (long form
|
||
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
|
||
into a directory on a different filesystem than its `PATH` operand; each
|
||
operand is bounded by its own filesystem. The records below such a directory
|
||
count as outside the scanned operands, as described in "`scan` mode" above:
|
||
they are not deleted. That does not hold below an operand at or inside that
|
||
directory.
|
||
- Never record anything that is not on a persistent filesystem. A filesystem is
|
||
not persistent when the kernel makes it up or keeps it in memory: one of type
|
||
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
|
||
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
|
||
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
|
||
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
|
||
it names the database and before it takes the lock, `scan` lists the mounted
|
||
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
|
||
of mounted filesystems on macOS (on any other system it lists none), and notes
|
||
the device ID of each one of those types. On Linux that is the device ID the
|
||
list gives, so a filesystem that a later mount hides still gives its own. On
|
||
macOS it is read with `lstat` on the mount point, and where one filesystem is
|
||
mounted over another on the same mount point, the one mounted last counts. If
|
||
`scan` cannot list them, it fails (exit 1). The walk then never enters a
|
||
directory, and never records a file, on one of those devices, however it
|
||
reaches it: as an operand, below one, or through a bind mount, which keeps the
|
||
device ID of what it binds. No flag turns this off. For each operand,
|
||
directory or file it leaves out this way, `scan` prints a line on stderr
|
||
naming it and the type of its filesystem. It prints that line as it prints a
|
||
warning, so the line never lands inside a progress display:
|
||
|
||
```
|
||
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
|
||
```
|
||
|
||
That line is not a warning, and the path is not counted as skipped. An
|
||
earlier scan's records at or below it count as outside the scanned operands,
|
||
as described in "`scan` mode" above: they are not deleted. That does not
|
||
hold below an operand at or inside it.
|
||
|
||
- Below an operand, never descend into a default exclude: a path that holds the
|
||
kernel's view of the running system, or temporary files, so that
|
||
`sfdupes scan /` walks only the files worth comparing. The default excludes
|
||
depend on the system, and `sfdupes scan --help` lists those of the system it
|
||
runs on:
|
||
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
|
||
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
||
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
|
||
everything in `/System/Volumes/Data` through the firmlinks, so walking it
|
||
as well would record each of those files twice.
|
||
- On Windows, relative to the root of the drive an operand lies on: the
|
||
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
|
||
`hiberfil.sys`, and the directories `System Volume Information`,
|
||
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
|
||
matches one path component, as it does in any default exclude. sfdupes
|
||
does not build for Windows yet, so this list is not used.
|
||
- On any other system there are none.
|
||
|
||
`--no-default-excludes` turns them all off; what is not on a persistent
|
||
filesystem stays left out. An operand at or inside a default exclude is
|
||
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
|
||
persistent filesystem. For each default exclude its walk does not enter,
|
||
`scan` prints a line on stderr naming it, as it prints a warning:
|
||
|
||
```
|
||
scan: skipped /var/tmp (default exclude)
|
||
```
|
||
|
||
That line is not a warning, and the directory is not counted as skipped. A
|
||
default exclude that is not on a persistent filesystem, as `/proc` is, gets
|
||
the line for its filesystem instead. One that is itself an operand gets no
|
||
line, since it is walked as that operand. When the walk does not enter a
|
||
default exclude, the records below it count as outside the scanned operands,
|
||
as described in "`scan` mode" above: they are not deleted. That does not
|
||
hold below an operand at or inside it.
|
||
|
||
- On a per-path error, such as permission denied or an unreadable file: print a
|
||
one-line warning to stderr, skip the path, and continue. Per-file errors never
|
||
abort the run; the final summary reports how many were skipped. As specified
|
||
above, a skipped path that has a database record from an earlier scan loses
|
||
that record. An unreadable directory subtree likewise loses its records. This
|
||
is accepted: the database mirrors what the latest scan could actually verify.
|
||
The exception is an operand dropped before the database was read, whose record
|
||
counts as outside the scanned operands.
|
||
- A file or directory removed while the scan runs is not an error. The scan
|
||
prints no line for it and does not count it as skipped. A path gone when the
|
||
walk reaches it gets no record. A file gone when the hash phase opens it gets
|
||
none either. The records an earlier scan left at or below such a path are
|
||
deleted.
|
||
|
||
Concurrency: the walk phase, which also stats files, and the hash phase each use
|
||
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
|
||
parallelizes across directories, and hashing across files. `--workers` must be
|
||
at least 1: a smaller value is a usage error, reported in one line on stderr
|
||
with exit 2 before anything is scanned. Both phases are seek-bound on spinning
|
||
disks, so raising `--workers` well past the core count can help on pools with
|
||
many spindles. The main goroutine owns the database writes and the progress
|
||
counts. Each progress display draws the counts from a goroutine of its own, and
|
||
must never block the workers.
|
||
|
||
`scan` writes nothing to stdout. The summary line on stderr reports the files
|
||
seen this run broken down by disposition, plus skips:
|
||
|
||
```
|
||
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
|
||
```
|
||
|
||
(`removed` counts deleted database records, which are not part of the files-seen
|
||
total.)
|
||
|
||
### `report` mode
|
||
|
||
`report` reads every record from the database and takes no positional arguments.
|
||
|
||
**`report` must never touch the filesystem being analyzed.** It does not stat,
|
||
open, or otherwise access any path that appears in the records; its only I/O is
|
||
reading the database and writing stdout/stderr. `report` must produce identical
|
||
output whether or not the scanned filesystem is still mounted.
|
||
|
||
Processing:
|
||
|
||
- Group the records by the key `(size, head, tail, content)`.
|
||
- Every group with two or more paths is a duplicate group.
|
||
- Within each group, sort paths lexicographically (byte order). The first path
|
||
is the group's `first`; every other path is a `dupe`.
|
||
- Order groups by size descending (biggest reclaimable space first), tie-broken
|
||
by `first` path ascending. Output must be fully deterministic for a given
|
||
database state.
|
||
|
||
`report` runs in two phases, each with its own progress display on stderr (see
|
||
"Progress"):
|
||
|
||
- `open` opens the database.
|
||
- `read` gets the number of records from `totals`, then reads the records in
|
||
signature order, largest size first.
|
||
|
||
Signature order, largest size first, is the `files_signature` index read
|
||
backwards, so SQLite sorts nothing and returns the first record at once. The
|
||
records sharing a signature come together, and so do the records of one size, so
|
||
`report` holds only the duplicate groups of the size it is reading, and writes
|
||
them, ordered by `first` path, once it reaches a smaller size. Its memory holds
|
||
at most the duplicate paths of one size at a time, never every record.
|
||
|
||
#### Report output format
|
||
|
||
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
|
||
group of N):
|
||
|
||
```
|
||
first dupe size
|
||
/srv/a/big.iso /srv/b/big-copy.iso 4294967296
|
||
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296
|
||
```
|
||
|
||
Paths are raw bytes and may hold any byte except NUL, so the path columns
|
||
(`first` and `dupe`) are escaped to keep every row one line of tab-separated
|
||
fields: a backslash is written as `\\`, a tab as `\t`, a newline as `\n`, and a
|
||
carriage return as `\r`. Every other byte is written unchanged, including bytes
|
||
that are not valid UTF-8. Undoing those four escapes gives back the stored path.
|
||
Grouping and ordering use the stored path, not the escaped one. The warnings
|
||
`scan` prints on stderr are escaped the same way, so each warning is one line.
|
||
|
||
Summary to stderr: records read, number of duplicate groups, number of dupe
|
||
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||
units.
|
||
|
||
### `trees` mode
|
||
|
||
`trees` reads the same database as `report` (no positional arguments) and
|
||
reports **entire duplicate directory trees**: directories under which the exact
|
||
same set of relative paths exists with the exact same file signatures.
|
||
|
||
**`trees` must never touch the filesystem being analyzed** — the same rule as
|
||
`report`. The directory hierarchy is reconstructed purely from the paths in the
|
||
records, split on `/`.
|
||
|
||
Definitions:
|
||
|
||
- A file's **signature** is `(size, head, tail, content)` — mtime is
|
||
informational and excluded.
|
||
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
|
||
serialize the directory's child entries — for a file child, its name and
|
||
signature; for a subdirectory child, its name and that subdirectory's digest —
|
||
sort the serialized entries byte-lexicographically, and hash the
|
||
concatenation. Names are part of the digest: two trees whose files differ only
|
||
in name are _not_ duplicates.
|
||
- Two directories are **duplicate trees** when their digests are equal. Equal
|
||
digests imply equal recursive file count and equal total byte size.
|
||
|
||
Known limitation (accepted): hard-linked paths are reported as duplicates by
|
||
`report` and count toward duplicate trees — their content is genuinely identical
|
||
— even though they share storage, so removing one reclaims no space. Inode
|
||
identity is used during the scan to avoid redundant reads but is not persisted
|
||
in the database.
|
||
|
||
Known limitation (accepted): only regular files that appear in the database
|
||
define a tree. Empty directories are invisible, and a file skipped during the
|
||
scan (e.g. permission error) in one copy but not the other will make
|
||
otherwise-identical trees compare as different.
|
||
|
||
Processing:
|
||
|
||
- Build the hierarchy, compute every directory's digest, and group directories
|
||
by digest. Every group with two or more directories is a duplicate-tree group.
|
||
- **Report only maximal trees.** A group is suppressed when its members' parents
|
||
are pairwise distinct directories that all share a single digest — such a
|
||
group is wholly implied by its parents' (or a further ancestor's) group.
|
||
Groups containing sibling directories, or members whose parents differ, are
|
||
always reported.
|
||
- Within each group, sort paths lexicographically (byte order); the first path
|
||
is `first`, every other path is a `dupe`.
|
||
- Order groups by total tree size descending, tie-broken by `first` path
|
||
ascending. Output must be fully deterministic for a given input.
|
||
|
||
`trees` runs in four phases, each with its own progress display on stderr (see
|
||
"Progress"):
|
||
|
||
- `open` opens the database.
|
||
- `read` gets the number of records from `totals`, then reads the records in
|
||
path order, building the hierarchy and computing the digests as it goes.
|
||
- `group` groups the directories by digest and picks the maximal groups.
|
||
- `write` writes the groups to stdout.
|
||
|
||
#### Trees output format
|
||
|
||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||
group of N). After the `first` and `dupe` paths come three numbers:
|
||
|
||
- `files`, the number of regular files in one copy of the tree, counted
|
||
recursively;
|
||
- `size`, the bytes of one copy, its files' sizes added up;
|
||
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||
added up.
|
||
|
||
```
|
||
first dupe files size disk_size
|
||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||
```
|
||
|
||
A file's `disk_size` is the space its blocks take on disk. It is less than the
|
||
file's size on a compressed or sparse file, and a little more on most other
|
||
files. Two copies of one tree can therefore differ in `disk_size`. When a file
|
||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||
full. Removing it from one copy then frees nothing, as the known limitation
|
||
under "`trees` mode" says.
|
||
|
||
The `first` and `dupe` paths are escaped as described under "Report output
|
||
format". The root directory's path is `/`.
|
||
|
||
Summary to stderr: records read, number of duplicate-tree groups, number of dupe
|
||
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||
units.
|
||
|
||
### `info` mode
|
||
|
||
`info` takes no positional arguments and prints the database's details on
|
||
stdout, one per line: a name, a tab, and the value. It opens the database the
|
||
way `report` does, read-only and without the scan lock, so it runs during a scan
|
||
and needs only read access (see "Database"). Like `report`, it never touches the
|
||
scanned filesystem.
|
||
|
||
```
|
||
database /var/lib/sfdupes/db.sqlite (default)
|
||
database file size 2147483648 (2.0 GiB)
|
||
-wal file size 4124152 (3.9 MiB)
|
||
modified 2026-10-16T12:02:50Z
|
||
schema version 1
|
||
records 12345678
|
||
size of recorded files 150000000000000 (136.4 TiB)
|
||
```
|
||
|
||
- `database` is the path and where it came from, as on the line every subcommand
|
||
prints first on stderr (see "Database").
|
||
- `database file size` and `-wal file size` are in bytes, then in human units.
|
||
The `-wal` line is there only when a `-wal` file is beside the database, as
|
||
while a scan runs.
|
||
- `modified` is the time the database file was last changed, in UTC. While the
|
||
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
||
- `schema version` is the database's `PRAGMA user_version`. A database with
|
||
another version than this build's is refused, as by `report`.
|
||
- `records` is the number of records. `size of recorded files` is the sum of the
|
||
sizes of all records, in bytes, then in human units.
|
||
|
||
`info` runs in two phases, each with its own progress display on stderr (see
|
||
"Progress"):
|
||
|
||
- `open` opens the database.
|
||
- `count` reads the schema version and the row of `totals` (see "Database").
|
||
Both reads run in one read-only transaction, so the numbers agree with each
|
||
other while a scan is committing.
|
||
|
||
`info` writes its lines once the `count` display has ended, so on a terminal
|
||
they come after the displays.
|
||
|
||
### Progress
|
||
|
||
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
|
||
`info`; rendering in the style of `pv` is the model. All progress goes to
|
||
stderr.
|
||
|
||
`scan`, `report`, `trees` and `info` must always show that they are working and
|
||
not hung. Each phase gets its own display from the moment it starts. The display
|
||
names the phase and its place among the command's phases. It shows the phase's
|
||
progress, its elapsed time and its time remaining.
|
||
|
||
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
||
`[4/4] update`):
|
||
|
||
- `load` gets the number of records from `totals` first, then reads the records
|
||
against that number. Until it has the number, the display says it is counting
|
||
the records.
|
||
- `walk` has no exact total. Its estimate is the number of records the last scan
|
||
left under the operands, known once `load` is done and shown with a `~`. The
|
||
time remaining is unknown once the walk passes the estimate, and when there
|
||
are no such records, as on a first scan.
|
||
- `hash` and `update` have exact totals. Only files that actually need hashing
|
||
appear in the hash total, so its time remaining is meaningful.
|
||
|
||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||
`[4/4] write`, see "`trees` mode"):
|
||
|
||
- `open` has no total. While it opens the database, waiting on a busy one
|
||
included, the display says it is opening the database.
|
||
- `read` gets the number of records from `totals` first, then reads the records
|
||
against that number. Until it has the number, the display says it is counting
|
||
the records.
|
||
- `group` has the directories `read` found as its total, and `write` the
|
||
duplicate-tree groups.
|
||
|
||
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
|
||
|
||
- `open` has no total. While it opens the database, waiting on a busy one
|
||
included, the display says it is opening the database.
|
||
- `read` gets the number of records from `totals` first, then reads the records
|
||
against that number, writing the rows to stdout as it goes. Until it has the
|
||
number, the display says it is counting the records.
|
||
|
||
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
||
|
||
- `open` has no total. While it opens the database, waiting on a busy one
|
||
included, the display says it is opening the database.
|
||
- `count` has no total either, because it reads one row: the display says it is
|
||
counting the records until it has the row, and its time remaining is unknown.
|
||
|
||
Each display is one line holding:
|
||
|
||
- the phase and its place among the phases
|
||
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
||
has a total, the items processed, or what the phase is waiting for
|
||
- current rate (items/s), naming the items the phase counts: files, records,
|
||
directories or groups. It covers the items completed over the last 5 seconds,
|
||
or, when stderr is not a terminal, since the line before, so it falls to 0
|
||
while nothing completes, and a burst of items after a wait shows as a burst
|
||
rather than as a steady rate. Once no item has completed for 10 seconds, the
|
||
word `stalled` and where the phase waits take the rate's place (see below).
|
||
- elapsed time, labelled `elapsed`
|
||
- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`.
|
||
It comes from the phase's average rate since it started counting, so that it
|
||
does not swing with each burst.
|
||
|
||
Example lines, as printed when stderr is not a terminal:
|
||
|
||
```
|
||
[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
|
||
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
|
||
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s
|
||
```
|
||
|
||
When no item has completed for 10 seconds, as while one large file is read, a
|
||
disk is slow or the database is busy, the line says `stalled` and where the
|
||
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
|
||
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
|
||
may wait on either. `walk` says `disk`. `hash` says `disk/db`, as it reads files
|
||
and writes their records to the database between them. `load`, `update` and the
|
||
`read` phase of `trees` say `db`. `trees`' `write` says `out`, and its `group`,
|
||
which works in memory, says `cpu`. The `read` phase of `report` writes the
|
||
report while it reads the records, so it says `db/out`. A phase still waiting
|
||
for its total already says what it waits for, and says nothing more.
|
||
|
||
Additional requirements:
|
||
|
||
- On a terminal, every display starts with the progress-bar library's spinner in
|
||
its first column, followed by its line. Each display is drawn again several
|
||
times a second, so its spinner turns and its elapsed time keeps moving while a
|
||
phase waits for its next item.
|
||
- On a terminal, a phase with an exact total ends its line with its percent and
|
||
a bar 10 columns wide. This percent takes the place of the line's own. The bar
|
||
stays at the terminal's right edge, so it does not move as the line grows or
|
||
shrinks. A phase with nothing to do shows no bar. Every phase has an exact
|
||
total once it has counted what it will do, except these:
|
||
- `scan`'s `walk`;
|
||
- the `open` phase of `report` and `trees`;
|
||
- both phases of `info`.
|
||
|
||
On a terminal 100 columns wide:
|
||
|
||
```
|
||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||
```
|
||
|
||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||
terminal is. A line too wide for the terminal first loses whichever of these
|
||
it shows: its rate, the words of a stalled phase, or what the phase is waiting
|
||
for. The other four parts are kept whole:
|
||
- the phase and its place;
|
||
- the progress;
|
||
- the elapsed time;
|
||
- the time remaining.
|
||
|
||
The line loses its bar only when those four and the bar do not fit. It is
|
||
cut short at the terminal's edge only when those four alone are too wide,
|
||
and then the end of the time remaining is lost first. That happens hours
|
||
into a scan of millions of files, on a terminal narrower than about 75
|
||
columns. On a terminal 100 columns wide, the stalled line of the example
|
||
above keeps its words and its bar. The next draw fits the line to a terminal
|
||
made narrower while a phase runs. It erases the old line within the new
|
||
width, and the display goes on redrawing in place.
|
||
|
||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||
runs (whether or not an item has completed, and never more often), and once
|
||
more as the phase ends.
|
||
- Progress must be non-blocking with respect to the worker pool: the main
|
||
goroutine counts the items, and each display draws them from a goroutine of
|
||
its own.
|
||
- A warning printed during a phase always lands on a line of its own, never
|
||
inside the progress display. So does every line written to stdout when stdout
|
||
and stderr are the same file (the same device and inode): one terminal, as in
|
||
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
|
||
sent to. When they are not, stdout is written without waiting on the display,
|
||
and the display keeps being drawn while a slow reader of stdout holds up a
|
||
write.
|
||
- A phase that stops short of its total, as an interrupted one does, is left
|
||
showing the count it stopped at rather than filled up.
|
||
|
||
### Error handling and exit codes
|
||
|
||
- `0`: success, even if individual files were skipped with warnings.
|
||
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
|
||
already running against the same database, the database cannot be
|
||
created/opened/read/written, a missing database for `report`/`trees`/`info`,
|
||
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
|
||
below).
|
||
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
|
||
`--workers` below 1, and `report`/`trees`/`info` with any positional
|
||
argument).
|
||
|
||
A stdout write failure, such as a full disk, is reported in one line on stderr
|
||
and exits 1. Two cases never reach sfdupes as a failed write:
|
||
|
||
- When the reader of a stdout pipe exits early, as in `sfdupes report | head`,
|
||
the next write ends sfdupes with `SIGPIPE`, quietly and without a summary, the
|
||
way `cat` or `sort` end. The shell reports the signal (status 141 in most
|
||
shells), not exit 1.
|
||
- When stdout is closed outright (`sfdupes report >&-`), the Go runtime opens
|
||
`/dev/null` in its place before sfdupes starts, so the output is discarded and
|
||
the run succeeds, as with `> /dev/null`.
|
||
|
||
`scan` stops cleanly on `SIGINT` (Ctrl-C) or `SIGTERM`. Its workers stop taking
|
||
work, each finishing at most the directory listing or file it is reading; the
|
||
progress display is finished; and the records it has hashed but not yet
|
||
committed are committed, so the next scan does not hash them again. Apart from
|
||
that commit it starts no further writes or deletions: records are deleted only
|
||
after a complete walk, so those under paths an interrupted walk never reached
|
||
are kept. The database is closed and the lock released as on any other exit, the
|
||
line `scan: interrupted after N files` goes to stderr, N being the number of
|
||
files the walk reached, and the exit code is 1. The next scan skips the records
|
||
already written and converges as usual.
|
||
|
||
After the first signal `scan` stops catching them, so a second one ends it at
|
||
once, as an uncaught signal does: the records not yet committed are lost, and
|
||
the database is left valid, as when any scan dies (see "Database"). A `SIGINT`
|
||
that `scan` inherits as ignored, as a script's background job does, stays
|
||
ignored.
|
||
|
||
## Entrypoints
|
||
|
||
This repository adheres to the
|
||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||
standard: the normalized executables in `script/` are the entrypoints for the
|
||
development workflow, and the `Makefile` targets are thin shims that call them.
|
||
Every script is POSIX `sh`, resolves the repository root itself so it can be run
|
||
from any working directory, and may be invoked directly. The provided
|
||
entrypoints are:
|
||
|
||
- `script/bootstrap` — install everything needed to build and develop this
|
||
repository, idempotently, assuming nothing is present. `git`, `make`, and `go`
|
||
come from the first of nix, apt, brew, or apk found on the host, and are
|
||
presence-checked only. An installed node is used when its major version is 22;
|
||
otherwise node 22.17.0 is installed through nvm, which comes from a release
|
||
archive whose sha256 the script checks, and yarn 1.22.22 is activated under it
|
||
through corepack. With an installed node 22, a yarn already on `PATH` is used
|
||
as it is; otherwise yarn 1.22.22 is activated through corepack, or installed
|
||
with `npm` when there is no corepack. `yarn install --frozen-lockfile` then
|
||
installs the prettier that `package.json` and `yarn.lock` pin. `golangci-lint`
|
||
is never installed: it runs in Docker (see `script/lint`). `docker` is not
|
||
installed either; testing, linting and the image build need it. Ends with
|
||
`go mod download`.
|
||
- `script/setup` — make a fresh clone ready for development: runs
|
||
`script/bootstrap`, then `script/install-precommit`.
|
||
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
|
||
need the name call it, so they stay identical across repositories.
|
||
- `script/test` — run the test suite under the race detector, with a 90-second
|
||
timeout and coverage enabled, rerunning verbosely on failure so the logs show
|
||
which test failed. It builds the `Dockerfile`'s `test` phase alone and writes
|
||
no image. The race detector needs cgo and a C compiler, which the build never
|
||
uses, so the phase starts from a digest-pinned Debian `golang` image, which
|
||
has `gcc`. The tests run as `nobody`, because several of them make a file
|
||
unreadable and root reads it anyway.
|
||
- `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase
|
||
alone and writes no image: in the digest-pinned `golangci/golangci-lint`
|
||
image, the gofmt check, `golangci-lint config verify` and `golangci-lint run`
|
||
run as build steps, so a successful build is a clean lint. The linter is never
|
||
run on the host, which makes a working `docker` the one prerequisite for
|
||
linting.
|
||
- `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every
|
||
Markdown file with prettier, at the settings in `.prettierrc` (4-space
|
||
indents, prose wrapped at 80 columns). Both run on the host, prettier through
|
||
yarn at the version `yarn.lock` pins. When the node on `PATH` is not node 22,
|
||
this loads the node 22.17.0 that `script/bootstrap` installed through nvm.
|
||
- `script/fmt-check` — the read-only counterpart of `script/fmt`: prints any
|
||
unformatted file and exits non-zero instead of writing. gofmt and prettier
|
||
both run every time, and each names itself when it fails. The `Dockerfile`'s
|
||
`lint` phase runs the same gofmt check.
|
||
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
|
||
that order. Modifies nothing. The first two need `docker`, the third what
|
||
`script/bootstrap` installs.
|
||
- `script/docker` — build the Docker image, tagged with the name from
|
||
`script/projectname`, passing the output of
|
||
`git describe --tags --always --dirty` (or `unknown` when that is empty) as
|
||
the `VERSION` build argument. The `Dockerfile`'s build stage depends on its
|
||
`lint` and `test` phases, so this also runs the tests and the linter.
|
||
- `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the
|
||
image as `script/docker` does. This is what the Gitea workflow runs on push; a
|
||
successful run means every check passed. The tests and the linter run twice:
|
||
once in `script/check`, and again as stages of the image build.
|
||
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
|
||
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
|
||
`script/check`.
|
||
- `script/install-precommit` — install the git pre-commit hook, as
|
||
`.git/hooks/pre-commit`, that runs `script/precommit`.
|
||
|
||
Every `docker build` in `script/` passes a new `CACHEBUST` build argument. In
|
||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
||
that install system packages and download the Go modules, or right after `FROM`
|
||
in a stage with neither, and every step that lints, tests or compiles comes
|
||
after it, so every `RUN` step after it runs again each time. On an unchanged
|
||
tree Docker would otherwise serve the gate steps from its cache, and the build
|
||
would pass having run no test and no linter. The layers above that line stay in
|
||
Docker's build cache until the base image, their own lines, `go.mod` or `go.sum`
|
||
change, or until that cache is cleared; only then are the packages installed and
|
||
the modules downloaded again. Go's build cache is kept between builds on the
|
||
steps that compile, so only what changed is compiled again; the tests run with
|
||
`-count=1`, so none reports a stored result. `golangci-lint run` also keeps
|
||
golangci-lint's cache, and takes a package's findings from it only when the
|
||
package's files and those of the packages it imports, `go.mod`, the linters and
|
||
their settings in `.golangci.yml`, and the golangci-lint version are all
|
||
unchanged.
|
||
|
||
## Build
|
||
|
||
The `script/` entrypoints above are where the implementations live; the
|
||
`Makefile` targets are shims onto them, except `build`, which carries the
|
||
compile recipe:
|
||
|
||
- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
|
||
the default target.
|
||
- `make bootstrap` — install the build and development dependencies.
|
||
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
|
||
- `make test` — run the test suite under the race detector, in Docker (90-second
|
||
timeout; reruns with `-v` on failure; see `script/test`); requires `docker`.
|
||
- `make lint` — run `golangci-lint` with the repo config and the gofmt check, in
|
||
Docker (see `script/lint`); requires `docker`.
|
||
- `make fmt` / `make fmt-check` — format the Go sources and the Markdown /
|
||
verify formatting without writing, on the host; requires `go` and what
|
||
`make bootstrap` installs (see `script/fmt`).
|
||
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
|
||
`docker` and what `make bootstrap` installs.
|
||
- `make docker` — build the Docker image, which runs the gates as build stages.
|
||
- `make hooks` — install the pre-commit hook.
|
||
- `make clean` — remove the binary.
|
||
|
||
### Definition of done
|
||
|
||
All of the following, run in this directory, must pass:
|
||
|
||
1. `make check` passes (tests, lint, `gofmt`, prettier).
|
||
2. `make docker` succeeds.
|
||
3. Smoke test — create a throwaway tree in a temp dir (never test against real
|
||
data):
|
||
|
||
```sh
|
||
d=$(mktemp -d)
|
||
export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
|
||
mkdir -p "$d/a" "$d/b"
|
||
head -c 2000 /dev/urandom > "$d/a/one.bin"
|
||
cp "$d/a/one.bin" "$d/b/copy.bin"
|
||
cp "$d/a/one.bin" "$d/b/copy2.bin"
|
||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||
head -c 100 /dev/urandom > "$d/t1/sub/f2"
|
||
cp "$d/t1/f1" "$d/t2/f1"
|
||
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
|
||
cp "$d/t1/f1" "$d/t3/f1"
|
||
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
|
||
./sfdupes scan "$d"
|
||
./sfdupes report
|
||
./sfdupes trees
|
||
# incremental behavior (scan a subtree; records elsewhere persist):
|
||
./sfdupes scan "$d/a" # everything unchanged, nothing hashed
|
||
printf 'z' >> "$d/a/one.bin" # modify: next scan re-hashes it
|
||
rm "$d/a/unique.bin" # delete: next scan removes its record
|
||
./sfdupes scan "$d/a" # 1 updated, 1 removed
|
||
./sfdupes report
|
||
```
|
||
|
||
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
|
||
group (two dupe rows, `first` is the lexicographically smallest path);
|
||
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
|
||
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
|
||
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
|
||
ordered by size descending.
|
||
|
||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
|
||
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
|
||
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
|
||
|
||
Expected from the second `report` (after the modify/delete rescan):
|
||
`one.bin` has left its group (its content changed), so
|
||
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
|
||
database.
|
||
|
||
The test suite automates this scenario (see `scan_test.go`), plus a negative
|
||
check: `report` and `trees` operate on the database alone and never touch
|
||
the scanned filesystem.
|
||
|
||
## TODO
|
||
|
||
Tracked in [TODO.md](TODO.md).
|
||
|
||
## Non-goals
|
||
|
||
- No byte-for-byte compare, and no deletion or linking of duplicates. Files that
|
||
match are compared by a SHA-256 of the whole file below 100 MiB, and only by
|
||
samples at 100 MiB and over. The reports are advisory; acting on them is the
|
||
user's job.
|
||
- No persistence beyond the SQLite database described above; no export/import
|
||
formats.
|
||
- No daemon or filesystem watcher; scheduling rescans is cron's job.
|
||
|
||
## License
|
||
|
||
MIT. See [LICENSE](LICENSE).
|
||
|
||
## Author
|
||
|
||
[@sneak](https://sneak.berlin)
|