sfdupes/README.md

sfdupes

Description

sfdupes is an MIT-licensed Go CLI tool by @sneak that quickly finds candidate duplicate files, and ultimately whole duplicate directory trees, on very large filesystems. It does this without reading every byte of every file.

Two files are duplicates when they have the same size and the same three SHA-256 hashes. Two of the hashes cover the first and the last 64 KiB of the file. The third, the content hash, covers the whole file when it is under 100 MiB, so a match proves identical content. At 100 MiB and above it covers only 1 MiB at each gigabyte, so such a file is never read in full. A match there is a strong candidate rather than proof.

The intended use is finding duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers, where reading every byte of every file is prohibitively expensive. scan keeps a persistent SQLite database of file signatures between runs, so it can run from cron. The reports can be generated at any time from the most recent scan.

This README is the complete and authoritative specification.

Getting Started

make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv

scan walks one or more filesystem trees and maintains one database record per regular file (path, size, mtime, head hash, tail hash, content hash). The database persists between runs. A rescan hashes only the files that are new or changed, and removes the records of files that no longer exist. report reads the database and prints the file-level duplicates report. trees reads the same database and prints the duplicate-tree report. info prints the database's details: its path, the sizes of its files and the number of its records. A missing/invalid subcommand — or a scan invocation with no PATH operand — prints a usage message and exits 2.

The database defaults to /var/lib/sfdupes/db.sqlite and can be placed anywhere by setting SFDUPES_DATABASE. The intended deployment is a daily sfdupes scan cron job, with the reporting commands run interactively whenever needed; their results are as fresh as the last completed scan.

Install

With Go installed, this builds and installs the newest release:

go install sneak.berlin/go/sfdupes@latest

Until 1.0, releases are v0.0.N tags on the tip of the main branch: each time main moves and its tip passes script/cibuild, that tip is tagged with the next v0.0.N. So @latest installs main's tip once it has passed the build. @main installs the tip before it is tagged:

go install sneak.berlin/go/sfdupes@main

v0.0.1 is an early build without a database. It does not work as this README describes, and go.mod retracts it.

The binary goes to $(go env GOPATH)/bin, or to $GOBIN when that is set. To see which version an installed binary is:

sfdupes --version
go version -m "$(command -v sfdupes)"

sfdupes --version prints sfdupes VERSION. For a binary from go install, VERSION is the version Go recorded in it: the release tag, such as v0.0.3, or for @main a pseudo-version ending in the first 12 characters of the commit's hash. A binary built from a clone or into the Docker image carries the git tag or commit it was built from instead, and VERSION is dev only when the binary carries no version at all. go version -m prints what Go recorded in the binary; its mod line names the module's version. Binaries from v0.0.2 and earlier print dev, so check those with go version -m.

From a clone, make build writes the binary to ./sfdupes:

git clone https://git.eeqj.de/sneak/sfdupes.git
cd sfdupes
make build

Copy the binary to /usr/local/bin for the cron job below.

make docker builds the Docker image, tagged sfdupes, after running the tests and the linter (see "Build"). The image runs sfdupes as root with the database at its default path, so a bind mount of /var/lib/sfdupes keeps the database between runs. Mount the scanned tree at the same path inside the container as on the host; read-only is enough. The database records paths as the container sees them, so the reports then name the host's paths.

make docker
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
    sfdupes scan /srv
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv

Daily scan from cron

Run scan as root, so that it can read every file: a path it cannot read is skipped with a warning and loses its database record (see "Rules for the walk"). As a file /etc/cron.d/sfdupes:

30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log

• The database is /var/lib/sfdupes/db.sqlite, created with its directory by the first scan. To keep it elsewhere, set SFDUPES_DATABASE=/path/to/db.sqlite before the command on the same line.

• scan writes nothing to stdout. Its stderr, appended here to /var/log/sfdupes.log, holds these lines:

  • a first line naming the database (see "Database");
  • plain progress lines naming each phase and its place among the four, as [2/4] walk: one as the phase starts, one every 5 seconds while it runs, and one as it ends;
  • a warning for each path it skips;
  • a line for each default exclude its walk does not enter, and for each path it leaves out because it is not on a persistent filesystem (see "Rules for the walk");
  • the summary line (see "Progress" and "scan mode").

  The log grows with every scan; rotate it like any other.

• Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A scan that fails, or is stopped by SIGINT or SIGTERM, exits 1 with the reason among the last lines of the log; tail prints them, and cron mails them to root if the host can send mail.

• A scan still running when the next one starts carries on. The new one fails at once, and the lines cron mails include sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock).

• report, trees and info need only read access to the database (see "Database"). Under the usual umask of 022 the first scan creates it readable by every user, so an unprivileged user can run them against root's database.

Reading the reports

Each row of report names two copies of one file, and each row of trees two copies of one directory tree (see "Report output format" and "Trees output format"). In a group of copies, the path that sorts first byte by byte is first and every other path is a dupe of it. first says nothing about which copy is the original or the oldest; which copy to keep is your choice.

A row is a candidate, not proof:

• The reports read only the database, so they show the files as of the last scan; a file may have changed or gone since.
• A file of 100 MiB or more is compared only on samples of its content (see "Duplicate detection").
• Paths that are hard links to one file are listed as duplicates, but they share their data, so removing one frees nothing.

Compare a pair byte for byte before removing either copy. For the row /srv/a/big.iso, /srv/b/big-copy.iso, 4294967296:

cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
[ /srv/a/big.iso -ef /srv/b/big-copy.iso ] && echo "hard links"

cmp prints nothing and exits 0 only when every byte matches, and otherwise reports where the files differ. The second line prints hard links when the two paths are the same file, so removing either frees nothing.

A path holding a backslash, tab, newline or carriage return is escaped in the reports (see "Report output format"). Undo the escapes before using it. printf '%b' does exactly that, because every backslash in an escaped path starts one of the four escapes. Command substitution drops trailing newlines, so print an x after the path and remove it afterwards, or a path that ends in a newline names a different file:

p="$(printf '%bx' '/srv/a/tab\tname.txt')"; p="${p%x}"
cmp "$p" /srv/b/tab-copy.txt

Check a trees row with diff -r, which compares the two trees file by file and also names anything present in only one of them, such as an empty directory or a symlink, which trees does not see.

Rationale

Duplicate finders that hash every file in full do not scale to the target environment. It holds about 10 million files and 150 TB, on disks that may be slow or busy, such as a ZFS pool under resilver.

sfdupes reads a file under 100 MiB in full, and a larger one only at its two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it makes the comparison exact. A full read of every large file would take unreasonably long. Sampling cuts a 150 GiB file to about 150 MiB of reads, at the price of a comparison that is not exact.

The hashes are kept in a persistent database, so the expensive filesystem pass is incremental. A rescan reads only the files that are new or whose size or mtime changed. All analysis then works from the database alone.

The end goal is not individual files but whole duplicated trees, such as a download extracted twice. An operator can consider removing such a tree as a unit.

Design

Goals, in order:

1. Find whole duplicate trees, not just files. The end goal is to identify places where the exact same set of files and directories exists at two or more paths (duplicate extractions, duplicate downloads, copied project trees), so the operator can consider removing an entire subtree at once. File-level duplicate detection is the foundation; tree-level detection is built on top of it.
2. Bound the I/O per file. A file is read only when it is new or has changed. A file under 100 MiB is then read in full, and a larger one only at its ends and at 1 MiB per gigabyte (see "Duplicate detection" below). Therefore no file of 100 MiB or more is read in full. Scale target: tens of millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool under resilver). Holding one small record (path, size, mtime) per file in memory during a scan is acceptable; holding every file's hashes is not (they stay in the database). The reporting commands do not hold every file's hashes either: report reads the records in signature order and holds the duplicate paths of one size at a time, so its memory grows with the duplicates of the size that has the most of them, and trees reads the records in path order and keeps each directory's path, digest and totals, plus the hashes of only the files in the directories holding the record being read, so its memory grows with the number of directories and with the size of the largest directory.
3. Scan incrementally, analyze offline. The expensive filesystem scan maintains a persistent database, and an unchanged file is never read again on a rescan. All analysis (report, trees) works from the database alone and must never touch the scanned filesystem again. scan is designed to be cronned; the reports run at any time against the last completed scan.
4. Clean stream separation. Everything on stdout is machine-readable data. All progress, warnings, summaries, and help and usage text go to stderr. Never mix them.

Constraints

• Language: Go (module sneak.berlin/go/sfdupes). Binary name: sfdupes.
• Dependencies: standard library, github.com/spf13/cobra for the CLI, one progress-bar library (github.com/schollz/progressbar/v3), golang.org/x/term to tell whether stderr is a terminal, one SQLite driver (modernc.org/sqlite, pure Go, so builds keep cgo disabled), and golang.org/x/sys for flock(2) (the scan lock, see "Database"). github.com/spf13/viper is permitted if configuration-file support is ever needed, but is not currently used. No other third-party deps.
• Cross-compilation is not a concern. Builds run with cgo disabled (the Makefile exports CGO_ENABLED=0); the code must remain pure Go.
• Analysis modes (report, trees) must be deterministic: identical database contents, identical output, regardless of the order in which records were inserted.

Subcommands

Four subcommands, all implemented:

1. scan — walk the filesystem and synchronize the database: one signature record per regular file.
2. report — file-level duplicate report from the database.
3. trees — tree-level duplicate report: reconstruct the directory hierarchy from the database records, compute a Merkle-style digest per directory, and report maximal groups of identical trees.
4. info — the database's details: its path, the sizes of its files, its schema version, the number of its records and their total size.

sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes --version
sfdupes [command] --help

--workers N sets the size of each scan worker pool (default: the number of CPUs), -x (--one-file-system) keeps the walk of each operand on that operand's filesystem, and --no-default-excludes lets the walk enter the default excludes, a list of paths; all three are described under "scan mode". sfdupes --version (or -v) prints one line, sfdupes VERSION (see "Install"), to stdout and exits 0, writing nothing to stderr. -h or --help, alone or after a subcommand, prints the help text to stderr and exits 0, writing nothing to stdout.

Database

All four subcommands operate on a single SQLite database file:

• Location: the value of the SFDUPES_DATABASE environment variable when set and non-empty, otherwise /var/lib/sfdupes/db.sqlite. There is no command-line flag. The path names the file exactly, whatever characters it holds (?, # and % included); a relative path is relative to the working directory.

• Each subcommand first prints one line on stderr naming the database it uses, before anything else it prints (progress displays included), whether or not stderr is a terminal. The line gives the path with the working directory put in front of it when it is relative, and otherwise as set (a .. in it stays, so the line names the same file), escaped like the paths in a report (see "Report output format"), and says where the path came from:

  scan: new database /var/lib/sfdupes/db.sqlite (default)
  trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
  

  The working directory put in front is the one pwd -P prints, with its symlinks resolved. After a cd through a symlink to a directory, a path starting ../ therefore names a file beside the directory the symlink leads to, as it does for any other program. scan says new database when no file is at that path as it starts. report, trees and info print the line before they open the database, so the error for a missing database comes after it.

• scan creates the database (and its parent directory) on first use. report, trees and info require an existing database; a missing database file is a fatal error (exit 1) telling the user to run scan first.

• Only one scan runs against a database at a time. For its whole run, scan holds an exclusive flock(2) lock on a lock file beside the database, named by appending .lock to the database's path with symlinks resolved (/var/lib/sfdupes/db.sqlite.lock by default), taken before it walks the filesystem or opens the database. Every path to one database, through a symlink to the file or to a directory above it, names the same lock file. Just before it takes the lock, scan creates the database file, empty, if it is missing (through a symlink at the database path, it creates the symlink's target), so the lock file is named the same way before the first scan sets up the database as after. A second scan against the same database does not wait: it fails at once with a one-line error naming the lock file and exits 1, without walking anything or opening the database, and the running scan carries on. The lock file is created on first use, open to its owner only, and left in place: a leftover file blocks nothing, because the lock ends with the process holding it however it ends, a fatal error or an interrupt included, and deleting the file while a scan runs would let a second scan start. report, trees and info never take the lock, so they run during a scan.

• While scan runs, the database is in WAL journal mode with a busy timeout, so running a report while a cron scan is in progress is safe. The filesystem is authoritative; the database is an eventually-consistent reflection of it. Hashed records are committed in batched transactions while the scan is still running (keeping the WAL small and letting concurrent reports observe progress), so a report may see a scan's changes partially applied, and a scan that dies partway leaves a valid database holding every batch committed so far (an interrupted scan also commits the batch in progress, see "Error handling and exit codes"); the next scan skips those records and converges toward the filesystem.

• scan switches the database back to rollback-journal mode when it closes it, so between scans the database file alone holds the whole database. Each switch needs the database to itself: a scan that starts while a report is still reading waits for it up to the 10-second busy timeout, then fails; a scan that ends while a report has the database open warns and leaves the database in WAL mode until the next scan. report writes its rows while it reads the records, so it is still reading while its output is paused (a pager, a stalled pipe), and a scan started then fails after the busy timeout.

• A scan writes to the database and the files beside it at these points (its phases are described under "scan mode"):

  • As it starts, before the walk: the directory, an empty database file and the .lock file, each only if missing.
  • Then, still before the walk: the schema, in a new database, and the -wal and -shm files, which stay beside the database while the scan runs.
  • While it loads the existing records (load) and walks: no records. A first scan's database holds none, however long its walk has run.
  • During the hash phase: the first records, committed 10,000 at a time.
  • In the update phase: the rest of the hashed records, and the deletions.
  • At the end, as it closes the database: the -wal and -shm files go away, unless a report still has the database open (see above).

• report, trees and info open the database read-only and need only read access to the database file, and no write access to its directory. While the database is in WAL mode they also read the -wal and -shm files beside it, which SQLite creates with the database file's permissions.

• Schema (PRAGMA user_version is the schema version, currently 1; a database with any other version is a fatal error. scan creates the schema and sets the version in one transaction, so a first scan stopped while doing so leaves an empty database the next scan sets up. A database at version 0 that already has a files table was therefore not made by sfdupes; every subcommand refuses it with an error telling the user to remove the file and rescan):

  CREATE TABLE files (
      path       BLOB PRIMARY KEY,  -- absolute path, raw bytes
      size       INTEGER NOT NULL,  -- bytes, from lstat
      mtime      INTEGER NOT NULL,  -- whole Unix seconds of the mtime, from lstat
      mtime_nsec INTEGER NOT NULL,  -- nanoseconds within that second, 0 to 999999999
      head       BLOB NOT NULL,     -- raw 32-byte SHA-256 of the first 64 KiB
      tail       BLOB NOT NULL,     -- raw 32-byte SHA-256 of the last 64 KiB
      content    BLOB NOT NULL      -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
  ) WITHOUT ROWID;
  CREATE INDEX files_signature ON files (size, head, tail, content);
  

  Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime and mtime_nsec hold the file's mtime to the nanosecond: mtime the whole Unix seconds, rounded down, and mtime_nsec the nanoseconds past that second. Split this way they hold any mtime a filesystem can record, one before 1678 or after 2262 included, which a single 64-bit count of nanoseconds cannot. They are used only for change detection and are not part of the duplicate key. Every record holds the three hashes that "Duplicate detection" below describes. The files_signature index lets report read the records in signature order without sorting the table.

Duplicate detection

Two files are duplicates when they have the same size and the same three hashes. scan computes the hashes and stores them in the database. report and trees group the files whose size and hashes all match, whichever scan recorded them.

scan gives every file it records three SHA-256 hashes:

• head, the hash of the file's first 64 KiB.
• tail, the hash of the file's last 64 KiB.
• content, the content hash described below.

In a file of 64 KiB or less, the first and the last 64 KiB are both the whole file, so all three hashes are the hash of the whole file.

The content hash depends on the file's size:

• Under 100 MiB, it is the SHA-256 of the whole file. Agreement on it is proof of identical content, barring a SHA-256 collision.
• At 100 MiB and above, it is the SHA-256 of samples, so the file is never read in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2 GiB and so on. The samples are fed into one hash in order. A sample that would run past the end of the file stops there.

The sampled hash is deliberately probabilistic. Bytes outside the samples and outside the first and last 64 KiB are never read. Two large files that differ only in such bytes are reported as duplicates. That is the price of never reading a 150 GB file end to end.

Only files of the same size are compared, so their samples always lie at the same offsets. A file under 100 MiB and one of 100 MiB or more never share a size, so a whole-file hash is never compared with a sampled one.

scan reads a file only to hash it, and only when the file is new or has changed since its record was written (see "scan mode").

scan mode

scan requires one or more PATH operands naming the trees to scan. There is no default path; invoking scan with no operand is a usage error (usage message on stderr, exit 2). An operand may be a directory or a regular file; an operand that does not exist is a fatal error (exit 1). Because database records persist between runs and are keyed by absolute path, each operand is resolved to an absolute, lexically cleaned path (symlinks are not resolved) before walking, so results do not depend on the working directory. All operands belong to a single scan and are enumerated concurrently: every operand seeds the shared walk worker pool. Overlapping operands are harmless — an operand that duplicates another or lies under another is dropped before walking, so every file is reached exactly once and produces one database record. An operand under another one is kept, though, and walked on its own when any directory from it (from its directory, for a regular file) up to but not including the operand covering it is a default exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with -x, is on a different filesystem than that operand, because that operand's walk does not enter such a directory and so does not reach it.

An operand is not scanned when it is:

• a symlink, which scan never follows, not even as an operand;
• a socket, FIFO or device node;
• at or inside a directory named .zfs.

An operand reached through a symlink into a .zfs directory is not scanned either.

This holds even when the operand lies under another operand. scan prints a one-line warning naming the path and what it is, counts it as skipped, and drops it from the scanned operands before reading the database. An operand beneath a symlink operand is still scanned. The records stored beneath the dropped operand are not deleted: they are treated like any other record outside the scanned operands. If it lies under another operand, they are under that operand instead, and are treated like any other record there. This is not an error: a scan whose every operand is dropped walks nothing and exits 0. An operand that is not on a persistent filesystem (see "Rules for the walk") is dropped the same way. The line scan prints for it is not a warning, though, and it is not counted as skipped.

scan synchronizes the database with the filesystem state under the scanned operands:

• A file not yet in the database is hashed and inserted.

• A file already in the database is skipped without reading its contents when its lstat size equals the recorded size and its lstat mtime is not newer than the recorded mtime. This is what makes a daily rescan cheap.

• A file whose mtime is newer than recorded, or whose size differs, is hashed again. Change detection compares the mtime to the nanosecond, as finely as the filesystem records it, so a same-size rewrite counts as a change whenever the filesystem gives it a later mtime than recorded, even within the same second.

• A database record whose path lies under one of the scanned operands but was not successfully processed this run is deleted. This removes records for deleted files. It also removes records for paths that failed to stat or hash this run: the database only ever contains signatures verified by the most recent scan that covered them (a subsequent successful scan re-adds such files).

• Database records outside the scanned operands are untouched, so disjoint trees can be scanned on different schedules into the same database.

• A record at or below a path the walk left out counts as outside the scanned operands, so it is not deleted. The walk leaves out a path that:

  • is a default exclude;
  • is not on a persistent filesystem;
  • with -x, is a directory on another filesystem than its operand.

  A filesystem mounted under an operand can therefore be scanned on its own schedule into the same database. Records below an operand at or inside such a path are not kept this way, since that operand's own walk verifies them.

scan runs four sequential phases over the whole scan. Parallelism lives inside the walk and hash phases, and batched database writes begin during the hash phase:

1. load — read the existing records. Each record under the operands is kept in memory, by path, with its size and its mtime, for the walk to compare each file with.
2. walk + stat — enumerate the trees under all PATH operands concurrently with the walk worker pool: every operand seeds the shared queue, and each worker reads one directory at a time, handing discovered subdirectories back to the queue and running lstat on each regular file as it is discovered (while the directory's metadata is still hot). Sequential directory enumeration is metadata-latency-bound and takes hours at tens of millions of files; per-directory parallelism is what makes the walk tractable on large or busy pools. The walk compares each file with its record as it goes, and an unchanged file needs nothing more. Every new or changed file is carried to the hash phase as a (path, size, mtime) record.
3. hash — the worker pool hashes every carried file as described in "Duplicate detection" above, which gives the file its head, tail and content. Zero-length files have constant hashes and are never opened. Files are hashed in inode order, which minimizes seeks on spinning disks. Paths that are hard links to the same inode are read once, and all share the one result. A hard-link backup farm therefore costs one read per inode, not per path. The phase total counts actual reads, so its progress and its time remaining are meaningful. Completed records are committed in batched transactions while hashing runs. A scan interrupted after hours therefore keeps everything hashed so far, and the next scan resumes cheaply, skipping records already written.
4. update — commit the final partial batch of hashed records. Then delete the records the scan did not verify: those of vanished files, and of paths that failed to stat or hash.

Rules for the walk:

• Only regular files. Skip directories, symlinks (do not follow, including symlink operands), sockets, FIFOs, and device nodes. An operand that is a symlink, socket, FIFO, or device node is dropped as described in "scan mode" above.

• Never descend into a directory named .zfs (ZFS snapshot pseudo-dirs; walking them would list every file once per snapshot), not even when it is an operand; such an operand is dropped the same way.

• Never record the scan's own database. Once the database and its .lock file are open, scan notes the device and inode of the database file, its -wal and -shm files and its .lock file, and the walk passes over a file with one of them, however it reaches it: as an operand, below one, or through a symlinked directory. It prints no line for such a file and does not count it as skipped. A record that an earlier scan left for one of them is deleted, like the record of any file the scan no longer verifies. There is no -journal file during the walk, because the database is in WAL mode then (see "Database").

• Filesystem boundaries are crossed by default. With -x (long form --one-file-system, following the GNU du/rsync convention), never descend into a directory on a different filesystem than its PATH operand; each operand is bounded by its own filesystem. The records below such a directory count as outside the scanned operands, as described in "scan mode" above: they are not deleted. That does not hold below an operand at or inside that directory.

• Never record anything that is not on a persistent filesystem. A filesystem is not persistent when the kernel makes it up or keeps it in memory: one of type proc, sysfs, devtmpfs, devpts, tmpfs, ramfs, cgroup, cgroup2, debugfs, tracefs, securityfs, pstore, bpf, configfs, fusectl, mqueue, hugetlbfs, efivarfs, binfmt_misc, autofs, nsfs, rpc_pipefs, selinuxfs or devfs (the macOS filesystem of /dev). After it names the database and before it takes the lock, scan lists the mounted filesystems, from /proc/self/mountinfo on Linux and from the system's list of mounted filesystems on macOS (on any other system it lists none), and notes the device ID of each one of those types. On Linux that is the device ID the list gives, so a filesystem that a later mount hides still gives its own. On macOS it is read with lstat on the mount point, and where one filesystem is mounted over another on the same mount point, the one mounted last counts. If scan cannot list them, it fails (exit 1). The walk then never enters a directory, and never records a file, on one of those devices, however it reaches it: as an operand, below one, or through a bind mount, which keeps the device ID of what it binds. No flag turns this off. For each operand, directory or file it leaves out this way, scan prints a line on stderr naming it and the type of its filesystem. It prints that line as it prints a warning, so the line never lands inside a progress display:

  scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
  

  That line is not a warning, and the path is not counted as skipped. An earlier scan's records at or below it count as outside the scanned operands, as described in "scan mode" above: they are not deleted. That does not hold below an operand at or inside it.

• Below an operand, never descend into a default exclude: a path that holds the kernel's view of the running system, or temporary files, so that sfdupes scan / walks only the files worth comparing. The default excludes depend on the system, and sfdupes scan --help lists those of the system it runs on:

  • On Linux: /proc, /sys, /dev, /run, /tmp and /var/tmp.
  • On macOS: /dev, /private/tmp, /private/var/tmp, /private/var/vm, /System/Volumes/VM and /System/Volumes/Data. A walk of / reaches everything in /System/Volumes/Data through the firmlinks, so walking it as well would record each of those files twice.
  • On Windows, relative to the root of the drive an operand lies on: the page, swap and hibernation files pagefile.sys, swapfile.sys and hiberfil.sys, and the directories System Volume Information, $Recycle.Bin, Windows\Temp and Users\*\AppData\Local\Temp, where * matches one path component, as it does in any default exclude. sfdupes does not build for Windows yet, so this list is not used.
  • On any other system there are none.

  --no-default-excludes turns them all off; what is not on a persistent filesystem stays left out. An operand at or inside a default exclude is still walked, so sfdupes scan /tmp/x works, unless it is not on a persistent filesystem. For each default exclude its walk does not enter, scan prints a line on stderr naming it, as it prints a warning:

  scan: skipped /var/tmp (default exclude)
  

  That line is not a warning, and the directory is not counted as skipped. A default exclude that is not on a persistent filesystem, as /proc is, gets the line for its filesystem instead. One that is itself an operand gets no line, since it is walked as that operand. When the walk does not enter a default exclude, the records below it count as outside the scanned operands, as described in "scan mode" above: they are not deleted. That does not hold below an operand at or inside it.

• On a per-path error, such as permission denied or an unreadable file: print a one-line warning to stderr, skip the path, and continue. Per-file errors never abort the run; the final summary reports how many were skipped. As specified above, a skipped path that has a database record from an earlier scan loses that record. An unreadable directory subtree likewise loses its records. This is accepted: the database mirrors what the latest scan could actually verify. The exception is an operand dropped before the database was read, whose record counts as outside the scanned operands.

• A file or directory removed while the scan runs is not an error. The scan prints no line for it and does not count it as skipped. A path gone when the walk reaches it gets no record. A file gone when the hash phase opens it gets none either. The records an earlier scan left at or below such a path are deleted.

Concurrency: the walk phase, which also stats files, and the hash phase each use a worker pool of --workers workers (default runtime.NumCPU()). The walk parallelizes across directories, and hashing across files. --workers must be at least 1: a smaller value is a usage error, reported in one line on stderr with exit 2 before anything is scanned. Both phases are seek-bound on spinning disks, so raising --workers well past the core count can help on pools with many spindles. The main goroutine owns the database writes and the progress counts. Each progress display draws the counts from a goroutine of its own, and must never block the workers.

scan writes nothing to stdout. The summary line on stderr reports the files seen this run broken down by disposition, plus skips:

scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped

(removed counts deleted database records, which are not part of the files-seen total.)

report mode

report reads every record from the database and takes no positional arguments.

report must never touch the filesystem being analyzed. It does not stat, open, or otherwise access any path that appears in the records; its only I/O is reading the database and writing stdout/stderr. report must produce identical output whether or not the scanned filesystem is still mounted.

Processing:

• Group the records by the key (size, head, tail, content).
• Every group with two or more paths is a duplicate group.
• Within each group, sort paths lexicographically (byte order). The first path is the group's first; every other path is a dupe.
• Order groups by size descending (biggest reclaimable space first), tie-broken by first path ascending. Output must be fully deterministic for a given database state.

report runs in two phases, each with its own progress display on stderr (see "Progress"): open opens the database; read counts the records, then reads them in signature order, largest size first. That is the order of the files_signature index read backwards, so SQLite sorts nothing and returns the first record at once. The records sharing a signature come together, and so do the records of one size, so report holds only the duplicate groups of the size it is reading, and writes them, ordered by first path, once it reaches a smaller size. Its memory holds at most the duplicate paths of one size at a time, never every record.

Report output format

TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a group of N):

first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296

Paths are raw bytes and may hold any byte except NUL, so the path columns (first and dupe) are escaped to keep every row one line of tab-separated fields: a backslash is written as \\, a tab as \t, a newline as \n, and a carriage return as \r. Every other byte is written unchanged, including bytes that are not valid UTF-8. Undoing those four escapes gives back the stored path. Grouping and ordering use the stored path, not the escaped one. The warnings scan prints on stderr are escaped the same way, so each warning is one line.

Summary to stderr: records read, number of duplicate groups, number of dupe files, and total reclaimable bytes (sum of size over all dupe rows) in human units.

trees mode

trees reads the same database as report (no positional arguments) and reports entire duplicate directory trees: directories under which the exact same set of relative paths exists with the exact same file signatures.

trees must never touch the filesystem being analyzed — the same rule as report. The directory hierarchy is reconstructed purely from the paths in the records, split on /.

Definitions:

• A file's signature is (size, head, tail, content) — mtime is informational and excluded.
• A directory's digest is a SHA-256 Merkle digest computed bottom-up: serialize the directory's child entries — for a file child, its name and signature; for a subdirectory child, its name and that subdirectory's digest — sort the serialized entries byte-lexicographically, and hash the concatenation. Names are part of the digest: two trees whose files differ only in name are not duplicates.
• Two directories are duplicate trees when their digests are equal. Equal digests imply equal recursive file count and equal total byte size.

Known limitation (accepted): hard-linked paths are reported as duplicates by report and count toward duplicate trees — their content is genuinely identical — even though they share storage, so removing one reclaims no space. Inode identity is used during the scan to avoid redundant reads but is not persisted in the database.

Known limitation (accepted): only regular files that appear in the database define a tree. Empty directories are invisible, and a file skipped during the scan (e.g. permission error) in one copy but not the other will make otherwise-identical trees compare as different.

Processing:

• Build the hierarchy, compute every directory's digest, and group directories by digest. Every group with two or more directories is a duplicate-tree group.
• Report only maximal trees. A group is suppressed when its members' parents are pairwise distinct directories that all share a single digest — such a group is wholly implied by its parents' (or a further ancestor's) group. Groups containing sibling directories, or members whose parents differ, are always reported.
• Within each group, sort paths lexicographically (byte order); the first path is first, every other path is a dupe.
• Order groups by total tree size descending, tie-broken by first path ascending. Output must be fully deterministic for a given input.

trees runs in four phases, each with its own progress display on stderr (see "Progress"): open opens the database; read counts the records, then reads them in path order, building the hierarchy and computing the digests as it goes; group groups the directories by digest and picks the maximal groups; write writes the groups to stdout.

Trees output format

TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a group of N). files is the recursive regular-file count of one copy of the tree; size is the recursive total byte size of one copy:

first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600

The first and dupe paths are escaped as described under "Report output format". The root directory's path is /.

Summary to stderr: records read, number of duplicate-tree groups, number of dupe trees, and total reclaimable bytes (sum of size over all dupe rows) in human units.

info mode

info takes no positional arguments and prints the database's details on stdout, one per line: a name, a tab, and the value. It opens the database the way report does, read-only and without the scan lock, so it runs during a scan and needs only read access (see "Database"). Like report, it never touches the scanned filesystem.

database	/var/lib/sfdupes/db.sqlite (default)
database file size	2147483648 (2.0 GiB)
-wal file size	4124152 (3.9 MiB)
modified	2026-10-16T12:02:50Z
schema version	1
records	12345678
size of recorded files	150000000000000 (136.4 TiB)

• database is the path and where it came from, as on the line every subcommand prints first on stderr (see "Database").
• database file size and -wal file size are in bytes, then in human units. The -wal line is there only when a -wal file is beside the database, as while a scan runs.
• modified is the time the database file was last changed, in UTC. While the database is in WAL mode, the newest changes may be in the -wal file alone.
• schema version is the database's PRAGMA user_version. A database with another version than this build's is refused, as by report.
• records is the number of records. size of recorded files adds up the sizes of all records, in bytes, then in human units.

info runs in two phases, each with its own progress display on stderr (see "Progress"): open opens the database; count reads the schema version and counts the records with one query, a single pass over the table, both in one read-only transaction, so the numbers agree with each other while a scan is committing. info writes its lines once the count display has ended, so on a terminal they come after the displays.

Progress

Use the progress-bar library for all progress of scan, report, trees and info; rendering in the style of pv is the model. All progress goes to stderr.

scan, report, trees and info must always show that they are working and not hung. Each phase gets its own display from the moment it starts. The display names the phase and its place among the command's phases. It shows the phase's progress, its elapsed time and its time remaining.

A scan has four phases ([1/4] load, [2/4] walk, [3/4] hash, [4/4] update):

• load counts the records first, then reads them against that count. While the count runs, the display says it is counting the records.
• walk has no exact total. Its estimate is the number of records the last scan left under the operands, known once load is done and shown with a ~. The time remaining is unknown once the walk passes the estimate, and when there are no such records, as on a first scan.
• hash and update have exact totals. Only files that actually need hashing appear in the hash total, so its time remaining is meaningful.

trees has four phases ([1/4] open, [2/4] read, [3/4] group, [4/4] write, see "trees mode"):

• open has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database.
• read counts the records first, then reads them against that count. While the count runs, the display says it is counting the records.
• group has the directories read found as its total, and write the duplicate-tree groups.

report has two phases ([1/2] open, [2/2] read, see "report mode"):

• open has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database.
• read counts the records first, then reads them against that count, writing the rows to stdout as it goes. While the count runs, the display says it is counting the records.

info has two phases ([1/2] open, [2/2] count, see "info mode"):

• open has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database.
• count has no total either, because one query counts everything: the display says it is counting the records until the query returns, and its time remaining is unknown.

Each display is one line holding:

• the phase and its place among the phases
• a [m/n] x% display (items processed / total items, percent); before a phase has a total, the items processed, or what the phase is waiting for
• current rate (items/s), naming the items the phase counts: files, records, directories or groups. It covers the items completed over the last 5 seconds, or, when stderr is not a terminal, since the line before, so it falls to 0 while nothing completes, and a burst of items after a wait shows as a burst rather than as a steady rate. Once no item has completed for 10 seconds, the word stalled and where the phase waits take the rate's place (see below).
• elapsed time, labelled elapsed
• time remaining, labelled remain: a duration such as 12m9s, or unknown. It comes from the phase's average rate since it started counting, so that it does not swing with each burst.

Example lines, as printed when stderr is not a terminal:

[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s

When no item has completed for 10 seconds, as while one large file is read, a disk is slow or the database is busy, the line says stalled and where the phase waits in place of the rate, which would read 0: disk for files, db for the database, out for stdout, and two of them, as disk/db, for a phase that may wait on either. walk says disk. hash says disk/db, as it reads files and writes their records to the database between them. load, update and the read phase of trees say db. trees' write says out, and its group, which works in memory, says cpu. The read phase of report writes the report while it reads the records, so it says db/out. A phase still waiting for its total already says what it waits for, and says nothing more.

Additional requirements:

• On a terminal, every display starts with the progress-bar library's spinner in its first column, followed by its line. Each display is drawn again several times a second, so its spinner turns and its elapsed time keeps moving while a phase waits for its next item.

• On a terminal, a phase with an exact total ends its line with its percent and a bar 10 columns wide. This percent takes the place of the line's own. The bar stays at the terminal's right edge, so it does not move as the line grows or shrinks. A phase with nothing to do shows no bar. Every phase has an exact total once it has counted what it will do, except these:

  • scan's walk;
  • the open phase of report and trees;
  • both phases of info.

  On a terminal 100 columns wide:

  ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s            12% |█         |
  

• On a terminal no line reaches the terminal's last column, however narrow the terminal is. A line too wide for the terminal first loses whichever of these it shows: its rate, the words of a stalled phase, or what the phase is waiting for. The other four parts are kept whole:

  • the phase and its place;
  • the progress;
  • the elapsed time;
  • the time remaining.

  The line loses its bar only when those four and the bar do not fit. It is cut short at the terminal's edge only when those four alone are too wide, and then the end of the time remaining is lost first. That happens hours into a scan of millions of files, on a terminal narrower than about 75 columns. On a terminal 100 columns wide, the stalled line of the example above keeps its words and its bar. The next draw fits the line to a terminal made narrower while a phase runs. It erases the old line within the new width, and the display goes on redrawing in place.

• When stderr is not a terminal (a pipe, a file, /dev/null), do not emit ANSI redraws: print the line the moment each phase starts, every 5 seconds while it runs (whether or not an item has completed, and never more often), and once more as the phase ends.

• Progress must be non-blocking with respect to the worker pool: the main goroutine counts the items, and each display draws them from a goroutine of its own.

• A warning printed during a phase always lands on a line of its own, never inside the progress display. So does every line written to stdout when stdout and stderr are the same file (the same device and inode): one terminal, as in a plain sfdupes report or sfdupes trees, or one pipe or file that both are sent to. When they are not, stdout is written without waiting on the display, and the display keeps being drawn while a slow reader of stdout holds up a write.

• A phase that stops short of its total, as an interrupted one does, is left showing the count it stopped at rather than filled up.

Error handling and exit codes

• 0: success, even if individual files were skipped with warnings.
• 1: fatal error (e.g., a PATH operand does not exist, another scan is already running against the same database, the database cannot be created/opened/read/written, a missing database for report/trees/info, stdout write failure), or a scan stopped by SIGINT or SIGTERM (see below).
• 2: usage error (including scan with no PATH operand, scan with --workers below 1, and report/trees/info with any positional argument).

A stdout write failure, such as a full disk, is reported in one line on stderr and exits 1. Two cases never reach sfdupes as a failed write:

• When the reader of a stdout pipe exits early, as in sfdupes report | head, the next write ends sfdupes with SIGPIPE, quietly and without a summary, the way cat or sort end. The shell reports the signal (status 141 in most shells), not exit 1.
• When stdout is closed outright (sfdupes report >&-), the Go runtime opens /dev/null in its place before sfdupes starts, so the output is discarded and the run succeeds, as with > /dev/null.

scan stops cleanly on SIGINT (Ctrl-C) or SIGTERM. Its workers stop taking work, each finishing at most the directory listing or file it is reading; the progress display is finished; and the records it has hashed but not yet committed are committed, so the next scan does not hash them again. Apart from that commit it starts no further writes or deletions: records are deleted only after a complete walk, so those under paths an interrupted walk never reached are kept. The database is closed and the lock released as on any other exit, the line scan: interrupted after N files goes to stderr, N being the number of files the walk reached, and the exit code is 1. The next scan skips the records already written and converges as usual.

After the first signal scan stops catching them, so a second one ends it at once, as an uncaught signal does: the records not yet committed are lost, and the database is left valid, as when any scan dies (see "Database"). A SIGINT that scan inherits as ignored, as a script's background job does, stays ignored.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: the normalized executables in script/ are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. Every script is POSIX sh, resolves the repository root itself so it can be run from any working directory, and may be invoked directly. The provided entrypoints are:

• script/bootstrap — install everything needed to build and develop this repository, idempotently, assuming nothing is present. git, make, and go come from the first of nix, apt, brew, or apk found on the host, and are presence-checked only. An installed node is used when its major version is 22; otherwise node 22.17.0 is installed through nvm, which comes from a release archive whose sha256 the script checks, and yarn 1.22.22 is activated under it through corepack. With an installed node 22, a yarn already on PATH is used as it is; otherwise yarn 1.22.22 is activated through corepack, or installed with npm when there is no corepack. yarn install --frozen-lockfile then installs the prettier that package.json and yarn.lock pin. golangci-lint is never installed: it runs in Docker (see script/lint). docker is not installed either; testing, linting and the image build need it. Ends with go mod download.
• script/setup — make a fresh clone ready for development: runs script/bootstrap, then script/install-precommit.
• script/projectname — print this project's name (sfdupes). Scripts that need the name call it, so they stay identical across repositories.
• script/test — run the test suite under the race detector, with a 90-second timeout and coverage enabled, rerunning verbosely on failure so the logs show which test failed. It builds the Dockerfile's test phase alone and writes no image. The race detector needs cgo and a C compiler, which the build never uses, so the phase starts from a digest-pinned Debian golang image, which has gcc. The tests run as nobody, because several of them make a file unreadable and root reads it anyway.
• script/lint — run the linter. It builds the Dockerfile's lint phase alone and writes no image: in the digest-pinned golangci/golangci-lint image, the gofmt check, golangci-lint config verify and golangci-lint run run as build steps, so a successful build is a clean lint. The linter is never run on the host, which makes a working docker the one prerequisite for linting.
• script/fmt — format in place: the Go sources with gofmt -s -w, and every Markdown file with prettier, at the settings in .prettierrc (4-space indents, prose wrapped at 80 columns). Both run on the host, prettier through yarn at the version yarn.lock pins. When the node on PATH is not node 22, this loads the node 22.17.0 that script/bootstrap installed through nvm.
• script/fmt-check — the read-only counterpart of script/fmt: prints any unformatted file and exits non-zero instead of writing. gofmt and prettier both run every time, and each names itself when it fails. The Dockerfile's lint phase runs the same gofmt check.
• script/check — run script/test, script/lint, and script/fmt-check, in that order. Modifies nothing. The first two need docker, the third what script/bootstrap installs.
• script/docker — build the Docker image, tagged with the name from script/projectname, passing the output of git describe --tags --always --dirty (or unknown when that is empty) as the VERSION build argument. The Dockerfile's build stage depends on its lint and test phases, so this also runs the tests and the linter.
• script/cibuild — run script/bootstrap, then script/check, then build the image as script/docker does. This is what the Gitea workflow runs on push; a successful run means every check passed. The tests and the linter run twice: once in script/check, and again as stages of the image build.
• script/precommit — run by the git pre-commit hook: go mod tidy must be a no-op (a resulting change to go.mod or go.sum fails the commit), then script/check.
• script/install-precommit — install the git pre-commit hook, as .git/hooks/pre-commit, that runs script/precommit.

Every docker build in script/ passes a new CACHEBUST build argument. In every stage of the Dockerfile, ARG CACHEBUST comes right after the steps that install system packages and download the Go modules, or right after FROM in a stage with neither, and every step that lints, tests or compiles comes after it, so every RUN step after it runs again each time. On an unchanged tree Docker would otherwise serve the gate steps from its cache, and the build would pass having run no test and no linter. The layers above that line stay in Docker's build cache until the base image, their own lines, go.mod or go.sum change, or until that cache is cleared; only then are the packages installed and the modules downloaded again. Go's build cache is kept between builds on the steps that compile, so only what changed is compiled again; the tests run with -count=1, so none reports a stored result. golangci-lint run also keeps golangci-lint's cache, and takes a package's findings from it only when the package's files and those of the packages it imports, go.mod, the linters and their settings in .golangci.yml, and the golangci-lint version are all unchanged.

Build

The script/ entrypoints above are where the implementations live; the Makefile targets are shims onto them, except build, which carries the compile recipe:

• make / make build — build the sfdupes binary (cgo disabled); building is the default target.
• make bootstrap — install the build and development dependencies.
• make setup — prepare a fresh clone: bootstrap plus the pre-commit hook.
• make test — run the test suite under the race detector, in Docker (90-second timeout; reruns with -v on failure; see script/test); requires docker.
• make lint — run golangci-lint with the repo config and the gofmt check, in Docker (see script/lint); requires docker.
• make fmt / make fmt-check — format the Go sources and the Markdown / verify formatting without writing, on the host; requires go and what make bootstrap installs (see script/fmt).
• make check — test, lint, and fmt-check; modifies nothing. Requires docker and what make bootstrap installs.
• make docker — build the Docker image, which runs the gates as build stages.
• make hooks — install the pre-commit hook.
• make clean — remove the binary.

Definition of done

All of the following, run in this directory, must pass:

1. make check passes (tests, lint, gofmt, prettier).

2. make docker succeeds.

3. Smoke test — create a throwaway tree in a temp dir (never test against real data):

   d=$(mktemp -d)
   export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
   mkdir -p "$d/a" "$d/b"
   head -c 2000 /dev/urandom > "$d/a/one.bin"
   cp "$d/a/one.bin" "$d/b/copy.bin"
   cp "$d/a/one.bin" "$d/b/copy2.bin"
   head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
   printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
   printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
   : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
   # duplicate trees: t1 and t2 are identical; t3 differs by one filename
   mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
   head -c 3000 /dev/urandom > "$d/t1/f1"
   head -c 100  /dev/urandom > "$d/t1/sub/f2"
   cp "$d/t1/f1" "$d/t2/f1"
   cp "$d/t1/sub/f2" "$d/t2/sub/f2"
   cp "$d/t1/f1" "$d/t3/f1"
   cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
   ./sfdupes scan "$d"
   ./sfdupes report
   ./sfdupes trees
   # incremental behavior (scan a subtree; records elsewhere persist):
   ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
   printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
   rm "$d/a/unique.bin"                   # delete: next scan removes its record
   ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
   ./sfdupes report
   

   Expected from the first report: one.bin/copy.bin/copy2.bin form one group (two dupe rows, first is the lexicographically smallest path); t1/f1/t2/f1/t3/f1 form one group; t1/sub/f2/ t2/sub/f2/t3/sub/f2renamed form one group; tiny1/tiny2 pair; empty1/empty2 pair; unique.bin and tiny3 appear nowhere; groups ordered by size descending.

   Expected from trees: exactly one row — first $d/t1, dupe $d/t2, 2 files, 3100 bytes. $d/t1/sub vs $d/t2/sub is suppressed as non-maximal (implied by the t1/t2 group), and t3 appears nowhere (its file set differs by name).

   Expected from the second report (after the modify/delete rescan): one.bin has left its group (its content changed), so copy.bin/copy2.bin remain as one pair, and unique.bin is gone from the database.

   The test suite automates this scenario (see scan_test.go), plus a negative check: report and trees operate on the database alone and never touch the scanned filesystem.

TODO

Tracked in TODO.md.

Non-goals

• No byte-for-byte compare, and no deletion or linking of duplicates. Files that match are compared by a SHA-256 of the whole file below 100 MiB, and only by samples at 100 MiB and over. The reports are advisory; acting on them is the user's job.
• No persistence beyond the SQLite database described above; no export/import formats.
• No daemon or filesystem watcher; scheduling rescans is cron's job.

License

MIT. See LICENSE.

Author

@sneak