sfdupes

Description

sfdupes is an MIT-licensed Go command-line tool by @sneak that finds candidate duplicate files, and whole duplicate directory trees, on large filesystems. It does this without reading every byte of every file.

Two files count as duplicates when they have the same size and the same three SHA-256 hashes. A hash is a short fingerprint of a stretch of bytes; two stretches with the same fingerprint are, in practice, identical. Two of the hashes cover the first and the last 64 KiB of the file. The third covers the whole file when the file is under 100 MiB, so a match proves identical content. At 100 MiB and above it covers only 1 MiB at each gigabyte, so a large file is never read in full. A match between two large files is a strong candidate, not proof.

sfdupes is meant for multi-terabyte servers, where reading every byte of every file takes too long. It has four subcommands. scan walks the filesystem and keeps its results between runs in an SQLite database, a database held in one file. A rescan reads only new and changed files, so scan can run every day from cron, the system's job scheduler. report, trees and info read the database alone and can run at any time.

Getting Started

make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv
./sfdupes info

scan walks the trees named by its operands, the paths given after scan on the command line. It keeps one database record per regular file: an ordinary file, as opposed to a directory, a symlink or a device. report prints the file-level duplicates. trees prints the duplicate directory trees. info prints the database's path, size and record count.

The database is /var/lib/sfdupes/db.sqlite unless SFDUPES_DATABASE names another file. The first scan creates it.

Every command writes its data to stdout and everything else to stderr: progress, warnings, summaries and help text. stdout and stderr are a program's two output streams. The shell shows both on the terminal, and > redirects stdout alone.

Install

With Go installed, this builds and installs the newest tagged release:

go install sneak.berlin/go/sfdupes@latest

@main installs the tip of the main branch instead:

go install sneak.berlin/go/sfdupes@main

The binary goes to $(go env GOPATH)/bin, or to $GOBIN when that is set.

sfdupes --version prints sfdupes VERSION and exits. For a binary from go install, VERSION is the version Go recorded in it. For @latest that is a release tag such as v0.0.3. For @main it is a pseudo-version, Go's version string for an untagged commit, which ends in the commit's hash. A binary built from a clone carries the git tag or commit it was built from. VERSION is dev only when the binary carries no version at all. To see everything Go recorded in an installed binary:

go version -m "$(command -v sfdupes)"

From a clone, make build writes the binary to ./sfdupes:

git clone https://git.eeqj.de/sneak/sfdupes.git
cd sfdupes
make build

Copy the binary to /usr/local/bin for the cron job below.

make docker builds a Docker image tagged sfdupes. The build runs the tests and the linter first (see "Build"). Inside the image sfdupes runs as root, with the database at its default path. Bind-mount /var/lib/sfdupes to keep the database between runs. Mount the tree to scan at the same path inside the container as on the host; read-only is enough. The database then records the host's paths, and so do the reports.

make docker
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
    sfdupes scan /srv
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv

Daily scan from cron

Run scan as root, so that it can read every file. A path it cannot read is skipped with a warning and loses its database record. As the file /etc/cron.d/sfdupes:

30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log

• The first scan creates /var/lib/sfdupes/db.sqlite and its directory. To keep the database elsewhere, put SFDUPES_DATABASE=/path/to/db.sqlite before the command on the same line.
• scan writes nothing to stdout. Its stderr, appended here to /var/log/sfdupes.log, holds:
  • one line naming the database;
  • a progress line as each phase starts, one every 5 seconds while it runs, and one as it ends;
  • a warning for each path it skips;
  • a line for each path the walk leaves out on purpose (see "Rules for the walk");
  • the summary line.
• The log grows with every scan; rotate it like any other log.
• Skipped paths do not fail a scan. It still exits 0, and cron sends no mail.
• A scan that fails exits 1, with the reason among the last lines of the log. tail prints those lines, and cron mails them to root if the host can send mail.
• A scan still running when the next one starts carries on. The new one fails at once, and the lines cron mails include sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock).
• report, trees and info need only read access to the database. With the usual umask of 022, the first scan creates it readable by every user, so an unprivileged user can run them against root's database.

Reading the reports

Each row of report names two copies of one file. Each row of trees names two copies of one directory tree. Within a group of copies, the path that sorts first byte by byte is first, and every other path is a dupe of it. first says nothing about which copy is the original or the oldest. Which copy to keep is your choice.

A row is a candidate, not proof:

• The reports show the files as they were at the last scan. A file may have changed or gone since.
• A file of 100 MiB or more is compared only on samples of its content (see "Duplicate detection").
• Two paths that are hard links, two names for one file, are listed as duplicates. They share their data, so removing one frees nothing.

Compare a pair byte for byte before removing either copy. For the row /srv/a/big.iso, /srv/b/big-copy.iso, 4294967296:

cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
[ /srv/a/big.iso -ef /srv/b/big-copy.iso ] && echo "hard links"

cmp prints nothing and exits 0 when every byte matches. Otherwise it reports where the files differ. The second line prints hard links when both paths are the same file.

A path that holds a backslash, tab, newline or carriage return is escaped in the reports (see "Report output format"). Undo the escapes before using such a path. printf '%b' does that, because every backslash in an escaped path starts one of the four escapes. Command substitution drops trailing newlines, so print an x after the path and remove it afterwards:

p="$(printf '%bx' '/srv/a/tab\tname.txt')"; p="${p%x}"
cmp "$p" /srv/b/tab-copy.txt

Check a trees row with diff -r. It compares the two trees file by file. It also names anything present in only one of them, such as an empty directory or a symlink, which trees does not see.

Rationale

Duplicate finders that hash every file in full do not scale to the target environment. That is about ten million files and 150 TB. The disks may be slow or busy, as a ZFS storage pool is while it rebuilds a disk.

sfdupes reads a file under 100 MiB in full. It reads a larger file only at its two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it makes the comparison exact. Sampling cuts a 150 GiB file to about 150 MiB of reads, at the price of a comparison that is not exact.

The hashes live in a database that persists between runs, so the expensive filesystem pass is incremental. A rescan reads only the files that are new or whose size or modification time (mtime) changed. All analysis then works from the database alone.

The goal is whole duplicated trees, not only single files: a download extracted twice, or a project copied once too often. Such a tree can be removed as a unit.

Design

Goals

In order:

1. Find whole duplicate trees, not just files. The aim is to find places where the same set of files and directories exists at two or more paths. A whole subtree can then be removed at once. File-level detection is the foundation; tree-level detection is built on top of it.
2. Bound the reads per file. A file is read only when it is new or has changed. A file under 100 MiB is then read in full, and a larger one only at its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read in full. The target is a filesystem of about ten million files and 150 TB, on disks that may be slow or busy. During a scan, one small record per file is held in memory: its path, size and mtime. The hashes stay in the database. report holds the duplicate paths of one file size at a time. trees holds one entry per directory, plus the files of the directories it is reading.
3. Scan incrementally, analyze offline. The scan keeps a persistent database, and an unchanged file is never read again. report, trees and info work from the database alone and never touch the scanned filesystem. scan is built to run from cron; the reports run at any time, against what the database holds.
4. Keep the streams apart. Everything on stdout is machine-readable data. Progress, warnings, summaries, help and usage text go to stderr.

Subcommands

• scan walks the filesystem and synchronizes the database: one record per regular file.
• report prints the file-level duplicate report from the database.
• trees prints the duplicate-tree report from the database.
• info prints the database's details.

sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes --version
sfdupes [command] --help

A missing or unknown subcommand prints the usage text on stderr and exits 2.

The reference below is also the program's specification.

Non-goals

• No byte-for-byte compare, and no deleting or linking of duplicates. The reports are advisory; acting on them is your job.
• No persistence beyond the SQLite database; no export or import formats.
• No daemon or filesystem watcher; scheduling rescans is cron's job.

Reference

Settings

• SFDUPES_DATABASE names the database file. When it is unset or empty, the database is /var/lib/sfdupes/db.sqlite. There is no command-line flag for it. The path names the file exactly, whatever characters it holds, ?, # and % included. A relative path is relative to the working directory.
• --workers N (scan) sets how many directories the walk reads at once, and how many files the hash phase reads at once. The default is the number of CPUs. N must be at least 1; a smaller value is a usage error, reported in one line on stderr with exit 2 before anything is scanned.
• -x, --one-file-system (scan) keeps the walk of each operand on that operand's filesystem, as the same flag does for du and rsync. By default the walk crosses filesystem boundaries.
• --no-default-excludes (scan) lets the walk enter the default excludes: the system directories the walk otherwise leaves out, such as /proc and /tmp on Linux (see "Rules for the walk").
• --version, -v prints sfdupes VERSION (see "Install") to stdout and exits 0, writing nothing to stderr.
• --help, -h, alone or after a subcommand, prints the help text to stderr and exits 0, writing nothing to stdout. sfdupes scan --help also lists the default excludes of the system it runs on.

Duplicate detection

Two files are duplicates when they have the same size and the same three hashes. A file's size and its three hashes together are its signature. scan computes the hashes and stores them in the database. report and trees group the files with the same signature, whichever scan recorded them.

Every record holds three SHA-256 hashes:

• head, the hash of the file's first 64 KiB;
• tail, the hash of the file's last 64 KiB;
• content, the content hash described below.

The content hash depends on the file's size:

• Under 100 MiB, it is the SHA-256 of the whole file. A match proves identical content, barring a SHA-256 collision.
• At 100 MiB and above, it is the SHA-256 of samples. A sample is the 1 MiB that starts at each gigabyte boundary: offset 0, 1 GiB, 2 GiB and so on. The samples go into one hash in order. A sample that would run past the end of the file stops there.

In a file of 64 KiB or less, each 64 KiB stretch is the whole file, so all three hashes are the hash of the whole file. An empty file has fixed hashes and is never opened.

The sampled hash is not exact. Bytes outside the samples and outside the first and last 64 KiB are never read. Two large files that differ only in such bytes are reported as duplicates. That is the price of never reading a 150 GiB file end to end.

Only files of the same size are compared, so their samples lie at the same offsets. A file under 100 MiB and one of 100 MiB or more never share a size, so a whole-file hash is never compared with a sampled one.

scan reads a file only to hash it, and only when the file is new or has changed since its record was written.

scan mode

scan takes one or more PATH operands naming the trees to scan. There is no default. An operand may be a directory or a regular file.

Operands

• scan with no operand is a usage error: the usage text goes to stderr, and the exit code is 2.
• An operand that does not exist is a fatal error, exit 1.
• Each operand is made absolute before the walk. A . or .. in it is resolved as text, and symlinks are not followed. Records are keyed by that absolute path, so results do not depend on the working directory.
• All operands belong to one scan, and all are walked at once by the same workers (see "Concurrency").
• An operand that repeats another, or lies under another, is dropped before the walk, without a line, so every file is reached once. An operand under another is kept when the outer operand's walk would not reach it. That happens when the inner operand, or a directory between the two, is one of these:
  • a default exclude (see "Rules for the walk");
  • a directory on a filesystem that is not persistent: one the kernel makes up or keeps in memory (see "Rules for the walk");
  • with -x, a directory on another filesystem than the outer operand.
• An operand that is a symlink, socket, FIFO or device node is not scanned, and neither is a directory named .zfs. scan prints a warning naming the path and what it is, counts it as skipped, and drops the operand. Another operand beneath it is still scanned. The scan keeps the records beneath a dropped operand, unless another operand covers them.
• An operand on a filesystem that is not persistent is dropped too. scan prints a line naming it and its filesystem type, and does not count it as skipped.
• A scan whose every operand is dropped walks nothing and exits 0.

What a scan does to the database

scan brings the database in line with the filesystem under the scanned operands:

• A file not yet in the database is hashed and inserted.

• A file already in the database is skipped without being read when its size equals the recorded size and its mtime is not newer than the recorded mtime. This is what makes a daily rescan cheap.

• A file whose size differs, or whose mtime is newer, is hashed again. The mtime is compared to the nanosecond, as finely as the filesystem records it. A same-size rewrite counts as a change whenever the filesystem gives it a later mtime, even within the same second.

• The scan deletes each record under a scanned operand whose file it could not verify. That covers deleted files, and also paths whose details or contents the scan could not read this run. So the database holds only what the latest scan over each path could verify. A later successful scan adds such files again.

• Records outside the scanned operands are left alone, so disjoint trees can be scanned on different schedules into one database.

• The scan keeps every record at or below a path that the walk leaves out on purpose. Such records count as outside the scanned operands. The walk leaves out:

  • a default exclude;
  • a path that is not on a persistent filesystem;
  • with -x, a directory on another filesystem than its operand.

  A filesystem mounted under an operand can therefore be scanned on its own schedule into the same database. An operand at or inside such a path is walked on its own. Its walk verifies the records below it, so they are deleted when their files are gone.

Phases

scan runs four phases in order. Parallel work happens inside the walk and hash phases.

1. load reads the existing records. Each record under the operands is kept in memory by path, with its size and mtime, for the walk to compare against.
2. walk lists the trees under all operands. Several workers (see "Concurrency") share one queue of directories to read. Each worker reads one directory at a time, hands the subdirectories it finds back to the queue, and stats each regular file as it finds it. To stat a file is to read its size, type and times without opening it. The walk compares each file with its record. An unchanged file needs nothing more. Every new or changed file is passed to the hash phase.
3. hash reads every new or changed file and computes its three hashes. Files are read in inode order. An inode is the filesystem's own record of a file, and files with nearby inode numbers usually lie near each other on disk, so this order keeps a spinning disk from seeking back and forth. Paths that are hard links to one file are read once and share the result, so a backup tree made of hard links costs one read per file. Completed records are committed to the database 10,000 at a time while hashing runs. A scan interrupted after hours therefore keeps everything hashed so far.
4. update commits the last partial batch of hashed records, then deletes the records the scan did not verify.

Reading a tree one directory at a time waits on the disk at every step and takes hours at millions of files. Reading many directories at once is what makes the walk practical on large or busy storage pools.

Rules for the walk

• Only regular files are recorded. Symlinks, sockets, FIFOs and device nodes are passed over, and symlinks are never followed.

• The walk never enters a directory named .zfs. Such a directory holds ZFS snapshots, and walking it would list every file once per snapshot.

• The walk never records the scan's own database files. Those are the database, its .lock file, and the -wal and -shm files SQLite keeps beside it while the scan runs (see "Concurrent reads"). It knows them by device and inode, so it passes over them however it reaches them, through a symlinked directory included. It prints no line for them and does not count them as skipped. A record an earlier scan left for one of them is deleted.

• By default the walk crosses filesystem boundaries. With -x, it never enters a directory on a different filesystem than its operand. The records below such a directory are kept, as described under "What a scan does to the database".

• The walk never records anything on a filesystem that is not persistent. A filesystem is not persistent when the kernel makes it up or keeps it in memory. These are the types, as sfdupes scan --help lists them on Linux and macOS:

  proc, sysfs, devtmpfs, devpts, tmpfs, ramfs
  cgroup, cgroup2, debugfs, tracefs, securityfs, pstore
  bpf, configfs, fusectl, mqueue, hugetlbfs, efivarfs
  binfmt_misc, autofs, nsfs, rpc_pipefs, selinuxfs, devfs
  

  devfs is the macOS filesystem of /dev; the others are Linux names. As it starts, scan lists the mounted filesystems and notes the device ID of each one of those types. The device ID is the number by which the kernel tells filesystems apart. On Linux the list comes from /proc/self/mountinfo, which gives each filesystem's own device ID. On macOS it comes from the system's mount list, and the device ID is read from the mount point. Where two filesystems share a mount point on macOS, the one mounted last counts. On any other system the list is empty. A list that scan cannot read is a fatal error, and nothing is scanned. No flag turns this rule off.

  The walk never enters a directory on one of those devices, and never records a file on one. That holds however the walk reaches the path, through a bind mount included. A bind mount shows a directory at a second path, and the files under it keep their device ID. For each path it leaves out this way, scan prints a line on stderr naming the path and its filesystem type:

  scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
  

  The path is not counted as skipped. The records below it are kept, as described under "What a scan does to the database".

• Below an operand, the walk does not enter a default exclude. The default excludes hold the kernel's view of the running system and temporary files, so that sfdupes scan / walks only the files worth comparing. They depend on the system, and sfdupes scan --help lists those of the system it runs on:

  • On Linux: /proc, /sys, /dev, /run, /tmp and /var/tmp.
  • On macOS: /dev, /private/tmp, /private/var/tmp, /private/var/vm, /System/Volumes/VM and /System/Volumes/Data. A walk of / reaches everything in /System/Volumes/Data already, through firmlinks: the links macOS uses to show the data volume inside the system volume. Walking it as well would record each of those files twice.
  • On any other system: none.

  --no-default-excludes turns the default excludes off. What is not on a persistent filesystem stays left out. An operand at or inside a default exclude is still walked, so sfdupes scan /tmp/x works when /tmp/x is on a persistent filesystem. For each default exclude the walk does not enter, scan prints a line on stderr naming it:

  scan: skipped /var/tmp (default exclude)
  

  The directory is not counted as skipped. A default exclude that is not on a persistent filesystem, as /proc is, gets the line for its filesystem instead. One that is itself an operand gets no line, because it is walked as that operand. The scan keeps the records below a default exclude that the walk did not enter, as described under "What a scan does to the database".

• A path that cannot be read, for lack of permission or any other error, gets a one-line warning on stderr and is skipped. The scan continues, and the summary line says how many paths were skipped. A skipped path that has a record from an earlier scan loses that record, and so does every path under an unreadable directory. The database mirrors what the latest scan could verify.

• A file or directory that disappears while the scan runs is not an error. The scan prints no line for it and does not count it as skipped. A path gone when the walk reaches it gets no record, and so does a file gone when the hash phase opens it. The records an earlier scan left at or below it are deleted.

Concurrency

The walk phase and the hash phase each use a pool of workers. A worker is one thread of the program, and each pool has --workers of them. The walk works on several directories at once, and the hash phase on several files at once. On spinning disks both phases wait mostly on seeks, the time the disk head takes to move. A --workers value well above the CPU count can therefore help on storage pools with many disks. The main thread of the program owns the database writes and the progress counts. Each progress display draws from a thread of its own and never blocks the workers.

What scan prints

scan writes nothing to stdout. On stderr it prints, in this order:

• one line naming the database (see "Database");
• for each dropped operand, in operand order, the warning or the line described under "Operands". A repeated operand, or one under another, gets neither;
• the progress display of each phase (see "Progress"), with a warning for each path it skips and a line for each path the walk leaves out on purpose, as they happen;
• the summary line.

The summary line reports the files seen this run, by what happened to each, plus the skips:

scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped

removed counts deleted records, which are not part of the files seen.

Stopping a scan

scan stops cleanly on SIGINT (Ctrl-C) or SIGTERM:

• The workers stop taking work. Each finishes at most the directory listing or the file it is reading, and the progress display is finished.
• The records hashed but not yet committed are committed, so the next scan does not hash them again.
• Nothing else is written or deleted. Records are deleted only after a complete walk, so the records under paths an interrupted walk never reached are kept.
• The database is closed and the lock released, as on any other exit.
• The line scan: interrupted after N files goes to stderr, N being the number of files the walk reached, and the exit code is 1.

The next scan skips the records already written and converges as usual.

After the first signal scan stops catching signals. A second one ends it at once, as any uncaught signal does: the records not yet committed are lost, and the database stays valid. scan catches SIGTERM even when it inherits it as ignored. A SIGINT that scan inherits as ignored, as a background job started from a script does, stays ignored.

report mode

report reads every record from the database and takes no positional arguments. It never touches the scanned filesystem: it does not stat, open or otherwise access any path in the records. Its only I/O is the database, stdout and stderr. Its output is the same whether or not the scanned filesystem is still mounted.

Processing:

• Records are grouped by signature: (size, head, tail, content).
• Every group with two or more paths is a duplicate group.
• Within a group, paths are sorted in byte order. The first is first, and every other path is a dupe.
• Groups are ordered by size, largest first, then by first path. The output is the same for the same database contents.

report has two phases, each with its own progress display on stderr (see "Progress"). open opens the database. read counts the records, then reads them in signature order, largest size first. That is the order of the files_signature index read backwards, so SQLite sorts nothing and returns the first record at once. Records with one signature come together, and so do the records of one size. report therefore holds only the duplicate groups of the size it is reading, and writes them, ordered by first path, once it reaches a smaller size.

Report output format

TSV (tab-separated values) on stdout: a header line, then one row per duplicate file. A group of N paths gives N-1 rows.

first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296

Paths are raw bytes and may hold any byte except NUL, the zero byte. The path columns are escaped, so that every row stays one line of tab-separated fields. A backslash is written \\, a tab \t, a newline \n and a carriage return \r. Every other byte is written unchanged, including bytes that are not valid UTF-8. Undoing the four escapes gives back the stored path. Grouping and ordering use the stored path. The warnings scan prints are escaped the same way, so each warning stays one line.

The summary line on stderr gives:

• the records read;
• the duplicate groups;
• the dupe files, which is the number of rows;
• the reclaimable bytes: the sum of size over all rows.

report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable

trees mode

trees reads the same database as report and takes no positional arguments. It reports whole duplicate directory trees. Two directories are duplicate trees when they hold the same relative paths, and each path holds a file with the same signature in both. Like report, it never touches the scanned filesystem. It rebuilds the directory hierarchy from the paths in the records, split on /.

Definitions:

• A file's signature is (size, head, tail, content), as for report. The mtime is not part of it.
• A directory's digest is a SHA-256 hash computed from the bottom up. Each child contributes one entry. A file's entry is its name and signature; a subdirectory's entry is its name and digest. The entries are sorted in byte order and hashed together. Names are part of the digest, so two trees whose files differ only in name are not duplicates.
• Two directories are duplicate trees when their digests are equal. Equal digests imply the same number of files and the same total size.

Processing:

• Build the hierarchy, compute every directory's digest, and group the directories by digest. Every group with two or more directories is a duplicate-tree group.
• Only the outermost duplicate trees are reported. A group is left out when its members' parents are distinct directories that all have the same digest: the parents' group, or an ancestor's, already implies it. A group holding sibling directories, or members whose parents have different digests, is always reported.
• Within a group, paths are sorted in byte order. The first is first, and every other path is a dupe.
• Groups are ordered by total tree size, largest first, then by first path. The output is the same for the same database contents.

Two limitations are accepted:

• Hard-linked paths count toward duplicate trees. Their content is identical, but they share storage, so removing one frees nothing. The scan uses inode identity to avoid reading a file twice, but does not store it.
• Only the regular files in the database define a tree. Empty directories are invisible. A file skipped during the scan in one copy but not in the other makes two otherwise identical trees differ.

trees has four phases, each with its own progress display on stderr (see "Progress"). open opens the database. read counts the records, then reads them in path order, building the hierarchy and computing the digests as it goes. group groups the directories by digest and picks the groups to report. write writes them to stdout.

Trees output format

TSV on stdout: a header line, then one row per duplicate tree. A group of N trees gives N-1 rows. files is the number of regular files in one copy of the tree, and size their total size in bytes:

first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600

The paths are escaped as described under "Report output format". The root directory's path is /.

The summary line on stderr gives:

• the records read;
• the duplicate-tree groups;
• the dupe trees, which is the number of rows;
• the reclaimable bytes: the sum of size over all rows.

trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable

info mode

info takes no positional arguments and prints the database's details on stdout, one per line: a name, a tab and the value. It opens the database as report does, read-only and without the scan lock, so it runs during a scan and needs only read access. It never touches the scanned filesystem.

database	/var/lib/sfdupes/db.sqlite (default)
database file size	2147483648 (2.0 GiB)
-wal file size	4124152 (3.9 MiB)
modified	2026-10-16T12:02:50Z
schema version	1
records	12345678
size of recorded files	150000000000000 (136.4 TiB)

• database is the path and where it came from, as on the line every subcommand prints first on stderr (see "Database").
• database file size and -wal file size are in bytes, then in human units. The -wal line appears only while a -wal file is beside the database, as during a scan.
• modified is when the database file was last changed, in UTC. While a scan runs, the newest changes may be in the -wal file alone (see "Concurrent reads").
• schema version is the database's PRAGMA user_version. A database with another version than this build's is refused, as by report.
• records is the number of records. size of recorded files adds up the sizes of all recorded files, in bytes, then in human units.

info has two phases, each with its own progress display on stderr (see "Progress"). open opens the database. count reads the schema version and counts the records in one read-only transaction, so the two agree with each other while a scan is committing. info writes its lines after the count display has ended.

Database

All four subcommands work on one SQLite database file.

Location

The database is the file SFDUPES_DATABASE names, when that is set and not empty, and otherwise /var/lib/sfdupes/db.sqlite.

Each subcommand first prints one line on stderr naming the database it uses, before anything else, whether or not stderr is a terminal. The line gives the path and where it came from:

scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)

A relative path is printed with the working directory in front of it, as pwd -P prints it, with symlinks resolved. The rest of the path is printed as set, so a .. in it stays, and the line names the same file. After a cd through a symlink, a ../ therefore counts from the directory the symlink leads to, as it does for any other program. The path is escaped like the paths in a report. scan says new database when no file is at that path as it starts. report, trees and info print the line before they open the database, so the error for a missing database comes after it.

Creation and access

• scan creates the database and its parent directory on first use.
• report, trees and info need an existing database. A missing database is a fatal error, exit 1, telling you to run scan first.
• report, trees and info open the database read-only. They need read access to the database file and no write access to its directory. While the database has -wal and -shm files beside it (see "Concurrent reads"), they read those too. SQLite creates them with the database file's permissions.

The scan lock

Only one scan runs against a database at a time.

• For its whole run, scan holds an exclusive lock, taken with the flock(2) system call, on a lock file beside the database. The lock file's name is the database's path, with symlinks resolved, plus .lock: /var/lib/sfdupes/db.sqlite.lock by default. Every path to one database therefore names the same lock file.
• scan takes the lock before it walks the filesystem or opens the database. Just before that, it creates the database file, empty, if it is missing, so that the lock file is named the same way before the first scan as after. A symlink at the database path is followed, and its target is created.
• A second scan against the same database does not wait. It fails at once with a one-line error naming the lock file and exits 1, having walked nothing and opened nothing. The running scan carries on.
• The lock file is created on first use, open to its owner only, and never deleted. A leftover file blocks nothing: the lock ends with the process that holds it, however that process ends. Deleting the file while a scan runs would let a second scan start.
• report, trees and info never take the lock, so they run during a scan.

Concurrent reads

• While scan runs, the database is in write-ahead log (WAL) mode. In WAL mode SQLite appends changes to a -wal file beside the database, and keeps an index of that file in a -shm file. Readers do not block the writer, so running a report during a scan is safe.
• Every connection has a busy timeout of 10 seconds: how long it waits for a database another connection has locked before it gives up.
• The filesystem is authoritative, and the database follows it. Hashed records are committed 10,000 at a time while the scan runs, so a report may see a scan's changes partly applied. A scan that dies partway leaves a valid database holding every batch committed so far, and the next scan carries on from there.
• When scan closes the database, it switches it back to rollback-journal mode, SQLite's default mode, in which the database file alone holds everything. Between scans a reader therefore needs no -wal or -shm file. Each switch needs the database to itself. A scan that starts while a report is still reading waits for it up to the busy timeout, then fails. A scan that ends while a report has the database open prints a warning and leaves the database in WAL mode until the next scan. report writes its rows while it reads, so it is still reading while a pager such as less, or a stalled pipe, holds up its output.

What scan writes, and when

• As it starts, before the walk: the directory, an empty database file and the .lock file, each only if missing.
• Then, still before the walk: the schema, the table and index described under "Schema", in a new database, and the -wal and -shm files, which stay beside the database while the scan runs.
• During the load and walk phases: no records. A first scan's database holds no records until the hash phase, however long its walk runs.
• During the hash phase: records, committed 10,000 at a time.
• During the update phase: the rest of the hashed records, and the deletions.
• As it closes the database: the -wal and -shm files go away, unless a report still has the database open.

Schema

PRAGMA user_version holds the schema version, currently 1. A database with any other version is a fatal error. scan creates the table and the index and sets the version in one transaction. A first scan stopped while doing so leaves an empty database, which the next scan sets up. A database at version 0 that already has a files table was not made by sfdupes. Every subcommand refuses it, with an error telling you to remove the file and rescan.

CREATE TABLE files (
    path       BLOB PRIMARY KEY,  -- absolute path, raw bytes
    size       INTEGER NOT NULL,  -- bytes, from lstat
    mtime      INTEGER NOT NULL,  -- whole Unix seconds of the mtime
    mtime_nsec INTEGER NOT NULL,  -- nanoseconds within that second
    head       BLOB NOT NULL,     -- raw 32-byte SHA-256 of the first 64 KiB
    tail       BLOB NOT NULL,     -- raw 32-byte SHA-256 of the last 64 KiB
    content    BLOB NOT NULL      -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content);

• path is a BLOB because Unix paths are raw bytes, not necessarily UTF-8.
• mtime holds the whole seconds of the file's mtime, rounded down, and mtime_nsec the nanoseconds past that second. Split this way, they hold any mtime a filesystem can record, before 1678 or after 2262 included, which one 64-bit count of nanoseconds cannot. They serve change detection only and are not part of the signature.
• head, tail and content each hold the raw 32 bytes of a SHA-256 (see "Duplicate detection"). A record whose hash is not 32 bytes is an error.
• files_signature lets report read the records in signature order without sorting the table.

Progress

scan, report, trees and info always show that they are working. Each phase gets a display of its own on stderr from the moment it starts. The display names the phase and its place among the command's phases, and shows the phase's progress, its elapsed time and its time remaining.

The phases, and what each display counts:

• scan: [1/4] load, [2/4] walk, [3/4] hash, [4/4] update.
  • load counts the records first, then reads them against that count. While it counts, the display says counting the records.
  • walk has no exact total. Its estimate is the number of records the last scan left under the operands, shown with a ~. A scan with no records under its operands, such as a first scan, has no estimate. The time remaining is unknown when there is no estimate, and once the walk passes the estimate.
  • hash and update have exact totals. Only the files that need hashing count toward the hash total, so its time remaining is meaningful.
• report: [1/2] open, [2/2] read.
  • open has no total. The display says opening the database, also while it waits on a busy one.
  • read counts the records first, then reads them against that count, writing rows to stdout as it goes.
• trees: [1/4] open, [2/4] read, [3/4] group, [4/4] write.
  • open and read are as for report, except that read writes nothing.
  • group has the directories read found as its total, and write the duplicate-tree groups.
• info: [1/2] open, [2/2] count.
  • open is as for report.
  • count has no total either, because one query counts everything. The display says counting the records until the query returns.

Each display is one line holding:

• the phase and its place among the phases;
• the progress, as [m/n] x%: the items done, the total and the percent. Before a phase has a total, the line shows the items done, or what the phase is waiting for;
• the current rate, in the items the phase counts: files, records, directories or groups. It covers the items completed over the last 5 seconds, or, when stderr is not a terminal, since the line before. It falls to 0 while nothing completes. A burst of items after a wait shows as a high rate for the next 5 seconds, then the rate falls again;
• the elapsed time, labelled elapsed;
• the time remaining, labelled remain: a duration such as 12m9s, or unknown. It comes from the phase's average rate since it started counting, so it does not swing with each burst.

Example lines, as printed when stderr is not a terminal:

[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s

When no item has completed for 10 seconds, the line says stalled and where the phase waits, in place of the rate. That happens while one large file is read, a disk is slow or the database is busy. The places are disk for files, db for the database, out for stdout and cpu for work in memory. A phase that may wait in two places names both, as disk/db:

• walk says disk.
• hash says disk/db: it reads files and writes their records between them.
• load, update and the read phase of trees say db.
• The read phase of report says db/out, because it writes the report while it reads.
• group of trees says cpu, and its write says out.
• A phase still waiting for its total says what it waits for, and nothing more.

On a terminal:

• Every display starts with a spinner, a turning character, followed by its line. The display is drawn again several times a second, so the spinner turns and the elapsed time keeps moving while a phase waits for its next item.

• A phase with an exact total ends its line with its percent and a bar 10 columns wide, at the terminal's right edge. The percent after the count is left out then. A phase with nothing to do shows no bar. Every phase has an exact total once it has counted its work, except these:

  • scan's walk;
  • the open phase of report and trees;
  • both phases of info.

  On a terminal 100 columns wide:

  ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s            12% |█         |
  

• No line reaches the terminal's last column, however narrow the terminal is. A line too wide for the terminal first leaves out its rate, or the words of a stalled phase, or what the phase is waiting for. It leaves out its bar next. It is cut short at the edge only when the four remaining parts do not fit on their own. Those are:

  • the phase and its place;
  • the progress;
  • the elapsed time;
  • the time remaining.

  That happens hours into a scan of millions of files, on a terminal narrower than about 75 columns. A terminal made narrower while a phase runs is fitted at the next draw.

• A warning printed during a phase lands on a line of its own, never inside the display. So does every line written to stdout when stdout and stderr are the same file: one terminal, or one pipe or file that both are sent to. When they are not, stdout is written without waiting on the display, and the display keeps being drawn while a slow reader holds up stdout.

When stderr is not a terminal, such as a pipe, a file or /dev/null, there are no redraws. Each phase prints its line as it starts, every 5 seconds while it runs, whether or not an item has completed, and once more as it ends.

A phase that stops short of its total, as an interrupted one does, keeps showing the count it stopped at.

Error handling and exit codes

• 0: success, even when files were skipped with warnings.
• 1: a fatal error, or a scan stopped by SIGINT or SIGTERM. A fatal error is one of these:
  • an operand that does not exist, or whose details scan cannot read;
  • a list of mounted filesystems that scan cannot read;
  • another scan already running against the same database;
  • a database that cannot be created, opened, read or written, its directory and its lock file included;
  • a relative database path when the working directory cannot be read;
  • a database with another schema version, or one at version 0 that already has a files table;
  • a record whose head, tail or content is not 32 bytes;
  • a missing database for report, trees or info;
  • a failed write to stdout.
• 2: a usage error, which is one of these:
  • a missing or unknown subcommand;
  • an unknown flag;
  • scan with no operand;
  • --workers without a value, or with a value that is not a whole number or is below 1;
  • report, trees or info with a positional argument.

A fatal error is reported in one line on stderr that starts with sfdupes:. A failed write to stdout, such as a full disk, is reported the same way. Two cases never reach sfdupes as a failed write:

• When the reader of a stdout pipe exits early, as in sfdupes report | head, the next write ends sfdupes with SIGPIPE, the signal for a write to a closed pipe. It ends quietly and without a summary, as cat or sort would. The shell reports the signal, status 141 in most shells, not exit 1.
• When stdout is closed outright, as in sfdupes report >&-, the Go runtime opens /dev/null in its place before sfdupes starts. The output is discarded, and the run succeeds, as with > /dev/null.

Entrypoints

This repository follows the Scripts to Rule Them All standard. The executables in script/ are the entrypoints of the development workflow, and the Makefile targets are thin shims that call them. Every script is POSIX sh, finds the repository root itself, and can be run from any directory.

• script/bootstrap installs what building and developing need. It can be run again at any time; installed tools are kept. git, make and go are checked for presence only. A missing one is installed from the first of nix, apt, brew or apk found on the host. An installed node is used when its major version is 22. Otherwise the script downloads nvm, checks the archive's SHA-256, and installs node 22.17.0 through it. It then activates yarn 1.22.22 under that node through corepack. With an installed node 22, a yarn already on PATH is used as it is. Otherwise yarn 1.22.22 is activated through corepack, or installed with npm when there is no corepack. yarn install --frozen-lockfile then installs the prettier that package.json and yarn.lock pin. The linter is never installed; it runs in Docker (see script/lint). docker is not installed either, and testing, linting and the image build need it. The script ends with go mod download.

• script/setup makes a fresh clone ready for development: script/bootstrap, then script/install-precommit.

• script/projectname prints sfdupes. Scripts that need the name call it, so they stay identical across repositories.

• script/test runs the test suite under the race detector, Go's check for unsynchronized access to memory from several threads. The suite runs with a 90-second timeout and coverage enabled. On failure it reruns the suite verbosely, so that the log names the failing test. It builds the Dockerfile's test phase alone and writes no image. The race detector needs cgo, Go's bridge to C code, and a C compiler. The phase therefore starts from a Debian golang image, which has gcc. The image is pinned by its digest, its content hash. The tests run as nobody, because several of them make a file unreadable, and root reads it anyway.

• script/lint runs the linter. It builds the Dockerfile's lint phase alone and writes no image. The phase starts from the golangci/golangci-lint image, pinned by its digest, and runs three build steps:

  • the gofmt check;
  • golangci-lint config verify;
  • golangci-lint run.

  A successful build is a clean lint. The linter never runs on the host, so a working docker is the one thing linting needs.

• script/fmt formats in place: the Go sources with gofmt -s -w, and every Markdown file with prettier. prettier indents by 4 spaces and wraps prose, as .prettierrc sets and as the script also passes on the command line. It wraps at 80 columns, its default width. Both run on the host; prettier runs through yarn at the version yarn.lock pins. When the node on PATH is not node 22, the script loads the node 22.17.0 that script/bootstrap installed.

• script/fmt-check is the read-only form of script/fmt: it names any unformatted file and exits non-zero instead of writing. gofmt and prettier both run every time, and each names itself when it fails. The Dockerfile's lint phase runs the same gofmt check.

• script/check runs script/test, script/lint and script/fmt-check, in that order, and changes nothing. The first two need docker; the third needs what script/bootstrap installs.

• script/docker builds the Docker image, tagged with the name from script/projectname. It passes git describe --tags --always --dirty, or unknown when that is empty, as the VERSION build argument. The build stage depends on the lint and test phases, so this also runs the tests and the linter.

• script/cibuild runs script/bootstrap, then script/check, then builds the image as script/docker does. The Gitea workflow runs it on every push, and a successful run means every check passed. The tests and the linter run twice: once in script/check, and again as stages of the image build.

• script/precommit runs from the git pre-commit hook. go mod tidy must change nothing, or the commit fails; then script/check runs.

• script/install-precommit installs .git/hooks/pre-commit, which runs script/precommit.

Every docker build in script/ passes a new CACHEBUST build argument. In every stage of the Dockerfile, ARG CACHEBUST comes after the steps that install system packages and download the Go modules. Every step that lints, tests or compiles comes after it. Every RUN step after it runs again on each build. Without that, Docker would serve the test and lint steps from its cache on an unchanged tree, and the build would pass having run neither. The layers above the line stay cached until the base image, their own lines, go.mod or go.sum change. Go's build cache is kept between builds, so only what changed is compiled again. The tests run with -count=1, so no test reports a stored result. golangci-lint run keeps the linter's own cache as well, so an unchanged package is not linted again.

Build

The script/ entrypoints above hold the implementations. The Makefile targets are shims onto them, except build, which carries the compile recipe:

• make or make build builds the sfdupes binary with cgo disabled. Building is the default target.
• make bootstrap installs the build and development dependencies.
• make setup prepares a fresh clone: bootstrap, then the pre-commit hook.
• make test runs the test suite under the race detector, in Docker (see script/test). It needs docker.
• make lint runs the linter and the gofmt check, in Docker (see script/lint). It needs docker.
• make fmt formats the Go sources and the Markdown on the host, and make fmt-check checks them without writing. Both need go and what make bootstrap installs.
• make check runs test, lint and fmt-check, and changes nothing. It needs docker and what make bootstrap installs.
• make docker builds the Docker image, which runs the tests and the linter as build stages.
• make hooks installs the pre-commit hook.
• make clean removes the binary.

Constraints

• The language is Go. The module is sneak.berlin/go/sfdupes, and the binary is sfdupes.
• The dependencies beyond the standard library are:
  • github.com/spf13/cobra, for the command line;
  • github.com/schollz/progressbar/v3, for the progress display;
  • golang.org/x/term, to tell whether stderr is a terminal;
  • modernc.org/sqlite, the SQLite driver;
  • golang.org/x/sys, for the scan lock and the list of mounted filesystems.
• modernc.org/sqlite is pure Go, so the binary builds without cgo. The Makefile exports CGO_ENABLED=0. The test build is the one exception: the race detector needs cgo (see script/test).
• report and trees are deterministic: the same database contents give the same output, whatever order the records were inserted in.

Definition of done

All of the following, run in this directory, must pass:

1. make check passes: the tests, the linter, gofmt and prettier.

2. make docker succeeds.

3. The smoke test below behaves as described. It builds a throwaway tree in a temporary directory; never test against real data.

   d=$(mktemp -d)
   export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
   mkdir -p "$d/a" "$d/b"
   head -c 2000 /dev/urandom > "$d/a/one.bin"
   cp "$d/a/one.bin" "$d/b/copy.bin"
   cp "$d/a/one.bin" "$d/b/copy2.bin"
   head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
   printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
   printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
   : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
   # duplicate trees: t1 and t2 are identical; t3 differs by one filename
   mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
   head -c 3000 /dev/urandom > "$d/t1/f1"
   head -c 100  /dev/urandom > "$d/t1/sub/f2"
   cp "$d/t1/f1" "$d/t2/f1"
   cp "$d/t1/sub/f2" "$d/t2/sub/f2"
   cp "$d/t1/f1" "$d/t3/f1"
   cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
   ./sfdupes scan "$d"
   ./sfdupes report
   ./sfdupes trees
   # incremental behavior (scan a subtree; records elsewhere persist):
   ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
   printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
   rm "$d/a/unique.bin"                   # delete: next scan removes its record
   ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
   ./sfdupes report
   

   Expected from the first report:

   • one.bin, copy.bin and copy2.bin form one group: two rows, with first the path that sorts first.
   • t1/f1, t2/f1 and t3/f1 form one group.
   • t1/sub/f2, t2/sub/f2 and t3/sub/f2renamed form one group.
   • tiny1 and tiny2 form a pair, and so do empty1 and empty2.
   • unique.bin and tiny3 appear nowhere.
   • The groups come in order of size, largest first.

   Expected from trees: exactly one row, with first $d/t1, dupe $d/t2, 2 files and 3100 bytes. The two sub directories are left out, because the t1 and t2 group implies them. t3 appears nowhere, since one of its file names differs.

   Expected from the second report, after the rescan: one.bin has left its group, because its content changed. copy.bin and copy2.bin remain as one pair. unique.bin is gone from the database.

   The test suite automates this scenario (see scan_test.go). It also checks that report and trees never touch the scanned filesystem.

TODO

Tracked in TODO.md.

License

MIT. See LICENSE.

Author

@sneak