sneak/sfdupes
1
0
Fork 0
Code Issues 29 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Add 64 KiB head/tail and content-hash duplicate ladder (closes #61) (#62)
check / check (push) Successful in 57s
Details

Browse Source
This commit was merged in pull request #62.
This commit is contained in:
clawbot
2026-09-22 16:40:43 +02:00
parent 7ac4f6b723
commit 29a65016d0
8 changed files with 555 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+94 -36
View File
@@ -5,14 +5,19 @@
`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading full file contents. Files
are considered duplicates when they have identical size, identical
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
across very large filesystems without reading every byte of every file.
Files are considered duplicates when their sizes are equal and they
agree on a short ladder of hashes. A file under 10 MiB is hashed in full
and compared directly. A larger file is gated first on the SHA-256 of
its first 64 KiB and of its last 64 KiB, and then compared on a content
hash — the SHA-256 of the whole file when it is under 50 MiB, or of
gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
the content hash is proof of identical content; at or above 50 MiB it is
a strong candidate signal rather than proof, because the gaps between
samples are never read. The intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive. `scan` maintains a persistent SQLite database of file
multi-terabyte ZFS servers where reading every byte of every file is
prohibitively expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.
@@ -29,7 +34,8 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
```
`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash). The
record per regular file (path, size, mtime, head hash, tail hash,
content hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
@@ -47,10 +53,13 @@ completed scan.
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
only from files whose size at least one other file shares, since a
size-unique file cannot be a duplicate — makes a full-filesystem sweep
tractable, and the signatures are kept in a persistent database, so
disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
whose size at least one other file shares, since a size-unique file
cannot be a duplicate; for those it reads the cheap end windows first
and a content hash second — the whole file below 50 MiB, but only
gigabyte-spaced samples at or above 50 MiB, so the largest files are
never read in full. This keeps a full-filesystem sweep tractable, and
the signatures are kept in a persistent database, so
the expensive filesystem pass is incremental: a rescan re-hashes only
files whose recorded mtime or size changed, and all analysis happens
offline from the database alone. The end goal is
@@ -68,14 +77,18 @@ Goals, in order:
downloads, copied project trees), so the operator can consider
removing an entire subtree at once. File-level duplicate detection is
the foundation; tree-level detection is built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file
(first and last 1024 bytes), and only files whose size at least
one other file shares are read at all — a size-unique file cannot
be a duplicate. Scale target: tens of millions of files, ~150 TB
filesystem, possibly slow or busy disks (ZFS pool under resilver).
Holding one small record (path, size, mtime) per file in memory
during a scan is acceptable; holding every file's hashes is not
(they stay in the database).
2. **Spend I/O in proportion to duplicate likelihood.** Only files
whose size at least one other file shares are read at all — a
size-unique file cannot be a duplicate. Those are compared by the
ladder in "Duplicate detection" below: a file under 10 MiB is hashed
in full, while a larger file is gated on cheap 64 KiB end windows
first, then a content hash that reads the whole file below 50 MiB
but only gigabyte-spaced 1 MiB samples at or above it, so the very
largest files are still never read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks
(ZFS pool under resilver). Holding one small record (path, size,
mtime) per file in memory during a scan is acceptable; holding
every file's hashes is not (they stay in the database).
3. **Scan incrementally, analyze offline.** The expensive filesystem
scan maintains a persistent database; an unchanged file is never
read again on a rescan. All analysis (`report`, `trees`) works from
@@ -141,26 +154,72 @@ All three subcommands operate on a single SQLite database file:
that dies partway leaves a valid database holding everything
hashed so far; the next scan skips those records and converges
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
- Schema (`PRAGMA user_version` is the schema version, currently 2; a
database with any other version is a fatal error). Version 2 added
the `content` column and the 64 KiB head/tail signature (replacing
the version 1 1 KiB end windows), so a version 1 database cannot be
reused: it is rejected and the tree must be rescanned from scratch.
```sql
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256, first 1 KiB
tail TEXT NOT NULL -- lowercase-hex SHA-256, last 1 KiB
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
) WITHOUT ROWID;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not
guaranteed UTF-8. `mtime` is used only for change detection; it is
not part of the duplicate key. `head` and `tail` are empty strings
when the file has never been hashed because its size was unique as
of the last scan that covered it; such records still define the
file for tree reconstruction but never participate in duplicate
groups.
not part of the duplicate key. For a file under 10 MiB `head`, `tail`,
and `content` all hold the whole-file hash (that range is hashed in
full, with no end windows); for a larger file `head` and `tail` hold
the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never
been hashed because its size was unique as of the last scan that
covered it; such records still define the file for tree
reconstruction but never participate in duplicate groups.
### Duplicate detection
Two files are duplicates only when they agree on every rung of this
ladder; a mismatch at any rung means they are not duplicates. `scan`
stores each file's hashes once, and `report` and `trees` group files by
the whole signature — size, `head`, `tail`, and `content` — so the
grouping is exactly this ladder applied across everything scanned into
the database, even across separate scans.
1. **Size.** Files of different sizes are never compared. Only files
whose size at least one other file shares are hashed at all.
2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed
in full and compared directly, with no separate end-window step —
small files are cheap to read to the last byte, and doing so makes
the comparison exact. `head`, `tail`, and `content` all hold this
whole-file SHA-256, so such a file's signature is decided entirely
by its size and its content.
3. **10 MiB and above: head and tail.** For a larger file, the SHA-256
of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
cheap gate that eliminates most same-size pairs before any bulk
reading. At 10 MiB and above the two windows never overlap.
4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the
entire file. Agreement here is proof of identical content (barring a
SHA-256 collision).
5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256:
the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, …
while inside the file, the final window truncated at end of file) is
fed, in order, into one hash. This is **deliberately probabilistic**
— the gaps between samples are never read, so two large files that
agree on every sample are reported as duplicates without being read
in full. It is the price of never reading a 150 GB file end to end.
Because size is already part of the signature, only equal-size files
reach this rung, so their sample boundaries always align.
`head`, `tail`, and `content` are one column each. A file below 10 MiB
and one at or above it never share a size, and neither do a file below
50 MiB and one at or above it, so a stored value is never ambiguous
between the whole-file, end-window, and sampled forms.
### `scan` mode
@@ -229,10 +288,9 @@ during the hash phase:
decides its fate. Size-unique files are never read: new or
changed ones are recorded without hashes in the update phase,
unchanged unhashed ones simply keep their records. Every file
with a shared size is hashed by the worker pool: read the first
`min(1024, size)` bytes and the last `min(1024, size)` bytes
(one read when `size <= 1024`, since the two windows coincide)
and compute the SHA-256 of each. Zero-length files have constant
with a shared size is hashed by the worker pool, computing the
full signature — head, tail, and content — described in "Duplicate
detection" below. Zero-length files have constant
hashes and are never opened. Files are hashed in **inode order**
(minimizing seeks on spinning disks), and paths that are hard
links to the same inode are **read once**, all sharing the one