sneak/sfdupes
1
0
Fork 0
Code Issues 27 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Add 64 KiB head/tail and content-hash duplicate ladder (closes #61)
check / check (push) Successful in 1m0s
Details

Browse Source 
Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.

The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.

Model: opus-4-8
This commit is contained in:
sneak
2026-09-22 13:58:57 +00:00
parent 7ac4f6b723
commit b80c7e805e
8 changed files with 446 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+82 -36
View File
@@ -5,14 +5,17 @@
`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading full file contents. Files
are considered duplicates when they have identical size, identical
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
across very large filesystems without reading every byte of every file.
Files are considered duplicates when their sizes are equal and they
agree on a short ladder of hashes: the SHA-256 of their first 64 KiB and
of their last 64 KiB, and then a content hash — the SHA-256 of the whole
file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when it
is 50 MiB or larger. Below 50 MiB this is proof of identical content; at
or above 50 MiB it is a strong candidate signal rather than proof,
because the gaps between samples are never read. The intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive. `scan` maintains a persistent SQLite database of file
multi-terabyte ZFS servers where reading every byte of every file is
prohibitively expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.
@@ -29,7 +32,8 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
```
`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash). The
record per regular file (path, size, mtime, head hash, tail hash,
content hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
@@ -47,10 +51,13 @@ completed scan.
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
only from files whose size at least one other file shares, since a
size-unique file cannot be a duplicate — makes a full-filesystem sweep
tractable, and the signatures are kept in a persistent database, so
disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
whose size at least one other file shares, since a size-unique file
cannot be a duplicate; for those it reads the cheap end windows first
and a content hash second — the whole file below 50 MiB, but only
gigabyte-spaced samples at or above 50 MiB, so the largest files are
never read in full. This keeps a full-filesystem sweep tractable, and
the signatures are kept in a persistent database, so
the expensive filesystem pass is incremental: a rescan re-hashes only
files whose recorded mtime or size changed, and all analysis happens
offline from the database alone. The end goal is
@@ -68,14 +75,17 @@ Goals, in order:
downloads, copied project trees), so the operator can consider
removing an entire subtree at once. File-level duplicate detection is
the foundation; tree-level detection is built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file
(first and last 1024 bytes), and only files whose size at least
one other file shares are read at all — a size-unique file cannot
be a duplicate. Scale target: tens of millions of files, ~150 TB
filesystem, possibly slow or busy disks (ZFS pool under resilver).
Holding one small record (path, size, mtime) per file in memory
during a scan is acceptable; holding every file's hashes is not
(they stay in the database).
2. **Spend I/O in proportion to duplicate likelihood.** Only files
whose size at least one other file shares are read at all — a
size-unique file cannot be a duplicate. Those are compared by the
ladder in "Duplicate detection" below: cheap 64 KiB end windows
first, then a content hash that reads the whole file below 50 MiB
but only gigabyte-spaced 1 MiB samples at or above it, so the very
largest files are still never read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks
(ZFS pool under resilver). Holding one small record (path, size,
mtime) per file in memory during a scan is acceptable; holding
every file's hashes is not (they stay in the database).
3. **Scan incrementally, analyze offline.** The expensive filesystem
scan maintains a persistent database; an unchanged file is never
read again on a rescan. All analysis (`report`, `trees`) works from
@@ -141,26 +151,63 @@ All three subcommands operate on a single SQLite database file:
that dies partway leaves a valid database holding everything
hashed so far; the next scan skips those records and converges
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
- Schema (`PRAGMA user_version` is the schema version, currently 2; a
database with any other version is a fatal error). Version 2 added
the `content` column and widened the end windows from 1 KiB to
64 KiB, so a version 1 database cannot be reused: it is rejected and
the tree must be rescanned from scratch.
```sql
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256, first 1 KiB
tail TEXT NOT NULL -- lowercase-hex SHA-256, last 1 KiB
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256, first 64 KiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256, last 64 KiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
) WITHOUT ROWID;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not
guaranteed UTF-8. `mtime` is used only for change detection; it is
not part of the duplicate key. `head` and `tail` are empty strings
when the file has never been hashed because its size was unique as
of the last scan that covered it; such records still define the
file for tree reconstruction but never participate in duplicate
groups.
not part of the duplicate key. `head`, `tail`, and `content` are
empty strings when the file has never been hashed because its size
was unique as of the last scan that covered it; such records still
define the file for tree reconstruction but never participate in
duplicate groups.
### Duplicate detection
Two files are duplicates only when they agree on every rung of this
ladder; a mismatch at any rung means they are not duplicates. `scan`
stores each file's hashes once, and `report` and `trees` group files by
the whole signature — size, `head`, `tail`, and `content` — so the
grouping is exactly this ladder applied across everything scanned into
the database, even across separate scans.
1. **Size.** Files of different sizes are never compared. Only files
whose size at least one other file shares are hashed at all.
2. **Head and tail.** The SHA-256 of the first 64 KiB (`head`) and of
the last 64 KiB (`tail`). When a file is 64 KiB or smaller the two
windows are the whole file and coincide, so `head` and `tail` are
equal and only one read is issued; when it is between one and two
windows the two windows overlap, which is harmless. These reads are
cheap and eliminate most same-size pairs before any bulk reading.
3. **Content, below 50 MiB.** The SHA-256 of the entire file. Agreement
here is proof of identical content (barring a SHA-256 collision).
4. **Content, 50 MiB and above.** A sampled SHA-256: the 1 MiB window
at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the
file, the final window truncated at end of file) is fed, in order,
into one hash. This is **deliberately probabilistic** — the gaps
between samples are never read, so two large files that agree on
every sample are reported as duplicates without being read in full.
It is the price of never reading a 150 GB file end to end. Because
size is already part of the signature, only equal-size files reach
this rung, so their sample boundaries always align.
`head`, `tail`, and `content` are one column each; a file below 50 MiB
and a file at or above it never share a size, so a `content` value is
never ambiguous between the whole-file and sampled forms.
### `scan` mode
@@ -229,10 +276,9 @@ during the hash phase:
decides its fate. Size-unique files are never read: new or
changed ones are recorded without hashes in the update phase,
unchanged unhashed ones simply keep their records. Every file
with a shared size is hashed by the worker pool: read the first
`min(1024, size)` bytes and the last `min(1024, size)` bytes
(one read when `size <= 1024`, since the two windows coincide)
and compute the SHA-256 of each. Zero-length files have constant
with a shared size is hashed by the worker pool, computing the
full signature — head, tail, and content — described in "Duplicate
detection" below. Zero-length files have constant
hashes and are never opened. Files are hashed in **inode order**
(minimizing seeks on spinning disks), and paths that are hard
links to the same inode are **read once**, all sharing the one