sneak/sfdupes
1
0
Fork 0
Code Issues 29 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Compute the content hash only when head and tail match (closes #61)
check / check (push) Successful in 1m9s
Details

Browse Source 
A file of 10 MiB or more now gets only its head and tail in the hash
phase. A new content phase, after the update phase, finds every record
of that size without a content hash whose size, head and tail match
another record's, anywhere in the database, checks each file with
lstat, and reads a group only while at least two members remain. It
reuses the hash worker pool, now given its hash function. report and
trees leave out records without a content hash. The README, help text
and TODO entry describe the gate; the schema stays at version 1.

Model: opus-5-5
This commit is contained in:
clawbot
2026-09-23 12:18:39 +00:00
parent 09a39ddf37
commit 89fc9e4595
12 changed files with 763 additions and 221 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+107 -60
View File
@@ -9,8 +9,9 @@ across very large filesystems without reading every byte of every file.
Files are considered duplicates when their sizes are equal and they
agree on a short ladder of hashes. A file under 10 MiB is hashed in full
and compared directly. A larger file is gated first on the SHA-256 of
its first 64 KiB and of its last 64 KiB, and then compared on a content
hash — the SHA-256 of the whole file when it is under 50 MiB, or of
its first 64 KiB and of its last 64 KiB, and only when its size and
both of those match another file's is it read for a content hash to
compare — the SHA-256 of the whole file when it is under 50 MiB, or of
gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
the content hash is proof of identical content; at or above 50 MiB it is
a strong candidate signal rather than proof, because the gaps between
@@ -37,7 +38,8 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
record per regular file (path, size, mtime, head hash, tail hash,
content hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
or changed, or that may have gained a duplicate since the last scan,
and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
report. `trees` reads the same database and prints the duplicate-tree
report. A missing/invalid subcommand — or a `scan` invocation with no
@@ -55,14 +57,17 @@ Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
whose size at least one other file shares, since a size-unique file
cannot be a duplicate; for those it reads the cheap end windows first
and a content hash second — the whole file below 50 MiB, but only
gigabyte-spaced samples at or above 50 MiB, so the largest files are
never read in full. This keeps a full-filesystem sweep tractable, and
the signatures are kept in a persistent database, so
the expensive filesystem pass is incremental: a rescan re-hashes only
files whose recorded mtime or size changed, and all analysis happens
offline from the database alone. The end goal is
cannot be a duplicate. Of those, a file under 10 MiB is read in full; a
larger one has its cheap end windows read first, and is read for a
content hash only when its size and both end windows match another
file's — the whole file below 50 MiB, but only gigabyte-spaced samples
at or above 50 MiB, so the largest files are never read in full. This
keeps a full-filesystem sweep tractable, and the signatures are kept in
a persistent database, so the expensive filesystem pass is incremental:
a rescan re-hashes only files whose recorded mtime or size changed, plus
— for its content hash — a file of 10 MiB or more whose size and end
windows have come to match another file's. All analysis happens offline
from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.
@@ -82,16 +87,19 @@ Goals, in order:
size-unique file cannot be a duplicate. Those are compared by the
ladder in "Duplicate detection" below: a file under 10 MiB is hashed
in full, while a larger file is gated on cheap 64 KiB end windows
first, then a content hash that reads the whole file below 50 MiB
but only gigabyte-spaced 1 MiB samples at or above it, so the very
largest files are still never read in full. Scale target: tens of
first, and gets a content hash only when its size and both end
windows match another file's. That hash reads the whole file below
50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so the
very largest files are still never read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks
(ZFS pool under resilver). Holding one small record (path, size,
mtime) per file in memory during a scan is acceptable; holding
every file's hashes is not (they stay in the database).
3. **Scan incrementally, analyze offline.** The expensive filesystem
scan maintains a persistent database; an unchanged file is never
read again on a rescan. All analysis (`report`, `trees`) works from
read again on a rescan, except to compute its content hash once a
file of 10 MiB or more comes to match another on size and both end
windows. All analysis (`report`, `trees`) works from
the database alone and must never touch the scanned filesystem
again. `scan` is designed to be cronned; the reports run at any
time against the last completed scan.
@@ -176,17 +184,20 @@ All three subcommands operate on a single SQLite database file:
the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never
been hashed because its size was unique as of the last scan that
covered it; such records still define the file for tree
reconstruction but never participate in duplicate groups.
covered it. `content` alone is empty for a file of 10 MiB or more
that matches no other record on size, `head`, and `tail` yet, or
whose content read failed. A record with an empty `content` still
defines the file for tree reconstruction but is not a duplicate
until a later scan fills it in.
### Duplicate detection
Two files are duplicates only when they agree on every rung of this
ladder; a mismatch at any rung means they are not duplicates. `scan`
stores each file's hashes once, and `report` and `trees` group files by
the whole signature — size, `head`, `tail`, and `content` — so the
grouping is exactly this ladder applied across everything scanned into
the database, even across separate scans.
stores each file's hashes, and `report` and `trees` group files by the
whole signature — size, `head`, `tail`, and `content` — so the grouping
is exactly this ladder applied across everything scanned into the
database, even across separate scans.
1. **Size.** Files of different sizes are never compared. Only files
whose size at least one other file shares are hashed at all.
@@ -199,7 +210,13 @@ the database, even across separate scans.
3. **10 MiB and above: head and tail.** For a larger file, the SHA-256
of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
cheap gate that eliminates most same-size pairs before any bulk
reading. At 10 MiB and above the two windows never overlap.
reading: the content hash of the next two rungs is computed only for
a file whose size, `head`, and `tail` match another file's, whether
that file is scanned in the same run or stored by an earlier scan.
A stored file that first gains such a match in a later scan gets its
content hash then; until it has one, its `content` is empty and it
is not a duplicate. At 10 MiB and above the two windows never
overlap.
4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the
entire file. Agreement here is proof of identical content (barring a
SHA-256 collision).
@@ -239,10 +256,10 @@ scanned operands:
- Only a file whose size at least one other file shares is ever
read: a size-unique file cannot be a duplicate, so it is recorded
without hashes (`head` and `tail` empty). The size census covers
every file walked this scan plus every database record outside
the scanned operands, so a possible duplicate of a separately
scanned tree is still recognized.
without hashes (`head`, `tail`, and `content` empty). The size
census covers every file walked this scan plus every database
record outside the scanned operands, so a possible duplicate of a
separately scanned tree is still recognized.
- A file not yet in the database is inserted: hashed when its size
is shared, without hashes otherwise.
- A file already in the database is **skipped without reading its
@@ -251,7 +268,10 @@ scanned operands:
makes a daily rescan cheap. Exception: an unchanged file whose
record lacks hashes is hashed — and its record updated — once its
size becomes shared, so hashing deferred by size-uniqueness
happens as soon as it could matter.
happens as soon as it could matter. Likewise, an unchanged file of
10 MiB or more whose record has no `content` hash is read for one
by the content phase below once its size, `head`, and `tail` match
another record's.
- A file whose mtime is newer than recorded, or whose size differs,
is processed as if new: re-hashed, or recorded without hashes,
per the shared-size rule.
@@ -260,12 +280,18 @@ scanned operands:
removes records for deleted files. It also removes records for
paths that failed to stat or hash this run: the database only ever
contains signatures verified by the most recent scan that covered
them (a subsequent successful scan re-adds such files).
them (a subsequent successful scan re-adds such files). A failed
content read in the content phase below removes nothing: the
record keeps its `head` and `tail`, with `content` empty.
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database.
database. The one exception is the content phase below: a stored
file of 10 MiB or more without a `content` hash is read for one,
wherever it lies, once its size, `head`, and `tail` match another
record's. If that file is gone or has changed since its record was
written, the record is left as it is.
`scan` runs **three sequential phases over the whole scan**.
`scan` runs **four sequential phases over the whole scan**.
Parallelism lives inside each phase; batched database writes begin
during the hash phase:
@@ -285,11 +311,13 @@ during the hash phase:
decides its fate. Size-unique files are never read: new or
changed ones are recorded without hashes in the update phase,
unchanged unhashed ones simply keep their records. Every file
with a shared size is hashed by the worker pool, computing the
full signature — head, tail, and content — described in "Duplicate
detection" below. Zero-length files have constant
hashes and are never opened. Files are hashed in **inode order**
(minimizing seeks on spinning disks), and paths that are hard
with a shared size is hashed by the worker pool as described in
"Duplicate detection" above: a file under 10 MiB in full, which
gives its `head`, `tail`, and `content` alike, and a larger file
only in its end windows, which give its `head` and `tail`; its
content hash is left to the content phase. Zero-length files have
constant hashes and are never opened. Files are hashed in **inode
order** (minimizing seeks on spinning disks), and paths that are hard
links to the same inode are **read once**, all sharing the one
result — a hard-link backup farm costs one read per inode, not
per path. The phase total counts actual reads, so progress and
@@ -301,6 +329,23 @@ during the hash phase:
records for size-unique new and changed files, and the deletions
for records the scan did not verify (vanished files, plus paths
that failed to stat or hash).
4. **content** — find every record of 10 MiB or more without a
`content` hash whose size, `head`, and `tail` equal another
record's, anywhere in the database: records from this scan and
records stored by earlier scans, inside or outside the scanned
operands. SQLite finds them, so only their records are loaded into
memory, never every file's hashes. Each such file is checked with
`lstat` first; one that is gone, is no longer a regular file, or
has changed (a different size, or an mtime newer than recorded)
keeps its record as it is and is not a duplicate. The files that
pass are read only if at least two records sharing their size,
`head`, and `tail` remain, counting those that already have a
`content` hash, so a file whose only matches are stale costs no
read. They are read by a worker pool as in the hash phase, in inode
order and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its
record keeps an empty `content`, so it is not a duplicate, and a
later scan tries again.
Rules for the walk:
@@ -318,18 +363,18 @@ Rules for the walk:
path, and continue. Per-file errors never abort the run; the final
summary reports how many were skipped. As specified above, a
skipped path that has a database record from an earlier scan loses
that record; an unreadable directory subtree likewise loses its
records (accepted: the database mirrors what the latest scan could
actually verify).
that record, unless only its content read failed; an unreadable
directory subtree likewise loses its records (accepted: the
database mirrors what the latest scan could actually verify).
Concurrency: the walk phase (which also stats files) and the hash
phase each use a worker pool of `--workers` workers (default
`runtime.NumCPU()`); the walk parallelizes across directories,
hashing across files. Both phases are seek-bound on spinning disks,
so raising `--workers` well past the core count can help on pools
with many spindles. The main goroutine owns partitioning, database
writes, and progress rendering; progress display must never block
the workers.
Concurrency: the walk phase (which also stats files), the hash phase,
and the content phase each use a worker pool of `--workers` workers
(default `runtime.NumCPU()`); the walk parallelizes across
directories, hashing across files. All three phases are seek-bound on
spinning disks, so raising `--workers` well past the core count can
help on pools with many spindles. The main goroutine owns
partitioning, database writes, and progress rendering; progress
display must never block the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
@@ -354,11 +399,12 @@ mounted.
Processing:
- Records without hashes (size-unique when last scanned) are
excluded: their content is unknown, so they are never reported as
duplicates.
- Records without a `content` hash (size-unique when last scanned,
or 10 MiB or more and not yet matched on size, `head`, and `tail`)
are excluded: their content is unknown, so they are never reported
as duplicates.
- Group the remaining records by the key
`(size, head_hash, tail_hash)`.
`(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The
first path is the group's `first`; every other path is a `dupe`.
@@ -394,11 +440,11 @@ the paths in the records, split on `/`.
Definitions:
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
informational and excluded. An unhashed record (empty hashes) has
- A file's **signature** is `(size, head, tail, content)` — mtime is
informational and excluded. A record without a `content` hash has
unknown content: its signature is treated as unique to that file,
so a tree containing an unhashed file never compares equal to any
other tree.
so a tree containing such a file never compares equal to any other
tree.
- A directory's **digest** is a SHA-256 Merkle digest computed
bottom-up: serialize the directory's child entries — for a file
child, its name and signature; for a subdirectory child, its name
@@ -461,10 +507,10 @@ Each phase gets its own display, rendered the moment the phase
starts — a scan must never look hung. Loading the existing-record
index (`load`) and the walk have no known totals while running: show
a live count, rate, and elapsed time (spinner-style, no percentage or
ETA). The hash and update phases
ETA). The hash, update, and content phases
have exact totals — only files that actually need hashing appear in
the hash total, so its ETA is meaningful. Required elements for the
bars with known totals:
the hash and content totals, so their ETAs are meaningful. Required
elements for the bars with known totals:
- elapsed time
- estimated time remaining
@@ -691,9 +737,10 @@ Tracked in [TODO.md](TODO.md).
## Non-goals
- No full-content verification, no byte-for-byte compare, no deletion
or linking of duplicates. The reports are advisory; acting on them is
the user's job.
- No byte-for-byte compare, and no deletion or linking of
duplicates. Files that match are compared by a SHA-256 of the whole
file below 50 MiB, and only by samples at 50 MiB and over. The
reports are advisory; acting on them is the user's job.
- No persistence beyond the SQLite database described above; no
export/import formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job.