sneak/sfdupes
1
0
Fork 0
Code Issues 27 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Hash files under 10 MiB in full, gate larger files on head/tail
check / check (push) Successful in 59s
Details

Browse Source 
Amends the issue 61 ladder per the owner's design change. A file under
10 MiB (new headTailMin) is now hashed in full and compared directly,
with no end-window step: its head, tail, and content all carry the
whole-file SHA-256, so its signature is decided by size and content
alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then
the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB
samples at or above. hashEnds drops its now-unreachable single-window
branch, and hashWhole errors if the file shrank below its recorded
size (the only read for the sub-10-MiB range). README, TODO, tests,
and the schema-version note updated to match.

Model: opus-4-8
This commit is contained in:
sneak
2026-09-22 14:22:17 +00:00
parent b80c7e805e
commit dc30ad9f31
5 changed files with 203 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+49 -37
View File
@@ -7,12 +7,14 @@
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading every byte of every file.
Files are considered duplicates when their sizes are equal and they
agree on a short ladder of hashes: the SHA-256 of their first 64 KiB and
of their last 64 KiB, and then a content hash — the SHA-256 of the whole
file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when it
is 50 MiB or larger. Below 50 MiB this is proof of identical content; at
or above 50 MiB it is a strong candidate signal rather than proof,
because the gaps between samples are never read. The intended use is
agree on a short ladder of hashes. A file under 10 MiB is hashed in full
and compared directly. A larger file is gated first on the SHA-256 of
its first 64 KiB and of its last 64 KiB, and then compared on a content
hash — the SHA-256 of the whole file when it is under 50 MiB, or of
gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
the content hash is proof of identical content; at or above 50 MiB it is
a strong candidate signal rather than proof, because the gaps between
samples are never read. The intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte of every file is
prohibitively expensive. `scan` maintains a persistent SQLite database of file
@@ -78,7 +80,8 @@ Goals, in order:
2. **Spend I/O in proportion to duplicate likelihood.** Only files
whose size at least one other file shares are read at all — a
size-unique file cannot be a duplicate. Those are compared by the
ladder in "Duplicate detection" below: cheap 64 KiB end windows
ladder in "Duplicate detection" below: a file under 10 MiB is hashed
in full, while a larger file is gated on cheap 64 KiB end windows
first, then a content hash that reads the whole file below 50 MiB
but only gigabyte-spaced 1 MiB samples at or above it, so the very
largest files are still never read in full. Scale target: tens of
@@ -153,28 +156,31 @@ All three subcommands operate on a single SQLite database file:
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 2; a
database with any other version is a fatal error). Version 2 added
the `content` column and widened the end windows from 1 KiB to
64 KiB, so a version 1 database cannot be reused: it is rejected and
the tree must be rescanned from scratch.
the `content` column and the 64 KiB head/tail signature (replacing
the version 1 1 KiB end windows), so a version 1 database cannot be
reused: it is rejected and the tree must be rescanned from scratch.
```sql
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256, first 64 KiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256, last 64 KiB
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
) WITHOUT ROWID;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not
guaranteed UTF-8. `mtime` is used only for change detection; it is
not part of the duplicate key. `head`, `tail`, and `content` are
empty strings when the file has never been hashed because its size
was unique as of the last scan that covered it; such records still
define the file for tree reconstruction but never participate in
duplicate groups.
not part of the duplicate key. For a file under 10 MiB `head`, `tail`,
and `content` all hold the whole-file hash (that range is hashed in
full, with no end windows); for a larger file `head` and `tail` hold
the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never
been hashed because its size was unique as of the last scan that
covered it; such records still define the file for tree
reconstruction but never participate in duplicate groups.
### Duplicate detection
@@ -187,27 +193,33 @@ the database, even across separate scans.
1. **Size.** Files of different sizes are never compared. Only files
whose size at least one other file shares are hashed at all.
2. **Head and tail.** The SHA-256 of the first 64 KiB (`head`) and of
the last 64 KiB (`tail`). When a file is 64 KiB or smaller the two
windows are the whole file and coincide, so `head` and `tail` are
equal and only one read is issued; when it is between one and two
windows the two windows overlap, which is harmless. These reads are
cheap and eliminate most same-size pairs before any bulk reading.
3. **Content, below 50 MiB.** The SHA-256 of the entire file. Agreement
here is proof of identical content (barring a SHA-256 collision).
4. **Content, 50 MiB and above.** A sampled SHA-256: the 1 MiB window
at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the
file, the final window truncated at end of file) is fed, in order,
into one hash. This is **deliberately probabilistic** — the gaps
between samples are never read, so two large files that agree on
every sample are reported as duplicates without being read in full.
It is the price of never reading a 150 GB file end to end. Because
size is already part of the signature, only equal-size files reach
this rung, so their sample boundaries always align.
2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed
in full and compared directly, with no separate end-window step —
small files are cheap to read to the last byte, and doing so makes
the comparison exact. `head`, `tail`, and `content` all hold this
whole-file SHA-256, so such a file's signature is decided entirely
by its size and its content.
3. **10 MiB and above: head and tail.** For a larger file, the SHA-256
of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
cheap gate that eliminates most same-size pairs before any bulk
reading. At 10 MiB and above the two windows never overlap.
4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the
entire file. Agreement here is proof of identical content (barring a
SHA-256 collision).
5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256:
the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, …
while inside the file, the final window truncated at end of file) is
fed, in order, into one hash. This is **deliberately probabilistic**
— the gaps between samples are never read, so two large files that
agree on every sample are reported as duplicates without being read
in full. It is the price of never reading a 150 GB file end to end.
Because size is already part of the signature, only equal-size files
reach this rung, so their sample boundaries always align.
`head`, `tail`, and `content` are one column each; a file below 50 MiB
and a file at or above it never share a size, so a `content` value is
never ambiguous between the whole-file and sampled forms.
`head`, `tail`, and `content` are one column each. A file below 10 MiB
and one at or above it never share a size, and neither do a file below
50 MiB and one at or above it, so a stored value is never ambiguous
between the whole-file, end-window, and sampled forms.
### `scan` mode
TODO.md
+12 -10
View File
@@ -29,17 +29,19 @@
# Completed Steps
- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
- replace the 1 KiB end-window sampling with the head/tail plus
content-hash ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
1 MiB samples at or above — and the end windows widen from 1 KiB to
64 KiB. Schema bumps to version 2 (new `content` column); a version 1
database is rejected and must be rescanned, which is required anyway
since every stored hash changed. `report` and `trees` group by the
extended signature, so the ladder is applied across the whole
database. README "Duplicate detection" documents every rung including
the probabilistic large-file path.
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
hashed in full and compared directly, with no end-window step — its
`head`, `tail`, and `content` all hold the whole-file hash. A file at
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
compared on a `content` hash — the whole file below 50 MiB,
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
(new `content` column); a version 1 database is rejected and must be
rescanned, which is required anyway since every stored hash changed.
`report` and `trees` group by the extended signature, so the ladder is
applied across the whole database. README "Duplicate detection"
documents every rung including the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes
db.go
+3 -2
View File
@@ -26,8 +26,9 @@ const databaseEnv = "SFDUPES_DATABASE"
// schemaVersion is the database schema version this build reads and
// writes, stored in PRAGMA user_version. Version 2 adds the content
// column and stores 65 KiB (rather than 1 KiB) end-window hashes, so a
// version 1 database is rejected and must be rescanned.
// column and the head/tail/content signature (replacing the version 1
// 1 KiB end windows), so a version 1 database is rejected and must be
// rescanned.
const schemaVersion = 2
// dbDirPerm is the mode for a database parent directory created by
scan.go
+51 -28
View File
@@ -18,13 +18,22 @@ import (
)
// The duplicate ladder (see hashSignature and README "Duplicate
// detection"). Same-size candidates are separated first by the hashes
// of their end windows, then by a content hash that is exact for
// smaller files and deliberately sampled for large ones.
// detection"). A same-size candidate below headTailMin is hashed in
// full and compared directly; a larger one is separated first by the
// hashes of its end windows, then by a content hash that is exact below
// wholeFileMax and deliberately sampled at or above it.
// headTailMin is the size threshold for the end-window gate. A file
// smaller than this is hashed in full directly, with no separate head
// and tail step: its head, tail, and content all carry the whole-file
// hash. A file this size or larger is separated first by its end
// windows.
const headTailMin = 10 * 1024 * 1024
// headTailWindow is the number of bytes hashed from each end of a file
// (the head and tail rungs). A file no larger than one window has head
// and tail equal to the hash of its whole content.
// at or above headTailMin (the head and tail rungs). Because
// headTailMin is far larger than two windows, the head and tail windows
// never overlap.
const headTailWindow = 64 * 1024
// wholeFileMax is the size boundary between the two content rungs: a
@@ -972,14 +981,17 @@ const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
"27ae41e4649b934ca495991b7852b855"
// hashSignature computes the three content hashes that, with the file
// size, form its duplicate signature: the SHA-256 of the first and last
// headTailWindow bytes (the head and tail rungs), and a content hash
// that is the SHA-256 of the whole file below wholeFileMax (the exact
// rung) or of gigabyte-spaced samples at or above it (the sampled,
// deliberately probabilistic rung). Two files are duplicates only when
// all four agree; any mismatch means not a duplicate. size is the value
// recorded when the file was statted; a zero-length file has constant
// hashes and is never opened.
// size, form its duplicate signature. A file below headTailMin is
// hashed in full and its whole-file SHA-256 is returned as head, tail,
// and content alike — that range takes no separate end-window step. For
// a file at or above headTailMin the head and tail are the SHA-256 of
// its first and last headTailWindow bytes, and content is the SHA-256
// of the whole file below wholeFileMax (the exact rung) or of
// gigabyte-spaced samples at or above it (the sampled, deliberately
// probabilistic rung). Two files are duplicates only when all four
// agree; any mismatch means not a duplicate. size is the value recorded
// when the file was statted; a zero-length file has constant hashes and
// is never opened.
func hashSignature(path string, size int64) (string, string, string, error) {
if size == 0 {
return emptyHash, emptyHash, emptyHash, nil
@@ -993,6 +1005,17 @@ func hashSignature(path string, size int64) (string, string, string, error) {
defer func() { _ = f.Close() }()
// Below the threshold the whole file is hashed directly, with no
// end-window step: head and tail both carry the whole-file hash.
if size < int64(headTailMin) {
content, err := hashWhole(f, size)
if err != nil {
return "", "", "", err
}
return content, content, content, nil
}
head, tail, err := hashEnds(f, size)
if err != nil {
return "", "", "", err
@@ -1007,14 +1030,11 @@ func hashSignature(path string, size int64) (string, string, string, error) {
}
// hashEnds returns the SHA-256 of the first and last headTailWindow
// bytes of f. The two windows overlap when the file is between one and
// two windows in size; when it is no larger than one window they
// coincide, so the head hash is reused as the tail and only one read is
// issued.
// bytes of f. It is called only for files at least headTailMin, which
// is far larger than two windows, so the windows never overlap and both
// reads are always full.
func hashEnds(f *os.File, size int64) (string, string, error) {
n := min(int64(headTailWindow), size)
buf := make([]byte, n)
buf := make([]byte, headTailWindow)
_, err := f.ReadAt(buf, 0)
if err != nil {
@@ -1024,11 +1044,7 @@ func hashEnds(f *os.File, size int64) (string, string, error) {
h := sha256.Sum256(buf)
head := hex.EncodeToString(h[:])
if size <= int64(headTailWindow) {
return head, head, nil
}
_, err = f.ReadAt(buf, size-n)
_, err = f.ReadAt(buf, size-int64(headTailWindow))
if err != nil {
return "", "", err
}
@@ -1050,16 +1066,23 @@ func hashContent(f *os.File, size int64) (string, error) {
}
// hashWhole returns the SHA-256 of the entire file. A SectionReader is
// used so the read is independent of the offset left by the end-window
// reads.
// used so the read is independent of the offset left by any end-window
// reads. Reading fewer than size bytes means the file shrank between
// the stat and the hash; that is an error rather than a hash of content
// that no longer matches the recorded size.
func hashWhole(f *os.File, size int64) (string, error) {
h := sha256.New()
_, err := io.Copy(h, io.NewSectionReader(f, 0, size))
n, err := io.Copy(h, io.NewSectionReader(f, 0, size))
if err != nil {
return "", err
}
if n != size {
return "", fmt.Errorf("read %d of %d bytes: %w", n, size,
io.ErrUnexpectedEOF)
}
return hex.EncodeToString(h.Sum(nil)), nil
}
scan_test.go
+88 -30
View File
@@ -53,10 +53,23 @@ func pattern(tag byte, n int) []byte {
return data
}
// TestHashSignatureEnds exercises the head and tail rungs across the
// window boundaries. Every file here is below wholeFileMax, so the
// content rung is a whole-file hash.
func TestHashSignatureEnds(t *testing.T) {
// sig returns a file's full signature (head, tail, content), failing the
// test on any error.
func sig(t *testing.T, path string, size int64) (string, string, string) {
t.Helper()
head, tail, content, err := hashSignature(path, size)
if err != nil {
t.Fatalf("hashSignature %s: %v", path, err)
}
return head, tail, content
}
// TestHashSignatureBelowThreshold verifies that a file below headTailMin
// is hashed in full and compared directly: head, tail, and content all
// carry the whole-file SHA-256, with no separate end-window step.
func TestHashSignatureBelowThreshold(t *testing.T) {
t.Parallel()
dir := t.TempDir()
@@ -65,13 +78,10 @@ func TestHashSignatureEnds(t *testing.T) {
name string
data []byte
}{
{"empty", nil},
{"one-byte", []byte("x")},
{"under-one-window", pattern(1, headTailWindow-1)},
{"exactly-one-window", pattern(2, headTailWindow)},
{"overlapping-windows", pattern(3, headTailWindow+headTailWindow/2)},
{"exactly-two-windows", pattern(4, 2*headTailWindow)},
{"beyond-two-windows", pattern(5, 3*headTailWindow)},
{"one-window", pattern(1, headTailWindow)},
{"several-windows", pattern(2, 3*headTailWindow)},
{"near-threshold", pattern(3, headTailMin-1)},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -79,28 +89,79 @@ func TestHashSignatureEnds(t *testing.T) {
p := writeFile(t, dir, c.name, c.data)
head, tail, content, err := hashSignature(p, int64(len(c.data)))
if err != nil {
t.Fatalf("hashSignature: %v", err)
}
head, tail, content := sig(t, p, int64(len(c.data)))
n := min(headTailWindow, len(c.data))
if want := hexSum(c.data[:n]); head != want {
t.Errorf("head = %s, want %s", head, want)
}
if want := hexSum(c.data[len(c.data)-n:]); tail != want {
t.Errorf("tail = %s, want %s", tail, want)
}
// Below wholeFileMax the content rung hashes the whole file.
if want := hexSum(c.data); content != want {
t.Errorf("content = %s, want whole-file %s", content, want)
whole := hexSum(c.data)
if head != whole || tail != whole || content != whole {
t.Errorf("head=%s tail=%s content=%s, want all whole-file %s",
head, tail, content, whole)
}
})
}
}
// TestHashSignatureEnds exercises the head and tail rungs, which apply
// only to files at least headTailMin. Sparse files keep the fixtures
// cheap: a difference in the first window changes only head, a
// difference in the last window changes only tail, and a difference
// between the windows changes neither end hash but does change the
// whole-file content rung (the file is below wholeFileMax).
func TestHashSignatureEnds(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// Between headTailMin and wholeFileMax: the end-window gate is active
// and the content rung is a whole-file hash.
const size = int64(headTailMin + 2*1024*1024)
base := sparseFile(t, dir, "ends-base", size)
headDiff := sparseFile(t, dir, "ends-head", size)
tailDiff := sparseFile(t, dir, "ends-tail", size)
midDiff := sparseFile(t, dir, "ends-mid", size)
pokeAt(t, headDiff, 0, []byte{1})
pokeAt(t, tailDiff, size-1, []byte{1})
pokeAt(t, midDiff, size/2, []byte{1})
bHead, bTail, bContent := sig(t, base, size)
h, tl, c := sig(t, headDiff, size)
if h == bHead {
t.Error("a byte in the first window did not change head")
}
if tl != bTail {
t.Error("a byte in the first window changed tail")
}
if c == bContent {
t.Error("a byte in the first window did not change content")
}
h, tl, c = sig(t, tailDiff, size)
if tl == bTail {
t.Error("a byte in the last window did not change tail")
}
if h != bHead {
t.Error("a byte in the last window changed head")
}
if c == bContent {
t.Error("a byte in the last window did not change content")
}
h, tl, c = sig(t, midDiff, size)
if h != bHead || tl != bTail {
t.Error("a byte between the windows changed an end hash")
}
if c == bContent {
t.Error("whole-file content rung ignored a byte between the windows")
}
}
func TestHashSignatureErrors(t *testing.T) {
t.Parallel()
@@ -189,10 +250,7 @@ func pokeAt(t *testing.T, path string, off int64, data []byte) {
func contentHash(t *testing.T, path string, size int64) string {
t.Helper()
_, _, content, err := hashSignature(path, size)
if err != nil {
t.Fatalf("hashSignature %s: %v", path, err)
}
_, _, content := sig(t, path, size)
return content
}