diff --git a/README.md b/README.md index 848af91..708ebc6 100644 --- a/README.md +++ b/README.md @@ -7,12 +7,14 @@ duplicate files — and, ultimately, entire duplicate directory trees — across very large filesystems without reading every byte of every file. Files are considered duplicates when their sizes are equal and they -agree on a short ladder of hashes: the SHA-256 of their first 64 KiB and -of their last 64 KiB, and then a content hash — the SHA-256 of the whole -file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when it -is 50 MiB or larger. Below 50 MiB this is proof of identical content; at -or above 50 MiB it is a strong candidate signal rather than proof, -because the gaps between samples are never read. The intended use is +agree on a short ladder of hashes. A file under 10 MiB is hashed in full +and compared directly. A larger file is gated first on the SHA-256 of +its first 64 KiB and of its last 64 KiB, and then compared on a content +hash — the SHA-256 of the whole file when it is under 50 MiB, or of +gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB +the content hash is proof of identical content; at or above 50 MiB it is +a strong candidate signal rather than proof, because the gaps between +samples are never read. The intended use is finding duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers where reading every byte of every file is prohibitively expensive. `scan` maintains a persistent SQLite database of file @@ -78,7 +80,8 @@ Goals, in order: 2. **Spend I/O in proportion to duplicate likelihood.** Only files whose size at least one other file shares are read at all — a size-unique file cannot be a duplicate. Those are compared by the - ladder in "Duplicate detection" below: cheap 64 KiB end windows + ladder in "Duplicate detection" below: a file under 10 MiB is hashed + in full, while a larger file is gated on cheap 64 KiB end windows first, then a content hash that reads the whole file below 50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so the very largest files are still never read in full. Scale target: tens of @@ -153,28 +156,31 @@ All three subcommands operate on a single SQLite database file: toward the filesystem. - Schema (`PRAGMA user_version` is the schema version, currently 2; a database with any other version is a fatal error). Version 2 added - the `content` column and widened the end windows from 1 KiB to - 64 KiB, so a version 1 database cannot be reused: it is rejected and - the tree must be rescanned from scratch. + the `content` column and the 64 KiB head/tail signature (replacing + the version 1 1 KiB end windows), so a version 1 database cannot be + reused: it is rejected and the tree must be rescanned from scratch. ```sql CREATE TABLE files ( path BLOB PRIMARY KEY, -- absolute path, raw bytes size INTEGER NOT NULL, -- bytes, from lstat mtime INTEGER NOT NULL, -- Unix seconds, from lstat - head TEXT NOT NULL, -- lowercase-hex SHA-256, first 64 KiB - tail TEXT NOT NULL, -- lowercase-hex SHA-256, last 64 KiB + head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB + tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples ) WITHOUT ROWID; ``` Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. `mtime` is used only for change detection; it is - not part of the duplicate key. `head`, `tail`, and `content` are - empty strings when the file has never been hashed because its size - was unique as of the last scan that covered it; such records still - define the file for tree reconstruction but never participate in - duplicate groups. + not part of the duplicate key. For a file under 10 MiB `head`, `tail`, + and `content` all hold the whole-file hash (that range is hashed in + full, with no end windows); for a larger file `head` and `tail` hold + the first- and last-64 KiB hashes and `content` the whole-file or + sampled hash. All three are empty strings when the file has never + been hashed because its size was unique as of the last scan that + covered it; such records still define the file for tree + reconstruction but never participate in duplicate groups. ### Duplicate detection @@ -187,27 +193,33 @@ the database, even across separate scans. 1. **Size.** Files of different sizes are never compared. Only files whose size at least one other file shares are hashed at all. -2. **Head and tail.** The SHA-256 of the first 64 KiB (`head`) and of - the last 64 KiB (`tail`). When a file is 64 KiB or smaller the two - windows are the whole file and coincide, so `head` and `tail` are - equal and only one read is issued; when it is between one and two - windows the two windows overlap, which is harmless. These reads are - cheap and eliminate most same-size pairs before any bulk reading. -3. **Content, below 50 MiB.** The SHA-256 of the entire file. Agreement - here is proof of identical content (barring a SHA-256 collision). -4. **Content, 50 MiB and above.** A sampled SHA-256: the 1 MiB window - at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the - file, the final window truncated at end of file) is fed, in order, - into one hash. This is **deliberately probabilistic** — the gaps - between samples are never read, so two large files that agree on - every sample are reported as duplicates without being read in full. - It is the price of never reading a 150 GB file end to end. Because - size is already part of the signature, only equal-size files reach - this rung, so their sample boundaries always align. +2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed + in full and compared directly, with no separate end-window step — + small files are cheap to read to the last byte, and doing so makes + the comparison exact. `head`, `tail`, and `content` all hold this + whole-file SHA-256, so such a file's signature is decided entirely + by its size and its content. +3. **10 MiB and above: head and tail.** For a larger file, the SHA-256 + of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a + cheap gate that eliminates most same-size pairs before any bulk + reading. At 10 MiB and above the two windows never overlap. +4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the + entire file. Agreement here is proof of identical content (barring a + SHA-256 collision). +5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256: + the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … + while inside the file, the final window truncated at end of file) is + fed, in order, into one hash. This is **deliberately probabilistic** + — the gaps between samples are never read, so two large files that + agree on every sample are reported as duplicates without being read + in full. It is the price of never reading a 150 GB file end to end. + Because size is already part of the signature, only equal-size files + reach this rung, so their sample boundaries always align. -`head`, `tail`, and `content` are one column each; a file below 50 MiB -and a file at or above it never share a size, so a `content` value is -never ambiguous between the whole-file and sampled forms. +`head`, `tail`, and `content` are one column each. A file below 10 MiB +and one at or above it never share a size, and neither do a file below +50 MiB and one at or above it, so a stored value is never ambiguous +between the whole-file, end-window, and sampled forms. ### `scan` mode diff --git a/TODO.md b/TODO.md index fb8e9ca..744f174 100644 --- a/TODO.md +++ b/TODO.md @@ -29,17 +29,19 @@ # Completed Steps -- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus +- replace the 1 KiB end-window sampling with the head/tail plus content-hash ladder (2026-09-22, branch `next`, closes - https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature - gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced - 1 MiB samples at or above — and the end windows widen from 1 KiB to - 64 KiB. Schema bumps to version 2 (new `content` column); a version 1 - database is rejected and must be rescanned, which is required anyway - since every stored hash changed. `report` and `trees` group by the - extended signature, so the ladder is applied across the whole - database. README "Duplicate detection" documents every rung including - the probabilistic large-file path. + https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is + hashed in full and compared directly, with no end-window step — its + `head`, `tail`, and `content` all hold the whole-file hash. A file at + 10 MiB or above is gated on the 64 KiB `head` and `tail`, then + compared on a `content` hash — the whole file below 50 MiB, + gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2 + (new `content` column); a version 1 database is rejected and must be + rescanned, which is required anyway since every stored hash changed. + `report` and `trees` group by the extended signature, so the ladder is + applied across the whole database. README "Duplicate detection" + documents every rung including the probabilistic large-file path. - remove the dead `files.dat` references from `Makefile`, `.gitignore` and `.dockerignore` (2026-09-21, branch `next`, closes diff --git a/db.go b/db.go index 19bb268..fb1b4dc 100644 --- a/db.go +++ b/db.go @@ -26,8 +26,9 @@ const databaseEnv = "SFDUPES_DATABASE" // schemaVersion is the database schema version this build reads and // writes, stored in PRAGMA user_version. Version 2 adds the content -// column and stores 65 KiB (rather than 1 KiB) end-window hashes, so a -// version 1 database is rejected and must be rescanned. +// column and the head/tail/content signature (replacing the version 1 +// 1 KiB end windows), so a version 1 database is rejected and must be +// rescanned. const schemaVersion = 2 // dbDirPerm is the mode for a database parent directory created by diff --git a/scan.go b/scan.go index e08cb76..44b0914 100644 --- a/scan.go +++ b/scan.go @@ -18,13 +18,22 @@ import ( ) // The duplicate ladder (see hashSignature and README "Duplicate -// detection"). Same-size candidates are separated first by the hashes -// of their end windows, then by a content hash that is exact for -// smaller files and deliberately sampled for large ones. +// detection"). A same-size candidate below headTailMin is hashed in +// full and compared directly; a larger one is separated first by the +// hashes of its end windows, then by a content hash that is exact below +// wholeFileMax and deliberately sampled at or above it. + +// headTailMin is the size threshold for the end-window gate. A file +// smaller than this is hashed in full directly, with no separate head +// and tail step: its head, tail, and content all carry the whole-file +// hash. A file this size or larger is separated first by its end +// windows. +const headTailMin = 10 * 1024 * 1024 // headTailWindow is the number of bytes hashed from each end of a file -// (the head and tail rungs). A file no larger than one window has head -// and tail equal to the hash of its whole content. +// at or above headTailMin (the head and tail rungs). Because +// headTailMin is far larger than two windows, the head and tail windows +// never overlap. const headTailWindow = 64 * 1024 // wholeFileMax is the size boundary between the two content rungs: a @@ -972,14 +981,17 @@ const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" + "27ae41e4649b934ca495991b7852b855" // hashSignature computes the three content hashes that, with the file -// size, form its duplicate signature: the SHA-256 of the first and last -// headTailWindow bytes (the head and tail rungs), and a content hash -// that is the SHA-256 of the whole file below wholeFileMax (the exact -// rung) or of gigabyte-spaced samples at or above it (the sampled, -// deliberately probabilistic rung). Two files are duplicates only when -// all four agree; any mismatch means not a duplicate. size is the value -// recorded when the file was statted; a zero-length file has constant -// hashes and is never opened. +// size, form its duplicate signature. A file below headTailMin is +// hashed in full and its whole-file SHA-256 is returned as head, tail, +// and content alike — that range takes no separate end-window step. For +// a file at or above headTailMin the head and tail are the SHA-256 of +// its first and last headTailWindow bytes, and content is the SHA-256 +// of the whole file below wholeFileMax (the exact rung) or of +// gigabyte-spaced samples at or above it (the sampled, deliberately +// probabilistic rung). Two files are duplicates only when all four +// agree; any mismatch means not a duplicate. size is the value recorded +// when the file was statted; a zero-length file has constant hashes and +// is never opened. func hashSignature(path string, size int64) (string, string, string, error) { if size == 0 { return emptyHash, emptyHash, emptyHash, nil @@ -993,6 +1005,17 @@ func hashSignature(path string, size int64) (string, string, string, error) { defer func() { _ = f.Close() }() + // Below the threshold the whole file is hashed directly, with no + // end-window step: head and tail both carry the whole-file hash. + if size < int64(headTailMin) { + content, err := hashWhole(f, size) + if err != nil { + return "", "", "", err + } + + return content, content, content, nil + } + head, tail, err := hashEnds(f, size) if err != nil { return "", "", "", err @@ -1007,14 +1030,11 @@ func hashSignature(path string, size int64) (string, string, string, error) { } // hashEnds returns the SHA-256 of the first and last headTailWindow -// bytes of f. The two windows overlap when the file is between one and -// two windows in size; when it is no larger than one window they -// coincide, so the head hash is reused as the tail and only one read is -// issued. +// bytes of f. It is called only for files at least headTailMin, which +// is far larger than two windows, so the windows never overlap and both +// reads are always full. func hashEnds(f *os.File, size int64) (string, string, error) { - n := min(int64(headTailWindow), size) - - buf := make([]byte, n) + buf := make([]byte, headTailWindow) _, err := f.ReadAt(buf, 0) if err != nil { @@ -1024,11 +1044,7 @@ func hashEnds(f *os.File, size int64) (string, string, error) { h := sha256.Sum256(buf) head := hex.EncodeToString(h[:]) - if size <= int64(headTailWindow) { - return head, head, nil - } - - _, err = f.ReadAt(buf, size-n) + _, err = f.ReadAt(buf, size-int64(headTailWindow)) if err != nil { return "", "", err } @@ -1050,16 +1066,23 @@ func hashContent(f *os.File, size int64) (string, error) { } // hashWhole returns the SHA-256 of the entire file. A SectionReader is -// used so the read is independent of the offset left by the end-window -// reads. +// used so the read is independent of the offset left by any end-window +// reads. Reading fewer than size bytes means the file shrank between +// the stat and the hash; that is an error rather than a hash of content +// that no longer matches the recorded size. func hashWhole(f *os.File, size int64) (string, error) { h := sha256.New() - _, err := io.Copy(h, io.NewSectionReader(f, 0, size)) + n, err := io.Copy(h, io.NewSectionReader(f, 0, size)) if err != nil { return "", err } + if n != size { + return "", fmt.Errorf("read %d of %d bytes: %w", n, size, + io.ErrUnexpectedEOF) + } + return hex.EncodeToString(h.Sum(nil)), nil } diff --git a/scan_test.go b/scan_test.go index f1eeb8d..3245442 100644 --- a/scan_test.go +++ b/scan_test.go @@ -53,10 +53,23 @@ func pattern(tag byte, n int) []byte { return data } -// TestHashSignatureEnds exercises the head and tail rungs across the -// window boundaries. Every file here is below wholeFileMax, so the -// content rung is a whole-file hash. -func TestHashSignatureEnds(t *testing.T) { +// sig returns a file's full signature (head, tail, content), failing the +// test on any error. +func sig(t *testing.T, path string, size int64) (string, string, string) { + t.Helper() + + head, tail, content, err := hashSignature(path, size) + if err != nil { + t.Fatalf("hashSignature %s: %v", path, err) + } + + return head, tail, content +} + +// TestHashSignatureBelowThreshold verifies that a file below headTailMin +// is hashed in full and compared directly: head, tail, and content all +// carry the whole-file SHA-256, with no separate end-window step. +func TestHashSignatureBelowThreshold(t *testing.T) { t.Parallel() dir := t.TempDir() @@ -65,13 +78,10 @@ func TestHashSignatureEnds(t *testing.T) { name string data []byte }{ - {"empty", nil}, {"one-byte", []byte("x")}, - {"under-one-window", pattern(1, headTailWindow-1)}, - {"exactly-one-window", pattern(2, headTailWindow)}, - {"overlapping-windows", pattern(3, headTailWindow+headTailWindow/2)}, - {"exactly-two-windows", pattern(4, 2*headTailWindow)}, - {"beyond-two-windows", pattern(5, 3*headTailWindow)}, + {"one-window", pattern(1, headTailWindow)}, + {"several-windows", pattern(2, 3*headTailWindow)}, + {"near-threshold", pattern(3, headTailMin-1)}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { @@ -79,28 +89,79 @@ func TestHashSignatureEnds(t *testing.T) { p := writeFile(t, dir, c.name, c.data) - head, tail, content, err := hashSignature(p, int64(len(c.data))) - if err != nil { - t.Fatalf("hashSignature: %v", err) - } + head, tail, content := sig(t, p, int64(len(c.data))) - n := min(headTailWindow, len(c.data)) - if want := hexSum(c.data[:n]); head != want { - t.Errorf("head = %s, want %s", head, want) - } - - if want := hexSum(c.data[len(c.data)-n:]); tail != want { - t.Errorf("tail = %s, want %s", tail, want) - } - - // Below wholeFileMax the content rung hashes the whole file. - if want := hexSum(c.data); content != want { - t.Errorf("content = %s, want whole-file %s", content, want) + whole := hexSum(c.data) + if head != whole || tail != whole || content != whole { + t.Errorf("head=%s tail=%s content=%s, want all whole-file %s", + head, tail, content, whole) } }) } } +// TestHashSignatureEnds exercises the head and tail rungs, which apply +// only to files at least headTailMin. Sparse files keep the fixtures +// cheap: a difference in the first window changes only head, a +// difference in the last window changes only tail, and a difference +// between the windows changes neither end hash but does change the +// whole-file content rung (the file is below wholeFileMax). +func TestHashSignatureEnds(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + + // Between headTailMin and wholeFileMax: the end-window gate is active + // and the content rung is a whole-file hash. + const size = int64(headTailMin + 2*1024*1024) + + base := sparseFile(t, dir, "ends-base", size) + headDiff := sparseFile(t, dir, "ends-head", size) + tailDiff := sparseFile(t, dir, "ends-tail", size) + midDiff := sparseFile(t, dir, "ends-mid", size) + + pokeAt(t, headDiff, 0, []byte{1}) + pokeAt(t, tailDiff, size-1, []byte{1}) + pokeAt(t, midDiff, size/2, []byte{1}) + + bHead, bTail, bContent := sig(t, base, size) + + h, tl, c := sig(t, headDiff, size) + if h == bHead { + t.Error("a byte in the first window did not change head") + } + + if tl != bTail { + t.Error("a byte in the first window changed tail") + } + + if c == bContent { + t.Error("a byte in the first window did not change content") + } + + h, tl, c = sig(t, tailDiff, size) + if tl == bTail { + t.Error("a byte in the last window did not change tail") + } + + if h != bHead { + t.Error("a byte in the last window changed head") + } + + if c == bContent { + t.Error("a byte in the last window did not change content") + } + + h, tl, c = sig(t, midDiff, size) + if h != bHead || tl != bTail { + t.Error("a byte between the windows changed an end hash") + } + + if c == bContent { + t.Error("whole-file content rung ignored a byte between the windows") + } +} + func TestHashSignatureErrors(t *testing.T) { t.Parallel() @@ -189,10 +250,7 @@ func pokeAt(t *testing.T, path string, off int64, data []byte) { func contentHash(t *testing.T, path string, size int64) string { t.Helper() - _, _, content, err := hashSignature(path, size) - if err != nil { - t.Fatalf("hashSignature %s: %v", path, err) - } + _, _, content := sig(t, path, size) return content }