Hash files under 10 MiB in full, gate larger files on head/tail

Amends the issue 61 ladder per the owner's design change. A file under
10 MiB (new headTailMin) is now hashed in full and compared directly,
with no end-window step: its head, tail, and content all carry the
whole-file SHA-256, so its signature is decided by size and content
alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then
the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB
samples at or above. hashEnds drops its now-unreachable single-window
branch, and hashWhole errors if the file shrank below its recorded
size (the only read for the sub-10-MiB range). README, TODO, tests,
and the schema-version note updated to match.

Model: opus-4-8

README.md

@@ -7,12 +7,14 @@
 duplicate files — and, ultimately, entire duplicate directory trees —
 across very large filesystems without reading every byte of every file.
 Files are considered duplicates when their sizes are equal and they
-agree on a short ladder of hashes: the SHA-256 of their first 64 KiB and
+agree on a short ladder of hashes. A file under 10 MiB is hashed in full
-of their last 64 KiB, and then a content hash — the SHA-256 of the whole
+and compared directly. A larger file is gated first on the SHA-256 of
-file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when it
+its first 64 KiB and of its last 64 KiB, and then compared on a content
-is 50 MiB or larger. Below 50 MiB this is proof of identical content; at
+hash — the SHA-256 of the whole file when it is under 50 MiB, or of
-or above 50 MiB it is a strong candidate signal rather than proof,
+gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
-because the gaps between samples are never read. The intended use is
+the content hash is proof of identical content; at or above 50 MiB it is
+a strong candidate signal rather than proof, because the gaps between
+samples are never read. The intended use is
 finding duplicate downloads and duplicated directory trees on
 multi-terabyte ZFS servers where reading every byte of every file is
 prohibitively expensive. `scan` maintains a persistent SQLite database of file
@@ -78,7 +80,8 @@ Goals, in order:
 2.  **Spend I/O in proportion to duplicate likelihood.** Only files
     whose size at least one other file shares are read at all — a
     size-unique file cannot be a duplicate. Those are compared by the
-    ladder in "Duplicate detection" below: cheap 64 KiB end windows
+    ladder in "Duplicate detection" below: a file under 10 MiB is hashed
+    in full, while a larger file is gated on cheap 64 KiB end windows
     first, then a content hash that reads the whole file below 50 MiB
     but only gigabyte-spaced 1 MiB samples at or above it, so the very
     largest files are still never read in full. Scale target: tens of
@@ -153,28 +156,31 @@ All three subcommands operate on a single SQLite database file:
     toward the filesystem.
 -   Schema (`PRAGMA user_version` is the schema version, currently 2; a
     database with any other version is a fatal error). Version 2 added
-    the `content` column and widened the end windows from 1 KiB to
+    the `content` column and the 64 KiB head/tail signature (replacing
-    64 KiB, so a version 1 database cannot be reused: it is rejected and
+    the version 1 1 KiB end windows), so a version 1 database cannot be
-    the tree must be rescanned from scratch.
+    reused: it is rejected and the tree must be rescanned from scratch.
 
     ```sql
     CREATE TABLE files (
         path    BLOB PRIMARY KEY,  -- absolute path, raw bytes
         size    INTEGER NOT NULL,  -- bytes, from lstat
         mtime   INTEGER NOT NULL,  -- Unix seconds, from lstat
-        head    TEXT NOT NULL,     -- lowercase-hex SHA-256, first 64 KiB
+        head    TEXT NOT NULL,     -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
-        tail    TEXT NOT NULL,     -- lowercase-hex SHA-256, last 64 KiB
+        tail    TEXT NOT NULL,     -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
         content TEXT NOT NULL      -- lowercase-hex SHA-256, whole file or samples
     ) WITHOUT ROWID;
     ```
 
     Paths are stored as BLOBs because Unix paths are raw bytes, not
     guaranteed UTF-8. `mtime` is used only for change detection; it is
-    not part of the duplicate key. `head`, `tail`, and `content` are
+    not part of the duplicate key. For a file under 10 MiB `head`, `tail`,
-    empty strings when the file has never been hashed because its size
+    and `content` all hold the whole-file hash (that range is hashed in
-    was unique as of the last scan that covered it; such records still
+    full, with no end windows); for a larger file `head` and `tail` hold
-    define the file for tree reconstruction but never participate in
+    the first- and last-64 KiB hashes and `content` the whole-file or
-    duplicate groups.
+    sampled hash. All three are empty strings when the file has never
+    been hashed because its size was unique as of the last scan that
+    covered it; such records still define the file for tree
+    reconstruction but never participate in duplicate groups.
 
 ### Duplicate detection
 
@@ -187,27 +193,33 @@ the database, even across separate scans.
 
 1.  **Size.** Files of different sizes are never compared. Only files
     whose size at least one other file shares are hashed at all.
-2.  **Head and tail.** The SHA-256 of the first 64 KiB (`head`) and of
+2.  **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed
-    the last 64 KiB (`tail`). When a file is 64 KiB or smaller the two
+    in full and compared directly, with no separate end-window step —
-    windows are the whole file and coincide, so `head` and `tail` are
+    small files are cheap to read to the last byte, and doing so makes
-    equal and only one read is issued; when it is between one and two
+    the comparison exact. `head`, `tail`, and `content` all hold this
-    windows the two windows overlap, which is harmless. These reads are
+    whole-file SHA-256, so such a file's signature is decided entirely
-    cheap and eliminate most same-size pairs before any bulk reading.
+    by its size and its content.
-3.  **Content, below 50 MiB.** The SHA-256 of the entire file. Agreement
+3.  **10 MiB and above: head and tail.** For a larger file, the SHA-256
-    here is proof of identical content (barring a SHA-256 collision).
+    of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
-4.  **Content, 50 MiB and above.** A sampled SHA-256: the 1 MiB window
+    cheap gate that eliminates most same-size pairs before any bulk
-    at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the
+    reading. At 10 MiB and above the two windows never overlap.
-    file, the final window truncated at end of file) is fed, in order,
+4.  **10 MiB and above, content below 50 MiB.** The SHA-256 of the
-    into one hash. This is **deliberately probabilistic** — the gaps
+    entire file. Agreement here is proof of identical content (barring a
-    between samples are never read, so two large files that agree on
+    SHA-256 collision).
-    every sample are reported as duplicates without being read in full.
+5.  **10 MiB and above, content 50 MiB and above.** A sampled SHA-256:
-    It is the price of never reading a 150 GB file end to end. Because
+    the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, …
-    size is already part of the signature, only equal-size files reach
+    while inside the file, the final window truncated at end of file) is
-    this rung, so their sample boundaries always align.
+    fed, in order, into one hash. This is **deliberately probabilistic**
+    — the gaps between samples are never read, so two large files that
+    agree on every sample are reported as duplicates without being read
+    in full. It is the price of never reading a 150 GB file end to end.
+    Because size is already part of the signature, only equal-size files
+    reach this rung, so their sample boundaries always align.
 
-`head`, `tail`, and `content` are one column each; a file below 50 MiB
+`head`, `tail`, and `content` are one column each. A file below 10 MiB
-and a file at or above it never share a size, so a `content` value is
+and one at or above it never share a size, and neither do a file below
-never ambiguous between the whole-file and sampled forms.
+50 MiB and one at or above it, so a stored value is never ambiguous
+between the whole-file, end-window, and sampled forms.
 
 ### `scan` mode
 

TODO.md

@@ -29,17 +29,19 @@
 
 # Completed Steps
 
-- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
+- replace the 1 KiB end-window sampling with the head/tail plus
   content-hash ladder (2026-09-22, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
+  https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
-  gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
+  hashed in full and compared directly, with no end-window step — its
-  1 MiB samples at or above — and the end windows widen from 1 KiB to
+  `head`, `tail`, and `content` all hold the whole-file hash. A file at
-  64 KiB. Schema bumps to version 2 (new `content` column); a version 1
+  10 MiB or above is gated on the 64 KiB `head` and `tail`, then
-  database is rejected and must be rescanned, which is required anyway
+  compared on a `content` hash — the whole file below 50 MiB,
-  since every stored hash changed. `report` and `trees` group by the
+  gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
-  extended signature, so the ladder is applied across the whole
+  (new `content` column); a version 1 database is rejected and must be
-  database. README "Duplicate detection" documents every rung including
+  rescanned, which is required anyway since every stored hash changed.
-  the probabilistic large-file path.
+  `report` and `trees` group by the extended signature, so the ladder is
+  applied across the whole database. README "Duplicate detection"
+  documents every rung including the probabilistic large-file path.
 
 - remove the dead `files.dat` references from `Makefile`, `.gitignore`
   and `.dockerignore` (2026-09-21, branch `next`, closes

db.go

@@ -26,8 +26,9 @@ const databaseEnv = "SFDUPES_DATABASE"
 
 // schemaVersion is the database schema version this build reads and
 // writes, stored in PRAGMA user_version. Version 2 adds the content
-// column and stores 65 KiB (rather than 1 KiB) end-window hashes, so a
+// column and the head/tail/content signature (replacing the version 1
-// version 1 database is rejected and must be rescanned.
+// 1 KiB end windows), so a version 1 database is rejected and must be
+// rescanned.
 const schemaVersion = 2
 
 // dbDirPerm is the mode for a database parent directory created by

scan.go

@@ -18,13 +18,22 @@ import (
 )
 
 // The duplicate ladder (see hashSignature and README "Duplicate
-// detection"). Same-size candidates are separated first by the hashes
+// detection"). A same-size candidate below headTailMin is hashed in
-// of their end windows, then by a content hash that is exact for
+// full and compared directly; a larger one is separated first by the
-// smaller files and deliberately sampled for large ones.
+// hashes of its end windows, then by a content hash that is exact below
+// wholeFileMax and deliberately sampled at or above it.
+
+// headTailMin is the size threshold for the end-window gate. A file
+// smaller than this is hashed in full directly, with no separate head
+// and tail step: its head, tail, and content all carry the whole-file
+// hash. A file this size or larger is separated first by its end
+// windows.
+const headTailMin = 10 * 1024 * 1024
 
 // headTailWindow is the number of bytes hashed from each end of a file
-// (the head and tail rungs). A file no larger than one window has head
+// at or above headTailMin (the head and tail rungs). Because
-// and tail equal to the hash of its whole content.
+// headTailMin is far larger than two windows, the head and tail windows
+// never overlap.
 const headTailWindow = 64 * 1024
 
 // wholeFileMax is the size boundary between the two content rungs: a
@@ -972,14 +981,17 @@ const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
 	"27ae41e4649b934ca495991b7852b855"
 
 // hashSignature computes the three content hashes that, with the file
-// size, form its duplicate signature: the SHA-256 of the first and last
+// size, form its duplicate signature. A file below headTailMin is
-// headTailWindow bytes (the head and tail rungs), and a content hash
+// hashed in full and its whole-file SHA-256 is returned as head, tail,
-// that is the SHA-256 of the whole file below wholeFileMax (the exact
+// and content alike — that range takes no separate end-window step. For
-// rung) or of gigabyte-spaced samples at or above it (the sampled,
+// a file at or above headTailMin the head and tail are the SHA-256 of
-// deliberately probabilistic rung). Two files are duplicates only when
+// its first and last headTailWindow bytes, and content is the SHA-256
-// all four agree; any mismatch means not a duplicate. size is the value
+// of the whole file below wholeFileMax (the exact rung) or of
-// recorded when the file was statted; a zero-length file has constant
+// gigabyte-spaced samples at or above it (the sampled, deliberately
-// hashes and is never opened.
+// probabilistic rung). Two files are duplicates only when all four
+// agree; any mismatch means not a duplicate. size is the value recorded
+// when the file was statted; a zero-length file has constant hashes and
+// is never opened.
 func hashSignature(path string, size int64) (string, string, string, error) {
 	if size == 0 {
 		return emptyHash, emptyHash, emptyHash, nil
@@ -993,6 +1005,17 @@ func hashSignature(path string, size int64) (string, string, string, error) {
 
 	defer func() { _ = f.Close() }()
 
+	// Below the threshold the whole file is hashed directly, with no
+	// end-window step: head and tail both carry the whole-file hash.
+	if size < int64(headTailMin) {
+		content, err := hashWhole(f, size)
+		if err != nil {
+			return "", "", "", err
+		}
+
+		return content, content, content, nil
+	}
+
 	head, tail, err := hashEnds(f, size)
 	if err != nil {
 		return "", "", "", err
@@ -1007,14 +1030,11 @@ func hashSignature(path string, size int64) (string, string, string, error) {
 }
 
 // hashEnds returns the SHA-256 of the first and last headTailWindow
-// bytes of f. The two windows overlap when the file is between one and
+// bytes of f. It is called only for files at least headTailMin, which
-// two windows in size; when it is no larger than one window they
+// is far larger than two windows, so the windows never overlap and both
-// coincide, so the head hash is reused as the tail and only one read is
+// reads are always full.
-// issued.
 func hashEnds(f *os.File, size int64) (string, string, error) {
-	n := min(int64(headTailWindow), size)
+	buf := make([]byte, headTailWindow)
-
-	buf := make([]byte, n)
 
 	_, err := f.ReadAt(buf, 0)
 	if err != nil {
@@ -1024,11 +1044,7 @@ func hashEnds(f *os.File, size int64) (string, string, error) {
 	h := sha256.Sum256(buf)
 	head := hex.EncodeToString(h[:])
 
-	if size <= int64(headTailWindow) {
+	_, err = f.ReadAt(buf, size-int64(headTailWindow))
-		return head, head, nil
-	}
-
-	_, err = f.ReadAt(buf, size-n)
 	if err != nil {
 		return "", "", err
 	}
@@ -1050,16 +1066,23 @@ func hashContent(f *os.File, size int64) (string, error) {
 }
 
 // hashWhole returns the SHA-256 of the entire file. A SectionReader is
-// used so the read is independent of the offset left by the end-window
+// used so the read is independent of the offset left by any end-window
-// reads.
+// reads. Reading fewer than size bytes means the file shrank between
+// the stat and the hash; that is an error rather than a hash of content
+// that no longer matches the recorded size.
 func hashWhole(f *os.File, size int64) (string, error) {
 	h := sha256.New()
 
-	_, err := io.Copy(h, io.NewSectionReader(f, 0, size))
+	n, err := io.Copy(h, io.NewSectionReader(f, 0, size))
 	if err != nil {
 		return "", err
 	}
 
+	if n != size {
+		return "", fmt.Errorf("read %d of %d bytes: %w", n, size,
+			io.ErrUnexpectedEOF)
+	}
+
 	return hex.EncodeToString(h.Sum(nil)), nil
 }
 

scan_test.go

@@ -53,10 +53,23 @@ func pattern(tag byte, n int) []byte {
 	return data
 }
 
-// TestHashSignatureEnds exercises the head and tail rungs across the
+// sig returns a file's full signature (head, tail, content), failing the
-// window boundaries. Every file here is below wholeFileMax, so the
+// test on any error.
-// content rung is a whole-file hash.
+func sig(t *testing.T, path string, size int64) (string, string, string) {
-func TestHashSignatureEnds(t *testing.T) {
+	t.Helper()
+
+	head, tail, content, err := hashSignature(path, size)
+	if err != nil {
+		t.Fatalf("hashSignature %s: %v", path, err)
+	}
+
+	return head, tail, content
+}
+
+// TestHashSignatureBelowThreshold verifies that a file below headTailMin
+// is hashed in full and compared directly: head, tail, and content all
+// carry the whole-file SHA-256, with no separate end-window step.
+func TestHashSignatureBelowThreshold(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
@@ -65,13 +78,10 @@ func TestHashSignatureEnds(t *testing.T) {
 		name string
 		data []byte
 	}{
-		{"empty", nil},
 		{"one-byte", []byte("x")},
-		{"under-one-window", pattern(1, headTailWindow-1)},
+		{"one-window", pattern(1, headTailWindow)},
-		{"exactly-one-window", pattern(2, headTailWindow)},
+		{"several-windows", pattern(2, 3*headTailWindow)},
-		{"overlapping-windows", pattern(3, headTailWindow+headTailWindow/2)},
+		{"near-threshold", pattern(3, headTailMin-1)},
-		{"exactly-two-windows", pattern(4, 2*headTailWindow)},
-		{"beyond-two-windows", pattern(5, 3*headTailWindow)},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
@@ -79,28 +89,79 @@ func TestHashSignatureEnds(t *testing.T) {
 
 			p := writeFile(t, dir, c.name, c.data)
 
-			head, tail, content, err := hashSignature(p, int64(len(c.data)))
+			head, tail, content := sig(t, p, int64(len(c.data)))
-			if err != nil {
-				t.Fatalf("hashSignature: %v", err)
-			}
 
-			n := min(headTailWindow, len(c.data))
+			whole := hexSum(c.data)
-			if want := hexSum(c.data[:n]); head != want {
+			if head != whole || tail != whole || content != whole {
-				t.Errorf("head = %s, want %s", head, want)
+				t.Errorf("head=%s tail=%s content=%s, want all whole-file %s",
-			}
+					head, tail, content, whole)
-
-			if want := hexSum(c.data[len(c.data)-n:]); tail != want {
-				t.Errorf("tail = %s, want %s", tail, want)
-			}
-
-			// Below wholeFileMax the content rung hashes the whole file.
-			if want := hexSum(c.data); content != want {
-				t.Errorf("content = %s, want whole-file %s", content, want)
 			}
 		})
 	}
 }
 
+// TestHashSignatureEnds exercises the head and tail rungs, which apply
+// only to files at least headTailMin. Sparse files keep the fixtures
+// cheap: a difference in the first window changes only head, a
+// difference in the last window changes only tail, and a difference
+// between the windows changes neither end hash but does change the
+// whole-file content rung (the file is below wholeFileMax).
+func TestHashSignatureEnds(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	// Between headTailMin and wholeFileMax: the end-window gate is active
+	// and the content rung is a whole-file hash.
+	const size = int64(headTailMin + 2*1024*1024)
+
+	base := sparseFile(t, dir, "ends-base", size)
+	headDiff := sparseFile(t, dir, "ends-head", size)
+	tailDiff := sparseFile(t, dir, "ends-tail", size)
+	midDiff := sparseFile(t, dir, "ends-mid", size)
+
+	pokeAt(t, headDiff, 0, []byte{1})
+	pokeAt(t, tailDiff, size-1, []byte{1})
+	pokeAt(t, midDiff, size/2, []byte{1})
+
+	bHead, bTail, bContent := sig(t, base, size)
+
+	h, tl, c := sig(t, headDiff, size)
+	if h == bHead {
+		t.Error("a byte in the first window did not change head")
+	}
+
+	if tl != bTail {
+		t.Error("a byte in the first window changed tail")
+	}
+
+	if c == bContent {
+		t.Error("a byte in the first window did not change content")
+	}
+
+	h, tl, c = sig(t, tailDiff, size)
+	if tl == bTail {
+		t.Error("a byte in the last window did not change tail")
+	}
+
+	if h != bHead {
+		t.Error("a byte in the last window changed head")
+	}
+
+	if c == bContent {
+		t.Error("a byte in the last window did not change content")
+	}
+
+	h, tl, c = sig(t, midDiff, size)
+	if h != bHead || tl != bTail {
+		t.Error("a byte between the windows changed an end hash")
+	}
+
+	if c == bContent {
+		t.Error("whole-file content rung ignored a byte between the windows")
+	}
+}
+
 func TestHashSignatureErrors(t *testing.T) {
 	t.Parallel()
 
@@ -189,10 +250,7 @@ func pokeAt(t *testing.T, path string, off int64, data []byte) {
 func contentHash(t *testing.T, path string, size int64) string {
 	t.Helper()
 
-	_, _, content, err := hashSignature(path, size)
+	_, _, content := sig(t, path, size)
-	if err != nil {
-		t.Fatalf("hashSignature %s: %v", path, err)
-	}
 
 	return content
 }