Add 64 KiB head/tail and content-hash duplicate ladder (closes #61)

Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.

The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.

Model: opus-4-8

README.md

@@ -5,14 +5,17 @@
 `sfdupes` is an MIT-licensed Go CLI tool by
 [@sneak](https://sneak.berlin) that quickly identifies *candidate*
 duplicate files — and, ultimately, entire duplicate directory trees —
-across very large filesystems without reading full file contents. Files
-are considered duplicates when they have identical size, identical
-SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
-1024 bytes. This is a strong candidate signal, not proof of identical
-content (the middle of the file is never read); the intended use is
+across very large filesystems without reading every byte of every file.
+Files are considered duplicates when their sizes are equal and they
+agree on a short ladder of hashes: the SHA-256 of their first 64 KiB and
+of their last 64 KiB, and then a content hash — the SHA-256 of the whole
+file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when it
+is 50 MiB or larger. Below 50 MiB this is proof of identical content; at
+or above 50 MiB it is a strong candidate signal rather than proof,
+because the gaps between samples are never read. The intended use is
 finding duplicate downloads and duplicated directory trees on
-multi-terabyte ZFS servers where reading every byte is prohibitively
-expensive. `scan` maintains a persistent SQLite database of file
+multi-terabyte ZFS servers where reading every byte of every file is
+prohibitively expensive. `scan` maintains a persistent SQLite database of file
 signatures that survives between runs, so it can be run from cron and
 the reports can be generated at any time from the most recent scan.
 
@@ -29,7 +32,8 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
 ```
 
 `scan` walks one or more filesystem trees and maintains one database
-record per regular file (path, size, mtime, head hash, tail hash). The
+record per regular file (path, size, mtime, head hash, tail hash,
+content hash). The
 database persists between runs; a rescan only hashes files that are new
 or changed, and removes records for files that no longer exist.
 `report` reads the database and prints the file-level duplicates
@@ -47,10 +51,13 @@ completed scan.
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
-disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
-only from files whose size at least one other file shares, since a
-size-unique file cannot be a duplicate — makes a full-filesystem sweep
-tractable, and the signatures are kept in a persistent database, so
+disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
+whose size at least one other file shares, since a size-unique file
+cannot be a duplicate; for those it reads the cheap end windows first
+and a content hash second — the whole file below 50 MiB, but only
+gigabyte-spaced samples at or above 50 MiB, so the largest files are
+never read in full. This keeps a full-filesystem sweep tractable, and
+the signatures are kept in a persistent database, so
 the expensive filesystem pass is incremental: a rescan re-hashes only
 files whose recorded mtime or size changed, and all analysis happens
 offline from the database alone. The end goal is
@@ -68,14 +75,17 @@ Goals, in order:
     downloads, copied project trees), so the operator can consider
     removing an entire subtree at once. File-level duplicate detection is
     the foundation; tree-level detection is built on top of it.
-2.  **Never read full file contents.** At most 2 KiB is read per file
-    (first and last 1024 bytes), and only files whose size at least
-    one other file shares are read at all — a size-unique file cannot
-    be a duplicate. Scale target: tens of millions of files, ~150 TB
-    filesystem, possibly slow or busy disks (ZFS pool under resilver).
-    Holding one small record (path, size, mtime) per file in memory
-    during a scan is acceptable; holding every file's hashes is not
-    (they stay in the database).
+2.  **Spend I/O in proportion to duplicate likelihood.** Only files
+    whose size at least one other file shares are read at all — a
+    size-unique file cannot be a duplicate. Those are compared by the
+    ladder in "Duplicate detection" below: cheap 64 KiB end windows
+    first, then a content hash that reads the whole file below 50 MiB
+    but only gigabyte-spaced 1 MiB samples at or above it, so the very
+    largest files are still never read in full. Scale target: tens of
+    millions of files, ~150 TB filesystem, possibly slow or busy disks
+    (ZFS pool under resilver). Holding one small record (path, size,
+    mtime) per file in memory during a scan is acceptable; holding
+    every file's hashes is not (they stay in the database).
 3.  **Scan incrementally, analyze offline.** The expensive filesystem
     scan maintains a persistent database; an unchanged file is never
     read again on a rescan. All analysis (`report`, `trees`) works from
@@ -141,26 +151,63 @@ All three subcommands operate on a single SQLite database file:
     that dies partway leaves a valid database holding everything
     hashed so far; the next scan skips those records and converges
     toward the filesystem.
--   Schema (`PRAGMA user_version` is the schema version, currently 1; a
-    database with any other version is a fatal error):
+-   Schema (`PRAGMA user_version` is the schema version, currently 2; a
+    database with any other version is a fatal error). Version 2 added
+    the `content` column and widened the end windows from 1 KiB to
+    64 KiB, so a version 1 database cannot be reused: it is rejected and
+    the tree must be rescanned from scratch.
 
     ```sql
     CREATE TABLE files (
-        path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
-        size  INTEGER NOT NULL,  -- bytes, from lstat
-        mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
-        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
-        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
+        path    BLOB PRIMARY KEY,  -- absolute path, raw bytes
+        size    INTEGER NOT NULL,  -- bytes, from lstat
+        mtime   INTEGER NOT NULL,  -- Unix seconds, from lstat
+        head    TEXT NOT NULL,     -- lowercase-hex SHA-256, first 64 KiB
+        tail    TEXT NOT NULL,     -- lowercase-hex SHA-256, last 64 KiB
+        content TEXT NOT NULL      -- lowercase-hex SHA-256, whole file or samples
     ) WITHOUT ROWID;
     ```
 
     Paths are stored as BLOBs because Unix paths are raw bytes, not
     guaranteed UTF-8. `mtime` is used only for change detection; it is
-    not part of the duplicate key. `head` and `tail` are empty strings
-    when the file has never been hashed because its size was unique as
-    of the last scan that covered it; such records still define the
-    file for tree reconstruction but never participate in duplicate
-    groups.
+    not part of the duplicate key. `head`, `tail`, and `content` are
+    empty strings when the file has never been hashed because its size
+    was unique as of the last scan that covered it; such records still
+    define the file for tree reconstruction but never participate in
+    duplicate groups.
+
+### Duplicate detection
+
+Two files are duplicates only when they agree on every rung of this
+ladder; a mismatch at any rung means they are not duplicates. `scan`
+stores each file's hashes once, and `report` and `trees` group files by
+the whole signature — size, `head`, `tail`, and `content` — so the
+grouping is exactly this ladder applied across everything scanned into
+the database, even across separate scans.
+
+1.  **Size.** Files of different sizes are never compared. Only files
+    whose size at least one other file shares are hashed at all.
+2.  **Head and tail.** The SHA-256 of the first 64 KiB (`head`) and of
+    the last 64 KiB (`tail`). When a file is 64 KiB or smaller the two
+    windows are the whole file and coincide, so `head` and `tail` are
+    equal and only one read is issued; when it is between one and two
+    windows the two windows overlap, which is harmless. These reads are
+    cheap and eliminate most same-size pairs before any bulk reading.
+3.  **Content, below 50 MiB.** The SHA-256 of the entire file. Agreement
+    here is proof of identical content (barring a SHA-256 collision).
+4.  **Content, 50 MiB and above.** A sampled SHA-256: the 1 MiB window
+    at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the
+    file, the final window truncated at end of file) is fed, in order,
+    into one hash. This is **deliberately probabilistic** — the gaps
+    between samples are never read, so two large files that agree on
+    every sample are reported as duplicates without being read in full.
+    It is the price of never reading a 150 GB file end to end. Because
+    size is already part of the signature, only equal-size files reach
+    this rung, so their sample boundaries always align.
+
+`head`, `tail`, and `content` are one column each; a file below 50 MiB
+and a file at or above it never share a size, so a `content` value is
+never ambiguous between the whole-file and sampled forms.
 
 ### `scan` mode
 
@@ -229,10 +276,9 @@ during the hash phase:
     decides its fate. Size-unique files are never read: new or
     changed ones are recorded without hashes in the update phase,
     unchanged unhashed ones simply keep their records. Every file
-    with a shared size is hashed by the worker pool: read the first
-    `min(1024, size)` bytes and the last `min(1024, size)` bytes
-    (one read when `size <= 1024`, since the two windows coincide)
-    and compute the SHA-256 of each. Zero-length files have constant
+    with a shared size is hashed by the worker pool, computing the
+    full signature — head, tail, and content — described in "Duplicate
+    detection" below. Zero-length files have constant
     hashes and are never opened. Files are hashed in **inode order**
     (minimizing seeks on spinning disks), and paths that are hard
     links to the same inode are **read once**, all sharing the one

TODO.md

@@ -29,6 +29,18 @@
 
 # Completed Steps
 
+- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
+  content-hash ladder (2026-09-22, branch `next`, closes
+  https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
+  gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
+  1 MiB samples at or above — and the end windows widen from 1 KiB to
+  64 KiB. Schema bumps to version 2 (new `content` column); a version 1
+  database is rejected and must be rescanned, which is required anyway
+  since every stored hash changed. `report` and `trees` group by the
+  extended signature, so the ladder is applied across the whole
+  database. README "Duplicate detection" documents every rung including
+  the probabilistic large-file path.
+
 - remove the dead `files.dat` references from `Makefile`, `.gitignore`
   and `.dockerignore` (2026-09-21, branch `next`, closes
   https://git.eeqj.de/sneak/sfdupes/issues/22)

db.go

@@ -25,8 +25,10 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
 const databaseEnv = "SFDUPES_DATABASE"
 
 // schemaVersion is the database schema version this build reads and
-// writes, stored in PRAGMA user_version.
-const schemaVersion = 1
+// writes, stored in PRAGMA user_version. Version 2 adds the content
+// column and stores 65 KiB (rather than 1 KiB) end-window hashes, so a
+// version 1 database is rejected and must be rescanned.
+const schemaVersion = 2
 
 // dbDirPerm is the mode for a database parent directory created by
 // scan.
@@ -36,22 +38,24 @@ const dbDirPerm = 0o755
 // BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
 const createTableSQL = `
 CREATE TABLE files (
-    path  BLOB PRIMARY KEY,
-    size  INTEGER NOT NULL,
-    mtime INTEGER NOT NULL,
-    head  TEXT NOT NULL,
-    tail  TEXT NOT NULL
+    path    BLOB PRIMARY KEY,
+    size    INTEGER NOT NULL,
+    mtime   INTEGER NOT NULL,
+    head    TEXT NOT NULL,
+    tail    TEXT NOT NULL,
+    content TEXT NOT NULL
 ) WITHOUT ROWID
 `
 
 // upsertSQL inserts one file record, replacing any existing record for
 // the same path.
 const upsertSQL = `
-INSERT INTO files (path, size, mtime, head, tail)
-VALUES (?, ?, ?, ?, ?)
+INSERT INTO files (path, size, mtime, head, tail, content)
+VALUES (?, ?, ?, ?, ?, ?)
 ON CONFLICT (path) DO UPDATE SET
     size = excluded.size, mtime = excluded.mtime,
-    head = excluded.head, tail = excluded.tail
+    head = excluded.head, tail = excluded.tail,
+    content = excluded.content
 `
 
 // errNoDatabase reports a missing database file for report/trees.
@@ -205,7 +209,7 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
 // loadFileRows reads every record from the files table.
 func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
 	rows, err := db.QueryContext(ctx,
-		"SELECT path, size, mtime, head, tail FROM files")
+		"SELECT path, size, mtime, head, tail, content FROM files")
 	if err != nil {
 		return nil, fmt.Errorf("read records: %w", err)
 	}
@@ -220,7 +224,8 @@ func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
 			r    scanRec
 		)
 
-		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail)
+		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail,
+			&r.content)
 		if err != nil {
 			return nil, fmt.Errorf("read record: %w", err)
 		}
@@ -348,7 +353,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
 
 	for _, r := range upserts {
 		_, err = st.ExecContext(ctx,
-			[]byte(r.path), r.size, r.mtime, r.head, r.tail)
+			[]byte(r.path), r.size, r.mtime, r.head, r.tail, r.content)
 		if err != nil {
 			return fmt.Errorf("upsert %s: %w", r.path, err)
 		}

report.go

@@ -17,14 +17,15 @@ const ioBufSize = 1 << 20
 const minGroupSize = 2
 
 // scanRec is one file record from the database. The signature (size,
-// head, tail) is the duplicate key; mtime is informational only and
-// used by scan for change detection.
+// head, tail, content) is the duplicate key; mtime is informational
+// only and used by scan for change detection.
 type scanRec struct {
-	size  int64
-	mtime int64
-	head  string
-	tail  string
-	path  string
+	size    int64
+	mtime   int64
+	head    string
+	tail    string
+	content string
+	path    string
 }
 
 // loadRecords opens the database and reads every file record for the
@@ -52,8 +53,9 @@ func loadRecords(ctx context.Context) ([]scanRec, error) {
 }
 
 // dupeGroup is one set of candidate-duplicate files: identical size,
-// head hash, and tail hash. paths is sorted lexicographically; the
-// first entry is the group's "first", the rest are dupes.
+// head hash, tail hash, and content hash. paths is sorted
+// lexicographically; the first entry is the group's "first", the rest
+// are dupes.
 type dupeGroup struct {
 	size  int64
 	paths []string
@@ -122,7 +124,9 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
 			continue
 		}
 
-		k := fileSig{size: r.size, head: r.head, tail: r.tail}
+		k := fileSig{
+			size: r.size, head: r.head, tail: r.tail, content: r.content,
+		}
 		groups[k] = append(groups[k], r.path)
 	}
 

report_test.go

@@ -38,6 +38,28 @@ func TestCollectDupeGroups(t *testing.T) {
 	}
 }
 
+func TestCollectDupeGroupsContentSeparates(t *testing.T) {
+	t.Parallel()
+
+	// Same size, head, and tail, but different content hashes: the final
+	// rung keeps them apart, so no group forms. Matching content groups.
+	recs := []scanRec{
+		{size: 100, head: "h", tail: "t", content: "c1", path: "/a"},
+		{size: 100, head: "h", tail: "t", content: "c2", path: "/b"},
+		{size: 100, head: "h", tail: "t", content: "c1", path: "/c"},
+	}
+
+	groups := collectDupeGroups(recs)
+	if len(groups) != 1 {
+		t.Fatalf("len(groups) = %d, want 1 (only the matching content)",
+			len(groups))
+	}
+
+	if !slices.Equal(groups[0].paths, []string{"/a", "/c"}) {
+		t.Errorf("group paths = %q, want /a /c", groups[0].paths)
+	}
+}
+
 func TestCollectDupeGroupsMtimeExcluded(t *testing.T) {
 	t.Parallel()
 

scan.go

@@ -7,6 +7,7 @@ import (
 	"database/sql"
 	"encoding/hex"
 	"fmt"
+	"io"
 	"io/fs"
 	"os"
 	"path/filepath"
@@ -16,8 +17,28 @@ import (
 	"syscall"
 )
 
-// chunk is the number of bytes hashed from each end of a file.
-const chunk = 1024
+// The duplicate ladder (see hashSignature and README "Duplicate
+// detection"). Same-size candidates are separated first by the hashes
+// of their end windows, then by a content hash that is exact for
+// smaller files and deliberately sampled for large ones.
+
+// headTailWindow is the number of bytes hashed from each end of a file
+// (the head and tail rungs). A file no larger than one window has head
+// and tail equal to the hash of its whole content.
+const headTailWindow = 64 * 1024
+
+// wholeFileMax is the size boundary between the two content rungs: a
+// file strictly smaller than this is content-hashed in full; a file
+// this size or larger is content-hashed by sampling.
+const wholeFileMax = 50 * 1024 * 1024
+
+// sampleStride is the spacing between content samples for large files:
+// one window is read at each gigabyte-aligned offset (0, 1 GiB, ...).
+const sampleStride = 1024 * 1024 * 1024
+
+// sampleWindow is the number of bytes read at each large-file sample
+// offset, truncated at end of file.
+const sampleWindow = 1024 * 1024
 
 // workQueueDepth bounds the job and result channels feeding the walk
 // and hash worker pools.
@@ -439,11 +460,12 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
 		s.resolve(rec.path)
 
 		s.batch = append(s.batch, scanRec{
-			size:  rec.size,
-			mtime: rec.mtime,
-			head:  r.head,
-			tail:  r.tail,
-			path:  rec.path,
+			size:    rec.size,
+			mtime:   rec.mtime,
+			head:    r.head,
+			tail:    r.tail,
+			content: r.content,
+			path:    rec.path,
 		})
 	}
 
@@ -834,13 +856,15 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
 	return statDev(st), st.Ino
 }
 
-// hashResult carries one inode run's head/tail hashes (or the error
-// that prevented hashing it) from the hash workers to the hash phase.
+// hashResult carries one inode run's signature hashes — head, tail, and
+// content — (or the error that prevented hashing it) from the hash
+// workers to the hash phase.
 type hashResult struct {
-	run  []fileRec
-	head string
-	tail string
-	err  error
+	run     []fileRec
+	head    string
+	tail    string
+	content string
+	err     error
 }
 
 // hashPool owns every goroutine of the hash worker pool: the feeder
@@ -930,11 +954,11 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 			continue
 		}
 
-		head, tail, err := hashHeadTail(run[0].path, run[0].size)
+		head, tail, content, err := hashSignature(run[0].path, run[0].size)
 
 		select {
 		case results <- hashResult{
-			run: run, head: head, tail: tail, err: err,
+			run: run, head: head, tail: tail, content: content, err: err,
 		}:
 		case <-ctx.Done():
 			return
@@ -942,47 +966,66 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 	}
 }
 
-// emptyHash is the lowercase-hex SHA-256 of the empty input: the head
-// and tail hash of every zero-length file.
+// emptyHash is the lowercase-hex SHA-256 of the empty input: the head,
+// tail, and content hash of every zero-length file.
 const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
 	"27ae41e4649b934ca495991b7852b855"
 
-// hashHeadTail returns the lowercase-hex SHA-256 of the first
-// min(chunk, size) bytes and of the last min(chunk, size) bytes of the
-// file at path. The two reads overlap when size < 2*chunk. size is the
-// value recorded when the file was statted; a zero-length file's
-// hashes are constant, so it is never even opened.
-func hashHeadTail(path string, size int64) (string, string, error) {
+// hashSignature computes the three content hashes that, with the file
+// size, form its duplicate signature: the SHA-256 of the first and last
+// headTailWindow bytes (the head and tail rungs), and a content hash
+// that is the SHA-256 of the whole file below wholeFileMax (the exact
+// rung) or of gigabyte-spaced samples at or above it (the sampled,
+// deliberately probabilistic rung). Two files are duplicates only when
+// all four agree; any mismatch means not a duplicate. size is the value
+// recorded when the file was statted; a zero-length file has constant
+// hashes and is never opened.
+func hashSignature(path string, size int64) (string, string, string, error) {
 	if size == 0 {
-		return emptyHash, emptyHash, nil
+		return emptyHash, emptyHash, emptyHash, nil
 	}
 
 	//nolint:gosec // hashing operator-supplied paths is the tool's purpose
 	f, err := os.Open(path)
 	if err != nil {
-		return "", "", err
+		return "", "", "", err
 	}
 
 	defer func() { _ = f.Close() }()
 
-	n := min(int64(chunk), size)
+	head, tail, err := hashEnds(f, size)
+	if err != nil {
+		return "", "", "", err
+	}
+
+	content, err := hashContent(f, size)
+	if err != nil {
+		return "", "", "", err
+	}
+
+	return head, tail, content, nil
+}
+
+// hashEnds returns the SHA-256 of the first and last headTailWindow
+// bytes of f. The two windows overlap when the file is between one and
+// two windows in size; when it is no larger than one window they
+// coincide, so the head hash is reused as the tail and only one read is
+// issued.
+func hashEnds(f *os.File, size int64) (string, string, error) {
+	n := min(int64(headTailWindow), size)
 
 	buf := make([]byte, n)
 
-	_, err = f.ReadAt(buf, 0)
+	_, err := f.ReadAt(buf, 0)
 	if err != nil {
 		return "", "", err
 	}
 
 	h := sha256.Sum256(buf)
+	head := hex.EncodeToString(h[:])
 
-	// When the whole file fits in one chunk the tail window is exactly
-	// the bytes just read: reuse the head hash instead of issuing a
-	// second read for every small file.
-	if size <= int64(chunk) {
-		hh := hex.EncodeToString(h[:])
-
-		return hh, hh, nil
+	if size <= int64(headTailWindow) {
+		return head, head, nil
 	}
 
 	_, err = f.ReadAt(buf, size-n)
@@ -992,5 +1035,55 @@ func hashHeadTail(path string, size int64) (string, string, error) {
 
 	t := sha256.Sum256(buf)
 
-	return hex.EncodeToString(h[:]), hex.EncodeToString(t[:]), nil
+	return head, hex.EncodeToString(t[:]), nil
+}
+
+// hashContent returns the content-rung hash of f: the SHA-256 of the
+// whole file when it is smaller than wholeFileMax, or of sampled
+// windows when it is that size or larger.
+func hashContent(f *os.File, size int64) (string, error) {
+	if size >= int64(wholeFileMax) {
+		return hashSamples(f, size)
+	}
+
+	return hashWhole(f, size)
+}
+
+// hashWhole returns the SHA-256 of the entire file. A SectionReader is
+// used so the read is independent of the offset left by the end-window
+// reads.
+func hashWhole(f *os.File, size int64) (string, error) {
+	h := sha256.New()
+
+	_, err := io.Copy(h, io.NewSectionReader(f, 0, size))
+	if err != nil {
+		return "", err
+	}
+
+	return hex.EncodeToString(h.Sum(nil)), nil
+}
+
+// hashSamples feeds sampleWindow bytes at each gigabyte-aligned offset
+// (0, sampleStride, 2*sampleStride, ... while inside the file), in
+// order, into one hash, each window truncated at end of file. This is
+// the probabilistic large-file rung: two files of equal size agreeing
+// on every sample are reported as duplicates without every byte being
+// read. Because size is part of the signature, files of different sizes
+// never reach this comparison, so the sample boundaries always align.
+func hashSamples(f *os.File, size int64) (string, error) {
+	h := sha256.New()
+	buf := make([]byte, sampleWindow)
+
+	for off := int64(0); off < size; off += int64(sampleStride) {
+		n := min(int64(sampleWindow), size-off)
+
+		_, err := f.ReadAt(buf[:n], off)
+		if err != nil {
+			return "", err
+		}
+
+		h.Write(buf[:n])
+	}
+
+	return hex.EncodeToString(h.Sum(nil)), nil
 }

scan_test.go

@@ -53,7 +53,10 @@ func pattern(tag byte, n int) []byte {
 	return data
 }
 
-func TestHashHeadTail(t *testing.T) {
+// TestHashSignatureEnds exercises the head and tail rungs across the
+// window boundaries. Every file here is below wholeFileMax, so the
+// content rung is a whole-file hash.
+func TestHashSignatureEnds(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
@@ -64,11 +67,11 @@ func TestHashHeadTail(t *testing.T) {
 	}{
 		{"empty", nil},
 		{"one-byte", []byte("x")},
-		{"under-one-chunk", pattern(1, chunk-1)},
-		{"exactly-one-chunk", pattern(2, chunk)},
-		{"overlapping-reads", pattern(3, chunk+chunk/2)},
-		{"exactly-two-chunks", pattern(4, 2*chunk)},
-		{"beyond-two-chunks", pattern(5, 3*chunk)},
+		{"under-one-window", pattern(1, headTailWindow-1)},
+		{"exactly-one-window", pattern(2, headTailWindow)},
+		{"overlapping-windows", pattern(3, headTailWindow+headTailWindow/2)},
+		{"exactly-two-windows", pattern(4, 2*headTailWindow)},
+		{"beyond-two-windows", pattern(5, 3*headTailWindow)},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
@@ -76,12 +79,12 @@ func TestHashHeadTail(t *testing.T) {
 
 			p := writeFile(t, dir, c.name, c.data)
 
-			head, tail, err := hashHeadTail(p, int64(len(c.data)))
+			head, tail, content, err := hashSignature(p, int64(len(c.data)))
 			if err != nil {
-				t.Fatalf("hashHeadTail: %v", err)
+				t.Fatalf("hashSignature: %v", err)
 			}
 
-			n := min(chunk, len(c.data))
+			n := min(headTailWindow, len(c.data))
 			if want := hexSum(c.data[:n]); head != want {
 				t.Errorf("head = %s, want %s", head, want)
 			}
@@ -89,36 +92,179 @@ func TestHashHeadTail(t *testing.T) {
 			if want := hexSum(c.data[len(c.data)-n:]); tail != want {
 				t.Errorf("tail = %s, want %s", tail, want)
 			}
+
+			// Below wholeFileMax the content rung hashes the whole file.
+			if want := hexSum(c.data); content != want {
+				t.Errorf("content = %s, want whole-file %s", content, want)
+			}
 		})
 	}
 }
 
-func TestHashHeadTailErrors(t *testing.T) {
+func TestHashSignatureErrors(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
 
-	_, _, err := hashHeadTail(filepath.Join(dir, "missing"), 1)
+	// A missing file: an error, and every hash left empty.
+	head, tail, content, err := hashSignature(filepath.Join(dir, "missing"), 1)
 	if err == nil {
 		t.Error("no error for a missing file")
 	}
 
+	if head != "" || tail != "" || content != "" {
+		t.Errorf("missing file returned hashes: %q %q %q", head, tail, content)
+	}
+
 	// A zero-length file has constant hashes and is never opened: even
 	// a missing path succeeds.
-	head, tail, err := hashHeadTail(filepath.Join(dir, "missing"), 0)
-	if err != nil || head != emptyHash || tail != emptyHash {
-		t.Errorf("empty: head=%q tail=%q err=%v, want constant hashes",
-			head, tail, err)
+	head, tail, content, err = hashSignature(filepath.Join(dir, "missing"), 0)
+	if err != nil ||
+		head != emptyHash || tail != emptyHash || content != emptyHash {
+		t.Errorf("empty: head=%q tail=%q content=%q err=%v, "+
+			"want constant hashes", head, tail, content, err)
 	}
 
 	// A file that shrank between the stat and hash passes: reading at
 	// the stat-reported size must fail rather than emit wrong hashes.
 	p := writeFile(t, dir, "shrunk", []byte("tiny"))
 
-	_, _, err = hashHeadTail(p, int64(2*chunk))
+	head, tail, content, err = hashSignature(p, int64(2*headTailWindow))
 	if err == nil {
 		t.Error("no error when the stat size exceeds the file size")
 	}
+
+	if head != "" || tail != "" || content != "" {
+		t.Errorf("shrunk file returned hashes: %q %q %q", head, tail, content)
+	}
+}
+
+// sparseFile creates a file that is logically size bytes long without
+// allocating blocks for the hole, so multi-gigabyte cases stay cheap.
+func sparseFile(t *testing.T, dir, name string, size int64) string {
+	t.Helper()
+
+	p := filepath.Join(dir, name)
+
+	f, err := os.Create(p) //nolint:gosec // test-controlled path
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = f.Truncate(size)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = f.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	return p
+}
+
+// pokeAt writes data into an existing file at off, leaving the rest of
+// the file (a sparse hole) untouched.
+func pokeAt(t *testing.T, path string, off int64, data []byte) {
+	t.Helper()
+
+	f, err := os.OpenFile(path, os.O_WRONLY, 0o600) //nolint:gosec // test path
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = f.WriteAt(data, off)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = f.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+}
+
+// contentHash returns just the content rung of a file's signature.
+func contentHash(t *testing.T, path string, size int64) string {
+	t.Helper()
+
+	_, _, content, err := hashSignature(path, size)
+	if err != nil {
+		t.Fatalf("hashSignature %s: %v", path, err)
+	}
+
+	return content
+}
+
+// TestContentRungBoundary checks the 50 MiB boundary between the two
+// content rungs: just below it the whole file is hashed and any byte
+// difference shows; at the boundary only the gigabyte-spaced samples are
+// hashed, so a difference outside a sample window is invisible.
+func TestContentRungBoundary(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	// A byte that lands outside the single [0, sampleWindow) sample a
+	// sub-gigabyte file has, but well inside the file.
+	const off = 10 * 1024 * 1024
+
+	// Just under the boundary: the whole-file rung sees the poked byte.
+	under := int64(wholeFileMax - 1)
+	underBase := sparseFile(t, dir, "under-base", under)
+	underPoked := sparseFile(t, dir, "under-poked", under)
+
+	pokeAt(t, underPoked, off, []byte{1})
+
+	if contentHash(t, underBase, under) == contentHash(t, underPoked, under) {
+		t.Error("whole-file rung ignored a byte difference below wholeFileMax")
+	}
+
+	// At the boundary: only [0, sampleWindow) is sampled, so the poked
+	// byte at off is invisible and the two content hashes match.
+	at := int64(wholeFileMax)
+	atBase := sparseFile(t, dir, "at-base", at)
+	atPoked := sparseFile(t, dir, "at-poked", at)
+
+	pokeAt(t, atPoked, off, []byte{1})
+
+	if contentHash(t, atBase, at) != contentHash(t, atPoked, at) {
+		t.Error("sampled rung saw a byte outside every sample window")
+	}
+}
+
+// TestContentRungMultiGigabyte exercises the sampled rung across several
+// gigabytes using sparse files: a difference inside the third sample
+// window (at offset 2*sampleStride) changes the hash, while a difference
+// in the gap after it does not.
+func TestContentRungMultiGigabyte(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+
+	// Three sample windows (offsets 0, 1 GiB, 2 GiB) plus a trailing gap
+	// that no sample covers.
+	size := int64(2*sampleStride + 2*sampleWindow)
+	thirdSample := int64(2 * sampleStride)
+	gap := thirdSample + int64(sampleWindow)
+
+	base := sparseFile(t, dir, "g-base", size)
+	inSample := sparseFile(t, dir, "g-insample", size)
+	inGap := sparseFile(t, dir, "g-ingap", size)
+
+	pokeAt(t, inSample, thirdSample, []byte{1})
+	pokeAt(t, inGap, gap, []byte{1})
+
+	baseHash := contentHash(t, base, size)
+
+	if contentHash(t, inSample, size) == baseHash {
+		t.Error("sample at 2 GiB was not read: difference there was invisible")
+	}
+
+	if contentHash(t, inGap, size) != baseHash {
+		t.Error("a byte in an unsampled gap changed the content hash")
+	}
 }
 
 // collectWalk runs a walk over roots and returns the emitted records

trees.go

@@ -13,9 +13,10 @@ import (
 
 // fileSig is a file's duplicate signature; mtime is excluded.
 type fileSig struct {
-	size int64
-	head string
-	tail string
+	size    int64
+	head    string
+	tail    string
+	content string
 }
 
 // treeNode is one directory reconstructed from the scan stream.
@@ -122,7 +123,9 @@ func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
 			node.files = make(map[string]fileSig)
 		}
 
-		sig := fileSig{size: r.size, head: r.head, tail: r.tail}
+		sig := fileSig{
+			size: r.size, head: r.head, tail: r.tail, content: r.content,
+		}
 
 		// An unhashed record (its size was unique when last scanned)
 		// has unknown content: give it a signature no other file can
@@ -188,7 +191,7 @@ func (n *treeNode) compute() {
 	for name, sig := range n.files {
 		entries = append(entries,
 			"f\x00"+name+"\x00"+strconv.FormatInt(sig.size, 10)+
-				"\x00"+sig.head+"\x00"+sig.tail)
+				"\x00"+sig.head+"\x00"+sig.tail+"\x00"+sig.content)
 		n.fileCount++
 		n.totalSize += sig.size
 	}