Correct four inaccurate comments in cancel_test.go and rename a constant (closes #33)

Cleanup of the non-blocking findings from the re-review of #6. The tests
were never wrong; only comments that misdescribed the mechanism they
document, and one constant whose name claimed the wrong quantity.

- State the property the walkClock tests rely on (the index load's
  Done cost is bounded and independent of record count) instead of the
  wrong "three consultations" figure.
- Stop the poolUnwind framing from implying every test is bounded at
  two seconds; the three tests that catch their regression only as the
  test binary's timeout now say so.
- Record that hashWorker's results-send abandon branch is reachable
  from the scan path and is covered by TestScanHashWriteFailureUnwindsPool,
  so every cancellation branch has a test.
- Rename walkCancelInFlightDirs to walkCancelInFlightFiles: it is a
  file count (same value).
- Note the deliberate departure from the one-test-file-per-source-file
  convention at the top of the file.

Model: opus-4-8

.dockerignore

@@ -2,6 +2,7 @@
 .claude
 .DS_Store
 sfdupes
+files.dat
 *.log
 *.out
 *.test

.gitignore

@@ -27,6 +27,7 @@ node_modules/
 *.log
 
 # Local scan data
+files.dat
 *.sqlite
 *.sqlite-shm
 *.sqlite-wal

Makefile

@@ -46,4 +46,4 @@ hooks:
 	@script/install-precommit
 
 clean:
-	rm -f $(BINARY)
+	rm -f $(BINARY) files.dat

README.md

@@ -5,19 +5,14 @@
 `sfdupes` is an MIT-licensed Go CLI tool by
 [@sneak](https://sneak.berlin) that quickly identifies *candidate*
 duplicate files — and, ultimately, entire duplicate directory trees —
-across very large filesystems without reading every byte of every file.
+across very large filesystems without reading full file contents. Files
-Files are considered duplicates when their sizes are equal and they
+are considered duplicates when they have identical size, identical
-agree on a short ladder of hashes. A file under 10 MiB is hashed in full
+SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
-and compared directly. A larger file is gated first on the SHA-256 of
+1024 bytes. This is a strong candidate signal, not proof of identical
-its first 64 KiB and of its last 64 KiB, and then compared on a content
+content (the middle of the file is never read); the intended use is
-hash — the SHA-256 of the whole file when it is under 50 MiB, or of
-gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
-the content hash is proof of identical content; at or above 50 MiB it is
-a strong candidate signal rather than proof, because the gaps between
-samples are never read. The intended use is
 finding duplicate downloads and duplicated directory trees on
-multi-terabyte ZFS servers where reading every byte of every file is
+multi-terabyte ZFS servers where reading every byte is prohibitively
-prohibitively expensive. `scan` maintains a persistent SQLite database of file
+expensive. `scan` maintains a persistent SQLite database of file
 signatures that survives between runs, so it can be run from cron and
 the reports can be generated at any time from the most recent scan.
 
@@ -34,8 +29,7 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
 ```
 
 `scan` walks one or more filesystem trees and maintains one database
-record per regular file (path, size, mtime, head hash, tail hash,
+record per regular file (path, size, mtime, head hash, tail hash). The
-content hash). The
 database persists between runs; a rescan only hashes files that are new
 or changed, and removes records for files that no longer exist.
 `report` reads the database and prints the file-level duplicates
@@ -53,13 +47,10 @@ completed scan.
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
-disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
+disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
-whose size at least one other file shares, since a size-unique file
+only from files whose size at least one other file shares, since a
-cannot be a duplicate; for those it reads the cheap end windows first
+size-unique file cannot be a duplicate — makes a full-filesystem sweep
-and a content hash second — the whole file below 50 MiB, but only
+tractable, and the signatures are kept in a persistent database, so
-gigabyte-spaced samples at or above 50 MiB, so the largest files are
-never read in full. This keeps a full-filesystem sweep tractable, and
-the signatures are kept in a persistent database, so
 the expensive filesystem pass is incremental: a rescan re-hashes only
 files whose recorded mtime or size changed, and all analysis happens
 offline from the database alone. The end goal is
@@ -77,18 +68,14 @@ Goals, in order:
     downloads, copied project trees), so the operator can consider
     removing an entire subtree at once. File-level duplicate detection is
     the foundation; tree-level detection is built on top of it.
-2.  **Spend I/O in proportion to duplicate likelihood.** Only files
+2.  **Never read full file contents.** At most 2 KiB is read per file
-    whose size at least one other file shares are read at all — a
+    (first and last 1024 bytes), and only files whose size at least
-    size-unique file cannot be a duplicate. Those are compared by the
+    one other file shares are read at all — a size-unique file cannot
-    ladder in "Duplicate detection" below: a file under 10 MiB is hashed
+    be a duplicate. Scale target: tens of millions of files, ~150 TB
-    in full, while a larger file is gated on cheap 64 KiB end windows
+    filesystem, possibly slow or busy disks (ZFS pool under resilver).
-    first, then a content hash that reads the whole file below 50 MiB
+    Holding one small record (path, size, mtime) per file in memory
-    but only gigabyte-spaced 1 MiB samples at or above it, so the very
+    during a scan is acceptable; holding every file's hashes is not
-    largest files are still never read in full. Scale target: tens of
+    (they stay in the database).
-    millions of files, ~150 TB filesystem, possibly slow or busy disks
-    (ZFS pool under resilver). Holding one small record (path, size,
-    mtime) per file in memory during a scan is acceptable; holding
-    every file's hashes is not (they stay in the database).
 3.  **Scan incrementally, analyze offline.** The expensive filesystem
     scan maintains a persistent database; an unchanged file is never
     read again on a rescan. All analysis (`report`, `trees`) works from
@@ -154,72 +141,26 @@ All three subcommands operate on a single SQLite database file:
     that dies partway leaves a valid database holding everything
     hashed so far; the next scan skips those records and converges
     toward the filesystem.
--   Schema (`PRAGMA user_version` is the schema version, currently 2; a
+-   Schema (`PRAGMA user_version` is the schema version, currently 1; a
-    database with any other version is a fatal error). Version 2 added
+    database with any other version is a fatal error):
-    the `content` column and the 64 KiB head/tail signature (replacing
-    the version 1 1 KiB end windows), so a version 1 database cannot be
-    reused: it is rejected and the tree must be rescanned from scratch.
 
     ```sql
     CREATE TABLE files (
         path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
         size  INTEGER NOT NULL,  -- bytes, from lstat
         mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
-        head    TEXT NOT NULL,     -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
+        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
-        tail    TEXT NOT NULL,     -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
+        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
-        content TEXT NOT NULL      -- lowercase-hex SHA-256, whole file or samples
     ) WITHOUT ROWID;
     ```
 
     Paths are stored as BLOBs because Unix paths are raw bytes, not
     guaranteed UTF-8. `mtime` is used only for change detection; it is
-    not part of the duplicate key. For a file under 10 MiB `head`, `tail`,
+    not part of the duplicate key. `head` and `tail` are empty strings
-    and `content` all hold the whole-file hash (that range is hashed in
+    when the file has never been hashed because its size was unique as
-    full, with no end windows); for a larger file `head` and `tail` hold
+    of the last scan that covered it; such records still define the
-    the first- and last-64 KiB hashes and `content` the whole-file or
+    file for tree reconstruction but never participate in duplicate
-    sampled hash. All three are empty strings when the file has never
+    groups.
-    been hashed because its size was unique as of the last scan that
-    covered it; such records still define the file for tree
-    reconstruction but never participate in duplicate groups.
-
-### Duplicate detection
-
-Two files are duplicates only when they agree on every rung of this
-ladder; a mismatch at any rung means they are not duplicates. `scan`
-stores each file's hashes once, and `report` and `trees` group files by
-the whole signature — size, `head`, `tail`, and `content` — so the
-grouping is exactly this ladder applied across everything scanned into
-the database, even across separate scans.
-
-1.  **Size.** Files of different sizes are never compared. Only files
-    whose size at least one other file shares are hashed at all.
-2.  **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed
-    in full and compared directly, with no separate end-window step —
-    small files are cheap to read to the last byte, and doing so makes
-    the comparison exact. `head`, `tail`, and `content` all hold this
-    whole-file SHA-256, so such a file's signature is decided entirely
-    by its size and its content.
-3.  **10 MiB and above: head and tail.** For a larger file, the SHA-256
-    of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
-    cheap gate that eliminates most same-size pairs before any bulk
-    reading. At 10 MiB and above the two windows never overlap.
-4.  **10 MiB and above, content below 50 MiB.** The SHA-256 of the
-    entire file. Agreement here is proof of identical content (barring a
-    SHA-256 collision).
-5.  **10 MiB and above, content 50 MiB and above.** A sampled SHA-256:
-    the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, …
-    while inside the file, the final window truncated at end of file) is
-    fed, in order, into one hash. This is **deliberately probabilistic**
-    — the gaps between samples are never read, so two large files that
-    agree on every sample are reported as duplicates without being read
-    in full. It is the price of never reading a 150 GB file end to end.
-    Because size is already part of the signature, only equal-size files
-    reach this rung, so their sample boundaries always align.
-
-`head`, `tail`, and `content` are one column each. A file below 10 MiB
-and one at or above it never share a size, and neither do a file below
-50 MiB and one at or above it, so a stored value is never ambiguous
-between the whole-file, end-window, and sampled forms.
 
 ### `scan` mode
 
@@ -288,9 +229,10 @@ during the hash phase:
     decides its fate. Size-unique files are never read: new or
     changed ones are recorded without hashes in the update phase,
     unchanged unhashed ones simply keep their records. Every file
-    with a shared size is hashed by the worker pool, computing the
+    with a shared size is hashed by the worker pool: read the first
-    full signature — head, tail, and content — described in "Duplicate
+    `min(1024, size)` bytes and the last `min(1024, size)` bytes
-    detection" below. Zero-length files have constant
+    (one read when `size <= 1024`, since the two windows coincide)
+    and compute the SHA-256 of each. Zero-length files have constant
     hashes and are never opened. Files are hashed in **inode order**
     (minimizing seeks on spinning disks), and paths that are hard
     links to the same inode are **read once**, all sharing the one

TODO.md

@@ -29,23 +29,10 @@
 
 # Completed Steps
 
-- replace the 1 KiB end-window sampling with the head/tail plus
+- correct four inaccurate comments in `cancel_test.go` and rename
-  content-hash ladder (2026-09-22, branch `next`, closes
+  `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-09-21,
-  https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
+  branch `issue-33-cancel-test-comments`, closes
-  hashed in full and compared directly, with no end-window step — its
+  https://git.eeqj.de/sneak/sfdupes/issues/33)
-  `head`, `tail`, and `content` all hold the whole-file hash. A file at
-  10 MiB or above is gated on the 64 KiB `head` and `tail`, then
-  compared on a `content` hash — the whole file below 50 MiB,
-  gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
-  (new `content` column); a version 1 database is rejected and must be
-  rescanned, which is required anyway since every stored hash changed.
-  `report` and `trees` group by the extended signature, so the ladder is
-  applied across the whole database. README "Duplicate detection"
-  documents every rung including the probabilistic large-file path.
-
-- remove the dead `files.dat` references from `Makefile`, `.gitignore`
-  and `.dockerignore` (2026-09-21, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/22)
 
 - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
   `Dockerfile.lint` (2026-08-10, branch `next`, closes

cancel_test.go

@@ -13,11 +13,20 @@ import (
 	"time"
 )
 
-// poolUnwind bounds how long a goroutine is given to leave a pool
+// This file gathers the tests for scan cancellation and worker-pool
-// after its context is cancelled. Only a failing run ever waits this
+// unwinding. Everything it exercises lives in scan.go, so by the repo's
-// long: a pool that ignored its cancellation parks forever, and this
+// convention of one test file per source file it would belong in
-// is what turns that into a failed assertion instead of a suite that
+// scan_test.go. It is kept separate on purpose: cancellation behaviour
-// hangs until the test binary's own timeout.
+// cuts across both the walk pool and the hash pool as a single concern,
+// and scan_test.go is already 900+ lines. That is the deliberate
+// exception the convention otherwise expects to be stated.
+
+// poolUnwind bounds how long a goroutine is given to leave a pool after
+// its context is cancelled. The tests that use it turn a pool that
+// ignored its cancellation — and so parks forever — into a failed
+// assertion within this bound instead of a hang. Not every test in this
+// file has that property: a few catch their regression only as the test
+// binary's own timeout, and each of those says so.
 const poolUnwind = 2 * time.Second
 
 // walkClock is a context whose cancellation is driven by the scan's
@@ -28,9 +37,12 @@ const poolUnwind = 2 * time.Second
 //
 // The accounting behind the n chosen by each test: every blocking
 // channel operation in the walk selects on Done, so the walk spends
-// one consultation per file event plus a couple per directory, while
+// one consultation per file event plus a couple per directory. The
-// the index load that runs ahead of it spends a small fixed number
+// index load that runs ahead of it also consults Done, but a bounded
-// (three) whatever the record count.
+// number of times that does not grow with the record count. The tests
+// depend on that property, not on the bound's exact value: each picks
+// an n comfortably above it and well short of the walk's total, so the
+// cancellation lands deep inside the walk whatever the record count.
 type walkClock struct {
 	n    int64
 	seen atomic.Int64
@@ -90,7 +102,11 @@ const (
 	walkCancelFilesPerDir = 20
 	walkCancelFiles       = walkCancelDirs * walkCancelFilesPerDir
 	walkCancelWorkers     = 4
-	walkCancelInFlightDirs = walkCancelWorkers * walkCancelFilesPerDir
+	// The files carried by the walkCancelWorkers directories already in
+	// flight when the scan is cancelled: each such directory can still
+	// emit its walkCancelFilesPerDir files. This is a file count, not a
+	// directory count.
+	walkCancelInFlightFiles = walkCancelWorkers * walkCancelFilesPerDir
 )
 
 // walkCancelAtDone is the consultation on which the fixture's context
@@ -154,6 +170,11 @@ func assertRecordsIntact(t *testing.T, db *sql.DB, before []string) {
 // phase and fail there instead, with a wrapped error rather than the
 // bare cancellation.
 //
+// The syncScan call here is not bounded by poolUnwind: a regression
+// that left a worker pool parked would hang it, and that regression is
+// caught only by the test binary's own timeout, not by a quick
+// assertion.
+//
 //nolint:paralleltest // counts goroutines: must not run beside others
 func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 	dir := buildWalkCancelTree(t)
@@ -211,10 +232,11 @@ func assertWalkGuardAborted(t *testing.T, st scanStats, err error) {
 	}
 
 	// The workers drop every directory still queued once the scan is
-	// cancelled, so only the directories already in flight can add to
+	// cancelled, so only the files in the directories already in flight
-	// the census after the fact. A census beyond that bound would mean
+	// can add to the census after the fact. A census beyond that bound
-	// the cancellation was not observed where it should have been.
+	// would mean the cancellation was not observed where it should have
-	limit := walkCancelAtDone + walkCancelInFlightDirs
+	// been.
+	limit := walkCancelAtDone + walkCancelInFlightFiles
 	if st.unchanged > limit {
 		t.Errorf("census covers %d files, want at most %d: the walk kept "+
 			"taking directories off the queue after cancellation",
@@ -412,7 +434,11 @@ func TestDispatchDirsClosesJobsWhenCancelled(t *testing.T) {
 
 // TestFeedHashJobsClosesJobsWhenCancelled checks that the hash feeder
 // abandons the runs it has not queued yet and still closes the job
-// channel, which is what lets the workers' range terminate.
+// channel, which is what lets the workers' range terminate. The
+// receive on jobs below is not bounded: a feeder that returned without
+// closing jobs would leave that receive with no sender and no close, so
+// this regression is caught by the test binary's timeout rather than by
+// a bounded assertion.
 func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
 	t.Parallel()
 
@@ -440,6 +466,16 @@ func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
 // nobody wants the hashes of — while still letting the range run out
 // so the pool tears down. The queued run names a file that does not
 // exist, so a worker that hashed it anyway would produce a result.
+//
+// hashWorker has a second cancellation exit: the send of a completed
+// result on the results channel. That branch is reachable from the
+// production scan path, not dead code — pool.stop() cancels the context
+// before it starts draining results, so a worker parked on that send
+// leaves through this case, freed by the drain rather than by an empty
+// jobs channel. It is exercised by TestScanHashWriteFailureUnwindsPool,
+// which strands every worker on a full results channel until stop()
+// unwinds the pool. With both branches covered, every cancellation
+// branch of the walk and hash pools has a test.
 func TestHashWorkerDropsQueuedRuns(t *testing.T) {
 	t.Parallel()
 
@@ -472,7 +508,10 @@ func TestHashWorkerDropsQueuedRuns(t *testing.T) {
 // TestHashPhaseCancelledReturnsContextError checks the result loop's
 // own exit: with the pool cancelled, no result will ever arrive, and
 // the loop must leave through the cancellation rather than wait for a
-// receive that cannot happen.
+// receive that cannot happen. This call is not bounded by poolUnwind: a
+// loop that dropped its cancellation case would block on that receive,
+// so the regression surfaces as the test binary's timeout rather than
+// as a bounded assertion.
 func TestHashPhaseCancelledReturnsContextError(t *testing.T) {
 	t.Parallel()
 

db.go

@@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
 const databaseEnv = "SFDUPES_DATABASE"
 
 // schemaVersion is the database schema version this build reads and
-// writes, stored in PRAGMA user_version. Version 2 adds the content
+// writes, stored in PRAGMA user_version.
-// column and the head/tail/content signature (replacing the version 1
+const schemaVersion = 1
-// 1 KiB end windows), so a version 1 database is rejected and must be
-// rescanned.
-const schemaVersion = 2
 
 // dbDirPerm is the mode for a database parent directory created by
 // scan.
@@ -43,20 +40,18 @@ CREATE TABLE files (
     size  INTEGER NOT NULL,
     mtime INTEGER NOT NULL,
     head  TEXT NOT NULL,
-    tail    TEXT NOT NULL,
+    tail  TEXT NOT NULL
-    content TEXT NOT NULL
 ) WITHOUT ROWID
 `
 
 // upsertSQL inserts one file record, replacing any existing record for
 // the same path.
 const upsertSQL = `
-INSERT INTO files (path, size, mtime, head, tail, content)
+INSERT INTO files (path, size, mtime, head, tail)
-VALUES (?, ?, ?, ?, ?, ?)
+VALUES (?, ?, ?, ?, ?)
 ON CONFLICT (path) DO UPDATE SET
     size = excluded.size, mtime = excluded.mtime,
-    head = excluded.head, tail = excluded.tail,
+    head = excluded.head, tail = excluded.tail
-    content = excluded.content
 `
 
 // errNoDatabase reports a missing database file for report/trees.
@@ -210,7 +205,7 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
 // loadFileRows reads every record from the files table.
 func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
 	rows, err := db.QueryContext(ctx,
-		"SELECT path, size, mtime, head, tail, content FROM files")
+		"SELECT path, size, mtime, head, tail FROM files")
 	if err != nil {
 		return nil, fmt.Errorf("read records: %w", err)
 	}
@@ -225,8 +220,7 @@ func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
 			r    scanRec
 		)
 
-		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail,
+		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail)
-			&r.content)
 		if err != nil {
 			return nil, fmt.Errorf("read record: %w", err)
 		}
@@ -354,7 +348,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
 
 	for _, r := range upserts {
 		_, err = st.ExecContext(ctx,
-			[]byte(r.path), r.size, r.mtime, r.head, r.tail, r.content)
+			[]byte(r.path), r.size, r.mtime, r.head, r.tail)
 		if err != nil {
 			return fmt.Errorf("upsert %s: %w", r.path, err)
 		}

report.go

@@ -17,14 +17,13 @@ const ioBufSize = 1 << 20
 const minGroupSize = 2
 
 // scanRec is one file record from the database. The signature (size,
-// head, tail, content) is the duplicate key; mtime is informational
+// head, tail) is the duplicate key; mtime is informational only and
-// only and used by scan for change detection.
+// used by scan for change detection.
 type scanRec struct {
 	size  int64
 	mtime int64
 	head  string
 	tail  string
-	content string
 	path  string
 }
 
@@ -53,9 +52,8 @@ func loadRecords(ctx context.Context) ([]scanRec, error) {
 }
 
 // dupeGroup is one set of candidate-duplicate files: identical size,
-// head hash, tail hash, and content hash. paths is sorted
+// head hash, and tail hash. paths is sorted lexicographically; the
-// lexicographically; the first entry is the group's "first", the rest
+// first entry is the group's "first", the rest are dupes.
-// are dupes.
 type dupeGroup struct {
 	size  int64
 	paths []string
@@ -124,9 +122,7 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
 			continue
 		}
 
-		k := fileSig{
+		k := fileSig{size: r.size, head: r.head, tail: r.tail}
-			size: r.size, head: r.head, tail: r.tail, content: r.content,
-		}
 		groups[k] = append(groups[k], r.path)
 	}
 

report_test.go

@@ -38,28 +38,6 @@ func TestCollectDupeGroups(t *testing.T) {
 	}
 }
 
-func TestCollectDupeGroupsContentSeparates(t *testing.T) {
-	t.Parallel()
-
-	// Same size, head, and tail, but different content hashes: the final
-	// rung keeps them apart, so no group forms. Matching content groups.
-	recs := []scanRec{
-		{size: 100, head: "h", tail: "t", content: "c1", path: "/a"},
-		{size: 100, head: "h", tail: "t", content: "c2", path: "/b"},
-		{size: 100, head: "h", tail: "t", content: "c1", path: "/c"},
-	}
-
-	groups := collectDupeGroups(recs)
-	if len(groups) != 1 {
-		t.Fatalf("len(groups) = %d, want 1 (only the matching content)",
-			len(groups))
-	}
-
-	if !slices.Equal(groups[0].paths, []string{"/a", "/c"}) {
-		t.Errorf("group paths = %q, want /a /c", groups[0].paths)
-	}
-}
-
 func TestCollectDupeGroupsMtimeExcluded(t *testing.T) {
 	t.Parallel()
 

scan.go

@@ -7,7 +7,6 @@ import (
 	"database/sql"
 	"encoding/hex"
 	"fmt"
-	"io"
 	"io/fs"
 	"os"
 	"path/filepath"
@@ -17,37 +16,8 @@ import (
 	"syscall"
 )
 
-// The duplicate ladder (see hashSignature and README "Duplicate
+// chunk is the number of bytes hashed from each end of a file.
-// detection"). A same-size candidate below headTailMin is hashed in
+const chunk = 1024
-// full and compared directly; a larger one is separated first by the
-// hashes of its end windows, then by a content hash that is exact below
-// wholeFileMax and deliberately sampled at or above it.
-
-// headTailMin is the size threshold for the end-window gate. A file
-// smaller than this is hashed in full directly, with no separate head
-// and tail step: its head, tail, and content all carry the whole-file
-// hash. A file this size or larger is separated first by its end
-// windows.
-const headTailMin = 10 * 1024 * 1024
-
-// headTailWindow is the number of bytes hashed from each end of a file
-// at or above headTailMin (the head and tail rungs). Because
-// headTailMin is far larger than two windows, the head and tail windows
-// never overlap.
-const headTailWindow = 64 * 1024
-
-// wholeFileMax is the size boundary between the two content rungs: a
-// file strictly smaller than this is content-hashed in full; a file
-// this size or larger is content-hashed by sampling.
-const wholeFileMax = 50 * 1024 * 1024
-
-// sampleStride is the spacing between content samples for large files:
-// one window is read at each gigabyte-aligned offset (0, 1 GiB, ...).
-const sampleStride = 1024 * 1024 * 1024
-
-// sampleWindow is the number of bytes read at each large-file sample
-// offset, truncated at end of file.
-const sampleWindow = 1024 * 1024
 
 // workQueueDepth bounds the job and result channels feeding the walk
 // and hash worker pools.
@@ -473,7 +443,6 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
 			mtime: rec.mtime,
 			head:  r.head,
 			tail:  r.tail,
-			content: r.content,
 			path:  rec.path,
 		})
 	}
@@ -865,14 +834,12 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
 	return statDev(st), st.Ino
 }
 
-// hashResult carries one inode run's signature hashes — head, tail, and
+// hashResult carries one inode run's head/tail hashes (or the error
-// content — (or the error that prevented hashing it) from the hash
+// that prevented hashing it) from the hash workers to the hash phase.
-// workers to the hash phase.
 type hashResult struct {
 	run  []fileRec
 	head string
 	tail string
-	content string
 	err  error
 }
 
@@ -963,11 +930,11 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 			continue
 		}
 
-		head, tail, content, err := hashSignature(run[0].path, run[0].size)
+		head, tail, err := hashHeadTail(run[0].path, run[0].size)
 
 		select {
 		case results <- hashResult{
-			run: run, head: head, tail: tail, content: content, err: err,
+			run: run, head: head, tail: tail, err: err,
 		}:
 		case <-ctx.Done():
 			return
@@ -975,138 +942,55 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 	}
 }
 
-// emptyHash is the lowercase-hex SHA-256 of the empty input: the head,
+// emptyHash is the lowercase-hex SHA-256 of the empty input: the head
-// tail, and content hash of every zero-length file.
+// and tail hash of every zero-length file.
 const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
 	"27ae41e4649b934ca495991b7852b855"
 
-// hashSignature computes the three content hashes that, with the file
+// hashHeadTail returns the lowercase-hex SHA-256 of the first
-// size, form its duplicate signature. A file below headTailMin is
+// min(chunk, size) bytes and of the last min(chunk, size) bytes of the
-// hashed in full and its whole-file SHA-256 is returned as head, tail,
+// file at path. The two reads overlap when size < 2*chunk. size is the
-// and content alike — that range takes no separate end-window step. For
+// value recorded when the file was statted; a zero-length file's
-// a file at or above headTailMin the head and tail are the SHA-256 of
+// hashes are constant, so it is never even opened.
-// its first and last headTailWindow bytes, and content is the SHA-256
+func hashHeadTail(path string, size int64) (string, string, error) {
-// of the whole file below wholeFileMax (the exact rung) or of
-// gigabyte-spaced samples at or above it (the sampled, deliberately
-// probabilistic rung). Two files are duplicates only when all four
-// agree; any mismatch means not a duplicate. size is the value recorded
-// when the file was statted; a zero-length file has constant hashes and
-// is never opened.
-func hashSignature(path string, size int64) (string, string, string, error) {
 	if size == 0 {
-		return emptyHash, emptyHash, emptyHash, nil
+		return emptyHash, emptyHash, nil
 	}
 
 	//nolint:gosec // hashing operator-supplied paths is the tool's purpose
 	f, err := os.Open(path)
 	if err != nil {
-		return "", "", "", err
+		return "", "", err
 	}
 
 	defer func() { _ = f.Close() }()
 
-	// Below the threshold the whole file is hashed directly, with no
+	n := min(int64(chunk), size)
-	// end-window step: head and tail both carry the whole-file hash.
-	if size < int64(headTailMin) {
-		content, err := hashWhole(f, size)
-		if err != nil {
-			return "", "", "", err
-		}
 
-		return content, content, content, nil
+	buf := make([]byte, n)
-	}
 
-	head, tail, err := hashEnds(f, size)
+	_, err = f.ReadAt(buf, 0)
-	if err != nil {
-		return "", "", "", err
-	}
-
-	content, err := hashContent(f, size)
-	if err != nil {
-		return "", "", "", err
-	}
-
-	return head, tail, content, nil
-}
-
-// hashEnds returns the SHA-256 of the first and last headTailWindow
-// bytes of f. It is called only for files at least headTailMin, which
-// is far larger than two windows, so the windows never overlap and both
-// reads are always full.
-func hashEnds(f *os.File, size int64) (string, string, error) {
-	buf := make([]byte, headTailWindow)
-
-	_, err := f.ReadAt(buf, 0)
 	if err != nil {
 		return "", "", err
 	}
 
 	h := sha256.Sum256(buf)
-	head := hex.EncodeToString(h[:])
 
-	_, err = f.ReadAt(buf, size-int64(headTailWindow))
+	// When the whole file fits in one chunk the tail window is exactly
+	// the bytes just read: reuse the head hash instead of issuing a
+	// second read for every small file.
+	if size <= int64(chunk) {
+		hh := hex.EncodeToString(h[:])
+
+		return hh, hh, nil
+	}
+
+	_, err = f.ReadAt(buf, size-n)
 	if err != nil {
 		return "", "", err
 	}
 
 	t := sha256.Sum256(buf)
 
-	return head, hex.EncodeToString(t[:]), nil
+	return hex.EncodeToString(h[:]), hex.EncodeToString(t[:]), nil
-}
-
-// hashContent returns the content-rung hash of f: the SHA-256 of the
-// whole file when it is smaller than wholeFileMax, or of sampled
-// windows when it is that size or larger.
-func hashContent(f *os.File, size int64) (string, error) {
-	if size >= int64(wholeFileMax) {
-		return hashSamples(f, size)
-	}
-
-	return hashWhole(f, size)
-}
-
-// hashWhole returns the SHA-256 of the entire file. A SectionReader is
-// used so the read is independent of the offset left by any end-window
-// reads. Reading fewer than size bytes means the file shrank between
-// the stat and the hash; that is an error rather than a hash of content
-// that no longer matches the recorded size.
-func hashWhole(f *os.File, size int64) (string, error) {
-	h := sha256.New()
-
-	n, err := io.Copy(h, io.NewSectionReader(f, 0, size))
-	if err != nil {
-		return "", err
-	}
-
-	if n != size {
-		return "", fmt.Errorf("read %d of %d bytes: %w", n, size,
-			io.ErrUnexpectedEOF)
-	}
-
-	return hex.EncodeToString(h.Sum(nil)), nil
-}
-
-// hashSamples feeds sampleWindow bytes at each gigabyte-aligned offset
-// (0, sampleStride, 2*sampleStride, ... while inside the file), in
-// order, into one hash, each window truncated at end of file. This is
-// the probabilistic large-file rung: two files of equal size agreeing
-// on every sample are reported as duplicates without every byte being
-// read. Because size is part of the signature, files of different sizes
-// never reach this comparison, so the sample boundaries always align.
-func hashSamples(f *os.File, size int64) (string, error) {
-	h := sha256.New()
-	buf := make([]byte, sampleWindow)
-
-	for off := int64(0); off < size; off += int64(sampleStride) {
-		n := min(int64(sampleWindow), size-off)
-
-		_, err := f.ReadAt(buf[:n], off)
-		if err != nil {
-			return "", err
-		}
-
-		h.Write(buf[:n])
-	}
-
-	return hex.EncodeToString(h.Sum(nil)), nil
 }

scan_test.go

@@ -53,23 +53,7 @@ func pattern(tag byte, n int) []byte {
 	return data
 }
 
-// sig returns a file's full signature (head, tail, content), failing the
+func TestHashHeadTail(t *testing.T) {
-// test on any error.
-func sig(t *testing.T, path string, size int64) (string, string, string) {
-	t.Helper()
-
-	head, tail, content, err := hashSignature(path, size)
-	if err != nil {
-		t.Fatalf("hashSignature %s: %v", path, err)
-	}
-
-	return head, tail, content
-}
-
-// TestHashSignatureBelowThreshold verifies that a file below headTailMin
-// is hashed in full and compared directly: head, tail, and content all
-// carry the whole-file SHA-256, with no separate end-window step.
-func TestHashSignatureBelowThreshold(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
@@ -78,10 +62,13 @@ func TestHashSignatureBelowThreshold(t *testing.T) {
 		name string
 		data []byte
 	}{
+		{"empty", nil},
 		{"one-byte", []byte("x")},
-		{"one-window", pattern(1, headTailWindow)},
+		{"under-one-chunk", pattern(1, chunk-1)},
-		{"several-windows", pattern(2, 3*headTailWindow)},
+		{"exactly-one-chunk", pattern(2, chunk)},
-		{"near-threshold", pattern(3, headTailMin-1)},
+		{"overlapping-reads", pattern(3, chunk+chunk/2)},
+		{"exactly-two-chunks", pattern(4, 2*chunk)},
+		{"beyond-two-chunks", pattern(5, 3*chunk)},
 	}
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
@@ -89,240 +76,49 @@ func TestHashSignatureBelowThreshold(t *testing.T) {
 
 			p := writeFile(t, dir, c.name, c.data)
 
-			head, tail, content := sig(t, p, int64(len(c.data)))
+			head, tail, err := hashHeadTail(p, int64(len(c.data)))
+			if err != nil {
+				t.Fatalf("hashHeadTail: %v", err)
+			}
 
-			whole := hexSum(c.data)
+			n := min(chunk, len(c.data))
-			if head != whole || tail != whole || content != whole {
+			if want := hexSum(c.data[:n]); head != want {
-				t.Errorf("head=%s tail=%s content=%s, want all whole-file %s",
+				t.Errorf("head = %s, want %s", head, want)
-					head, tail, content, whole)
+			}
+
+			if want := hexSum(c.data[len(c.data)-n:]); tail != want {
+				t.Errorf("tail = %s, want %s", tail, want)
 			}
 		})
 	}
 }
 
-// TestHashSignatureEnds exercises the head and tail rungs, which apply
+func TestHashHeadTailErrors(t *testing.T) {
-// only to files at least headTailMin. Sparse files keep the fixtures
-// cheap: a difference in the first window changes only head, a
-// difference in the last window changes only tail, and a difference
-// between the windows changes neither end hash but does change the
-// whole-file content rung (the file is below wholeFileMax).
-func TestHashSignatureEnds(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
 
-	// Between headTailMin and wholeFileMax: the end-window gate is active
+	_, _, err := hashHeadTail(filepath.Join(dir, "missing"), 1)
-	// and the content rung is a whole-file hash.
-	const size = int64(headTailMin + 2*1024*1024)
-
-	base := sparseFile(t, dir, "ends-base", size)
-	headDiff := sparseFile(t, dir, "ends-head", size)
-	tailDiff := sparseFile(t, dir, "ends-tail", size)
-	midDiff := sparseFile(t, dir, "ends-mid", size)
-
-	pokeAt(t, headDiff, 0, []byte{1})
-	pokeAt(t, tailDiff, size-1, []byte{1})
-	pokeAt(t, midDiff, size/2, []byte{1})
-
-	bHead, bTail, bContent := sig(t, base, size)
-
-	h, tl, c := sig(t, headDiff, size)
-	if h == bHead {
-		t.Error("a byte in the first window did not change head")
-	}
-
-	if tl != bTail {
-		t.Error("a byte in the first window changed tail")
-	}
-
-	if c == bContent {
-		t.Error("a byte in the first window did not change content")
-	}
-
-	h, tl, c = sig(t, tailDiff, size)
-	if tl == bTail {
-		t.Error("a byte in the last window did not change tail")
-	}
-
-	if h != bHead {
-		t.Error("a byte in the last window changed head")
-	}
-
-	if c == bContent {
-		t.Error("a byte in the last window did not change content")
-	}
-
-	h, tl, c = sig(t, midDiff, size)
-	if h != bHead || tl != bTail {
-		t.Error("a byte between the windows changed an end hash")
-	}
-
-	if c == bContent {
-		t.Error("whole-file content rung ignored a byte between the windows")
-	}
-}
-
-func TestHashSignatureErrors(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-
-	// A missing file: an error, and every hash left empty.
-	head, tail, content, err := hashSignature(filepath.Join(dir, "missing"), 1)
 	if err == nil {
 		t.Error("no error for a missing file")
 	}
 
-	if head != "" || tail != "" || content != "" {
-		t.Errorf("missing file returned hashes: %q %q %q", head, tail, content)
-	}
-
 	// A zero-length file has constant hashes and is never opened: even
 	// a missing path succeeds.
-	head, tail, content, err = hashSignature(filepath.Join(dir, "missing"), 0)
+	head, tail, err := hashHeadTail(filepath.Join(dir, "missing"), 0)
-	if err != nil ||
+	if err != nil || head != emptyHash || tail != emptyHash {
-		head != emptyHash || tail != emptyHash || content != emptyHash {
+		t.Errorf("empty: head=%q tail=%q err=%v, want constant hashes",
-		t.Errorf("empty: head=%q tail=%q content=%q err=%v, "+
+			head, tail, err)
-			"want constant hashes", head, tail, content, err)
 	}
 
 	// A file that shrank between the stat and hash passes: reading at
 	// the stat-reported size must fail rather than emit wrong hashes.
 	p := writeFile(t, dir, "shrunk", []byte("tiny"))
 
-	head, tail, content, err = hashSignature(p, int64(2*headTailWindow))
+	_, _, err = hashHeadTail(p, int64(2*chunk))
 	if err == nil {
 		t.Error("no error when the stat size exceeds the file size")
 	}
-
-	if head != "" || tail != "" || content != "" {
-		t.Errorf("shrunk file returned hashes: %q %q %q", head, tail, content)
-	}
-}
-
-// sparseFile creates a file that is logically size bytes long without
-// allocating blocks for the hole, so multi-gigabyte cases stay cheap.
-func sparseFile(t *testing.T, dir, name string, size int64) string {
-	t.Helper()
-
-	p := filepath.Join(dir, name)
-
-	f, err := os.Create(p) //nolint:gosec // test-controlled path
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = f.Truncate(size)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = f.Close()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	return p
-}
-
-// pokeAt writes data into an existing file at off, leaving the rest of
-// the file (a sparse hole) untouched.
-func pokeAt(t *testing.T, path string, off int64, data []byte) {
-	t.Helper()
-
-	f, err := os.OpenFile(path, os.O_WRONLY, 0o600) //nolint:gosec // test path
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	_, err = f.WriteAt(data, off)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = f.Close()
-	if err != nil {
-		t.Fatal(err)
-	}
-}
-
-// contentHash returns just the content rung of a file's signature.
-func contentHash(t *testing.T, path string, size int64) string {
-	t.Helper()
-
-	_, _, content := sig(t, path, size)
-
-	return content
-}
-
-// TestContentRungBoundary checks the 50 MiB boundary between the two
-// content rungs: just below it the whole file is hashed and any byte
-// difference shows; at the boundary only the gigabyte-spaced samples are
-// hashed, so a difference outside a sample window is invisible.
-func TestContentRungBoundary(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-
-	// A byte that lands outside the single [0, sampleWindow) sample a
-	// sub-gigabyte file has, but well inside the file.
-	const off = 10 * 1024 * 1024
-
-	// Just under the boundary: the whole-file rung sees the poked byte.
-	under := int64(wholeFileMax - 1)
-	underBase := sparseFile(t, dir, "under-base", under)
-	underPoked := sparseFile(t, dir, "under-poked", under)
-
-	pokeAt(t, underPoked, off, []byte{1})
-
-	if contentHash(t, underBase, under) == contentHash(t, underPoked, under) {
-		t.Error("whole-file rung ignored a byte difference below wholeFileMax")
-	}
-
-	// At the boundary: only [0, sampleWindow) is sampled, so the poked
-	// byte at off is invisible and the two content hashes match.
-	at := int64(wholeFileMax)
-	atBase := sparseFile(t, dir, "at-base", at)
-	atPoked := sparseFile(t, dir, "at-poked", at)
-
-	pokeAt(t, atPoked, off, []byte{1})
-
-	if contentHash(t, atBase, at) != contentHash(t, atPoked, at) {
-		t.Error("sampled rung saw a byte outside every sample window")
-	}
-}
-
-// TestContentRungMultiGigabyte exercises the sampled rung across several
-// gigabytes using sparse files: a difference inside the third sample
-// window (at offset 2*sampleStride) changes the hash, while a difference
-// in the gap after it does not.
-func TestContentRungMultiGigabyte(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-
-	// Three sample windows (offsets 0, 1 GiB, 2 GiB) plus a trailing gap
-	// that no sample covers.
-	size := int64(2*sampleStride + 2*sampleWindow)
-	thirdSample := int64(2 * sampleStride)
-	gap := thirdSample + int64(sampleWindow)
-
-	base := sparseFile(t, dir, "g-base", size)
-	inSample := sparseFile(t, dir, "g-insample", size)
-	inGap := sparseFile(t, dir, "g-ingap", size)
-
-	pokeAt(t, inSample, thirdSample, []byte{1})
-	pokeAt(t, inGap, gap, []byte{1})
-
-	baseHash := contentHash(t, base, size)
-
-	if contentHash(t, inSample, size) == baseHash {
-		t.Error("sample at 2 GiB was not read: difference there was invisible")
-	}
-
-	if contentHash(t, inGap, size) != baseHash {
-		t.Error("a byte in an unsampled gap changed the content hash")
-	}
 }
 
 // collectWalk runs a walk over roots and returns the emitted records

trees.go

@@ -16,7 +16,6 @@ type fileSig struct {
 	size int64
 	head string
 	tail string
-	content string
 }
 
 // treeNode is one directory reconstructed from the scan stream.
@@ -123,9 +122,7 @@ func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
 			node.files = make(map[string]fileSig)
 		}
 
-		sig := fileSig{
+		sig := fileSig{size: r.size, head: r.head, tail: r.tail}
-			size: r.size, head: r.head, tail: r.tail, content: r.content,
-		}
 
 		// An unhashed record (its size was unique when last scanned)
 		// has unknown content: give it a signature no other file can
@@ -191,7 +188,7 @@ func (n *treeNode) compute() {
 	for name, sig := range n.files {
 		entries = append(entries,
 			"f\x00"+name+"\x00"+strconv.FormatInt(sig.size, 10)+
-				"\x00"+sig.head+"\x00"+sig.tail+"\x00"+sig.content)
+				"\x00"+sig.head+"\x00"+sig.tail)
 		n.fileCount++
 		n.totalSize += sig.size
 	}