Compute the content hash only when head and tail match (closes #61)

A file of 10 MiB or more now gets only its 64 KiB head and tail in the
hash phase, so its content is read only when it can be a duplicate. A
new content phase after the update phase finds every group of records,
anywhere in the database, that share size, head and tail and include
one without a content hash. It checks every member with lstat and, when
at least two pass, reads those without a content hash through the
existing worker pool; a stale file does not count as a match. report
and trees leave out records without a content hash. The README, help
text and TODO entry describe the gate; the schema stays at version 1.

Lint suppressed: gosec on the file open in hashContentOnly, as in
hashSignature, and on one chmod in a test.

Model: opus-5-5

README.md

@@ -9,8 +9,9 @@ across very large filesystems without reading every byte of every file.
 Files are considered duplicates when their sizes are equal and they
 agree on a short ladder of hashes. A file under 10 MiB is hashed in full
 and compared directly. A larger file is gated first on the SHA-256 of
-its first 64 KiB and of its last 64 KiB, and then compared on a content
-hash — the SHA-256 of the whole file when it is under 50 MiB, or of
+its first 64 KiB and of its last 64 KiB, and only when its size and
+both of those match another file's is it read for a content hash to
+compare — the SHA-256 of the whole file when it is under 50 MiB, or of
 gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
 the content hash is proof of identical content; at or above 50 MiB it is
 a strong candidate signal rather than proof, because the gaps between
@@ -37,7 +38,8 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
 record per regular file (path, size, mtime, head hash, tail hash,
 content hash). The
 database persists between runs; a rescan only hashes files that are new
-or changed, and removes records for files that no longer exist.
+or changed, or that may have gained a duplicate since the last scan,
+and removes records for files that no longer exist.
 `report` reads the database and prints the file-level duplicates
 report. `trees` reads the same database and prints the duplicate-tree
 report. A missing/invalid subcommand — or a `scan` invocation with no
@@ -55,14 +57,17 @@ Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
 disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
 whose size at least one other file shares, since a size-unique file
-cannot be a duplicate; for those it reads the cheap end windows first
-and a content hash second — the whole file below 50 MiB, but only
-gigabyte-spaced samples at or above 50 MiB, so the largest files are
-never read in full. This keeps a full-filesystem sweep tractable, and
-the signatures are kept in a persistent database, so
-the expensive filesystem pass is incremental: a rescan re-hashes only
-files whose recorded mtime or size changed, and all analysis happens
-offline from the database alone. The end goal is
+cannot be a duplicate. Of those, a file under 10 MiB is read in full; a
+larger one has its cheap end windows read first, and is read for a
+content hash only when its size and both end windows match another
+file's — the whole file below 50 MiB, but only gigabyte-spaced samples
+at or above 50 MiB, so the largest files are never read in full. This
+keeps a full-filesystem sweep tractable, and the signatures are kept in
+a persistent database, so the expensive filesystem pass is incremental:
+a rescan re-hashes only files whose recorded mtime or size changed, plus
+— for its content hash — a file of 10 MiB or more whose size and end
+windows have come to match another file's. All analysis happens offline
+from the database alone. The end goal is
 not individual files but whole duplicated trees — duplicate
 extractions, duplicate downloads, copied project trees — which an
 operator can consider removing as a unit.
@@ -82,16 +87,19 @@ Goals, in order:
     size-unique file cannot be a duplicate. Those are compared by the
     ladder in "Duplicate detection" below: a file under 10 MiB is hashed
     in full, while a larger file is gated on cheap 64 KiB end windows
-    first, then a content hash that reads the whole file below 50 MiB
-    but only gigabyte-spaced 1 MiB samples at or above it, so the very
-    largest files are still never read in full. Scale target: tens of
+    first, and gets a content hash only when its size and both end
+    windows match another file's. That hash reads the whole file below
+    50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so the
+    very largest files are still never read in full. Scale target: tens of
     millions of files, ~150 TB filesystem, possibly slow or busy disks
     (ZFS pool under resilver). Holding one small record (path, size,
     mtime) per file in memory during a scan is acceptable; holding
     every file's hashes is not (they stay in the database).
 3.  **Scan incrementally, analyze offline.** The expensive filesystem
     scan maintains a persistent database; an unchanged file is never
-    read again on a rescan. All analysis (`report`, `trees`) works from
+    read again on a rescan, except to compute its content hash once a
+    file of 10 MiB or more comes to match another on size and both end
+    windows. All analysis (`report`, `trees`) works from
     the database alone and must never touch the scanned filesystem
     again. `scan` is designed to be cronned; the reports run at any
     time against the last completed scan.
@@ -176,17 +184,20 @@ All three subcommands operate on a single SQLite database file:
     the first- and last-64 KiB hashes and `content` the whole-file or
     sampled hash. All three are empty strings when the file has never
     been hashed because its size was unique as of the last scan that
-    covered it; such records still define the file for tree
-    reconstruction but never participate in duplicate groups.
+    covered it. For a file of 10 MiB or more, `content` stays empty
+    until the content phase of a scan (see "`scan` mode" below) has
+    read the file. A record with an empty `content` is never part of a
+    duplicate group, though it still defines the file for tree
+    reconstruction.
 
 ### Duplicate detection
 
 Two files are duplicates only when they agree on every rung of this
 ladder; a mismatch at any rung means they are not duplicates. `scan`
-stores each file's hashes once, and `report` and `trees` group files by
-the whole signature — size, `head`, `tail`, and `content` — so the
-grouping is exactly this ladder applied across everything scanned into
-the database, even across separate scans.
+stores each file's hashes, and `report` and `trees` group files by the
+whole signature — size, `head`, `tail`, and `content` — so the grouping
+is exactly this ladder applied across everything scanned into the
+database, even across separate scans.
 
 1.  **Size.** Files of different sizes are never compared. Only files
     whose size at least one other file shares are hashed at all.
@@ -199,7 +210,13 @@ the database, even across separate scans.
 3.  **10 MiB and above: head and tail.** For a larger file, the SHA-256
     of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
     cheap gate that eliminates most same-size pairs before any bulk
-    reading. At 10 MiB and above the two windows never overlap.
+    reading: the content hash of the next two rungs is computed only for
+    a file whose size, `head`, and `tail` match another file's, whether
+    that file is scanned in the same run or stored by an earlier scan.
+    A stored file that first gains such a match in a later scan gets its
+    content hash then; until it has one, its `content` is empty and it
+    is not a duplicate. At 10 MiB and above the two windows never
+    overlap.
 4.  **10 MiB and above, content below 50 MiB.** The SHA-256 of the
     entire file. Agreement here is proof of identical content (barring a
     SHA-256 collision).
@@ -239,10 +256,10 @@ scanned operands:
 
 -   Only a file whose size at least one other file shares is ever
     read: a size-unique file cannot be a duplicate, so it is recorded
-    without hashes (`head` and `tail` empty). The size census covers
-    every file walked this scan plus every database record outside
-    the scanned operands, so a possible duplicate of a separately
-    scanned tree is still recognized.
+    without hashes (`head`, `tail`, and `content` empty). The size
+    census covers every file walked this scan plus every database
+    record outside the scanned operands, so a possible duplicate of a
+    separately scanned tree is still recognized.
 -   A file not yet in the database is inserted: hashed when its size
     is shared, without hashes otherwise.
 -   A file already in the database is **skipped without reading its
@@ -251,7 +268,10 @@ scanned operands:
     makes a daily rescan cheap. Exception: an unchanged file whose
     record lacks hashes is hashed — and its record updated — once its
     size becomes shared, so hashing deferred by size-uniqueness
-    happens as soon as it could matter.
+    happens as soon as it could matter. Likewise, an unchanged file of
+    10 MiB or more whose record has no `content` hash is read for one
+    by the content phase below once its size, `head`, and `tail` match
+    another record's.
 -   A file whose mtime is newer than recorded, or whose size differs,
     is processed as if new: re-hashed, or recorded without hashes,
     per the shared-size rule.
@@ -260,12 +280,18 @@ scanned operands:
     removes records for deleted files. It also removes records for
     paths that failed to stat or hash this run: the database only ever
     contains signatures verified by the most recent scan that covered
-    them (a subsequent successful scan re-adds such files).
+    them (a subsequent successful scan re-adds such files). A failure
+    in the content phase below removes nothing: the record is left as
+    it is.
 -   Database records outside the scanned operands are untouched, so
     disjoint trees can be scanned on different schedules into the same
-    database.
+    database. The one exception is the content phase below: a stored
+    file of 10 MiB or more without a `content` hash is read for one,
+    wherever it lies, once its size, `head`, and `tail` match another
+    record's. If that file is gone or has changed since its record was
+    written, the record is left as it is.
 
-`scan` runs **three sequential phases over the whole scan**.
+`scan` runs **four sequential phases over the whole scan**.
 Parallelism lives inside each phase; batched database writes begin
 during the hash phase:
 
@@ -285,11 +311,13 @@ during the hash phase:
     decides its fate. Size-unique files are never read: new or
     changed ones are recorded without hashes in the update phase,
     unchanged unhashed ones simply keep their records. Every file
-    with a shared size is hashed by the worker pool, computing the
-    full signature — head, tail, and content — described in "Duplicate
-    detection" below. Zero-length files have constant
-    hashes and are never opened. Files are hashed in **inode order**
-    (minimizing seeks on spinning disks), and paths that are hard
+    with a shared size is hashed by the worker pool as described in
+    "Duplicate detection" above: a file under 10 MiB in full, which
+    gives its `head`, `tail`, and `content` alike, and a larger file
+    only in its end windows, which give its `head` and `tail`; its
+    content hash is left to the content phase. Zero-length files have
+    constant hashes and are never opened. Files are hashed in **inode
+    order** (minimizing seeks on spinning disks), and paths that are hard
     links to the same inode are **read once**, all sharing the one
     result — a hard-link backup farm costs one read per inode, not
     per path. The phase total counts actual reads, so progress and
@@ -301,6 +329,29 @@ during the hash phase:
     records for size-unique new and changed files, and the deletions
     for records the scan did not verify (vanished files, plus paths
     that failed to stat or hash).
+4.  **content** — find every record of 10 MiB or more without a
+    `content` hash whose size, `head`, and `tail` equal another
+    record's, anywhere in the database: records from this scan and
+    records stored by earlier scans, inside or outside the scanned
+    operands. SQLite finds them, so only the records to be read are
+    kept in memory, never every file's hashes. Every record sharing
+    their size, `head`, and `tail`, including one that already has a
+    `content` hash, has its file checked with `lstat` first. A file
+    that is gone, is no longer a regular file, or has changed (a
+    different size, or an mtime newer than recorded) keeps its record
+    as it is and does not count as a match for the others. Any other
+    `lstat` error is warned about and counted as skipped, with the same
+    result. If such a record has no `content` hash, it stays out of
+    duplicate groups; if it has one, it is still reported until a scan
+    covering its own tree updates or removes it. The files that pass
+    and have no `content` hash are read only if at least two of those
+    records pass, so a file whose only matches are stale costs no read;
+    a file that already has a `content` hash is never read again.
+    They are read by a worker pool as in the hash phase, in inode order
+    and once per inode, and their content hashes are committed in
+    batches. A failed read is warned about and counted as skipped; its
+    record keeps an empty `content`, so it is not a duplicate, and a
+    later scan tries again.
 
 Rules for the walk:
 
@@ -318,18 +369,18 @@ Rules for the walk:
     path, and continue. Per-file errors never abort the run; the final
     summary reports how many were skipped. As specified above, a
     skipped path that has a database record from an earlier scan loses
-    that record; an unreadable directory subtree likewise loses its
-    records (accepted: the database mirrors what the latest scan could
-    actually verify).
+    that record, unless it failed only in the content phase; an
+    unreadable directory subtree likewise loses its records (accepted:
+    the database mirrors what the latest scan could actually verify).
 
-Concurrency: the walk phase (which also stats files) and the hash
-phase each use a worker pool of `--workers` workers (default
-`runtime.NumCPU()`); the walk parallelizes across directories,
-hashing across files. Both phases are seek-bound on spinning disks,
-so raising `--workers` well past the core count can help on pools
-with many spindles. The main goroutine owns partitioning, database
-writes, and progress rendering; progress display must never block
-the workers.
+Concurrency: the walk phase (which also stats files), the hash phase,
+and the content phase each use a worker pool of `--workers` workers
+(default `runtime.NumCPU()`); the walk parallelizes across
+directories, hashing across files. All three phases are seek-bound on
+spinning disks, so raising `--workers` well past the core count can
+help on pools with many spindles. The main goroutine owns
+partitioning, database writes, and progress rendering; progress
+display must never block the workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the
 files seen this run broken down by disposition, plus skips:
@@ -354,11 +405,11 @@ mounted.
 
 Processing:
 
--   Records without hashes (size-unique when last scanned) are
+-   Records without a `content` hash (see "Database" above) are
     excluded: their content is unknown, so they are never reported as
     duplicates.
 -   Group the remaining records by the key
-    `(size, head_hash, tail_hash)`.
+    `(size, head, tail, content)`.
 -   Every group with two or more paths is a duplicate group.
 -   Within each group, sort paths lexicographically (byte order). The
     first path is the group's `first`; every other path is a `dupe`.
@@ -394,11 +445,11 @@ the paths in the records, split on `/`.
 
 Definitions:
 
--   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
-    informational and excluded. An unhashed record (empty hashes) has
+-   A file's **signature** is `(size, head, tail, content)` — mtime is
+    informational and excluded. A record without a `content` hash has
     unknown content: its signature is treated as unique to that file,
-    so a tree containing an unhashed file never compares equal to any
-    other tree.
+    so a tree containing such a file never compares equal to any other
+    tree.
 -   A directory's **digest** is a SHA-256 Merkle digest computed
     bottom-up: serialize the directory's child entries — for a file
     child, its name and signature; for a subdirectory child, its name
@@ -461,10 +512,13 @@ Each phase gets its own display, rendered the moment the phase
 starts — a scan must never look hung. Loading the existing-record
 index (`load`) and the walk have no known totals while running: show
 a live count, rate, and elapsed time (spinner-style, no percentage or
-ETA). The hash and update phases
-have exact totals — only files that actually need hashing appear in
-the hash total, so its ETA is meaningful. Required elements for the
-bars with known totals:
+ETA). The content phase's display (`content`) starts the same way,
+counting the records checked while SQLite finds the files to read and
+`lstat` checks them, then shows a bar once reading starts. The hash
+and update phases, and the content phase's reads, have exact totals —
+only files that actually need hashing appear in the hash and content
+totals, so their ETAs are meaningful. Required elements for the bars
+with known totals:
 
 -   elapsed time
 -   estimated time remaining
@@ -691,9 +745,10 @@ Tracked in [TODO.md](TODO.md).
 
 ## Non-goals
 
--   No full-content verification, no byte-for-byte compare, no deletion
-    or linking of duplicates. The reports are advisory; acting on them is
-    the user's job.
+-   No byte-for-byte compare, and no deletion or linking of
+    duplicates. Files that match are compared by a SHA-256 of the whole
+    file below 50 MiB, and only by samples at 50 MiB and over. The
+    reports are advisory; acting on them is the user's job.
 -   No persistence beyond the SQLite database described above; no
     export/import formats.
 -   No daemon or filesystem watcher; scheduling rescans is cron's job.