Compute the content hash only when head and tail match (closes #61)
check / check (push) Successful in 1m9s
check / check (push) Successful in 1m9s
A file of 10 MiB or more now gets only its head and tail in the hash phase. A new content phase, after the update phase, finds every record of that size without a content hash whose size, head and tail match another record's, anywhere in the database, checks each file with lstat, and reads a group only while at least two members remain. It reuses the hash worker pool, now given its hash function. report and trees leave out records without a content hash. The README, help text and TODO entry describe the gate; the schema stays at version 1. Model: opus-5-5
This commit is contained in:
@@ -34,11 +34,16 @@
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
|
||||
hashed in full and compared directly, with no end-window step — its
|
||||
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
||||
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
|
||||
compared on a `content` hash — the whole file below 50 MiB,
|
||||
gigabyte-spaced 1 MiB samples at or above. The `content` column is
|
||||
part of the version 1 schema. `report` and `trees` group by the
|
||||
extended signature, so the ladder is applied across the whole
|
||||
10 MiB or above gets only the 64 KiB `head` and `tail` in the hash
|
||||
phase; a new content phase, after the update phase, reads it for its
|
||||
`content` hash — the whole file below 50 MiB, gigabyte-spaced 1 MiB
|
||||
samples at or above — only when its size, `head`, and `tail` match
|
||||
another record's, from the same scan or stored by an earlier one, so
|
||||
a stored file gains its content hash when it gains a match. A file
|
||||
that is gone or has changed since its record was written is not
|
||||
read. The `content` column is part of the version 1 schema. `report`
|
||||
and `trees` group by the extended signature and leave out any record
|
||||
without a `content` hash, so the ladder is applied across the whole
|
||||
database. README "Duplicate detection" documents every rung including
|
||||
the probabilistic large-file path.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user