Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s

Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.

Judgement call: info drops its two hash-count lines, which now always
equal the record count.

Model: opus-5-5
This commit was merged in pull request #136.
This commit is contained in:
2026-10-26 02:02:48 +01:00
parent f4ddb58ae1
commit f7a746f912
16 changed files with 537 additions and 1774 deletions
+2 -7
View File
@@ -283,7 +283,6 @@ func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/1"},
scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/2"},
scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/1"},
scanRec{size: 50, path: "/x/unhashed"},
)
reversed := slices.Clone(recs)
@@ -419,16 +418,12 @@ func TestDupeGroups(t *testing.T) {
func TestDupeGroupsContentSeparates(t *testing.T) {
t.Parallel()
// Same size, head, and tail, but different content hashes: the final
// rung keeps them apart, so no group forms. Matching content groups.
// Records without a content hash never group, not even with each
// other.
// Same size, head, and tail: /b differs from the others only in its
// content hash, so it joins no group, while /a and /c group.
recs := []scanRec{
{size: 100, head: "h", tail: "t", content: "c1", path: "/a"},
{size: 100, head: "h", tail: "t", content: "c2", path: "/b"},
{size: 100, head: "h", tail: "t", content: "c1", path: "/c"},
{size: 100, head: "h", tail: "t", path: "/d"},
{size: 100, head: "h", tail: "t", path: "/e"},
}
groups := dupeGroupsOf(t, recs)