sneak/sfdupes

next into main: the duplicate check, hash storage, display and .zfs fixes from the trial #137

Open

clawbot wants to merge 4 commits from next into main

Author	SHA1	Message	Date
clawbot	ee09cda355	Never scan an operand inside a .zfs directory (closes #142 ) check / check (push) Waiting to run Details An operand inside a directory named .zfs was walked, because the operand check looked only at the operand's own name. The check now also looks at every directory above the operand, both as given and with its symlinks resolved. Such an operand gets a warning, counts as skipped and is not walked, whether given alone or under another operand. The check runs before overlapping operands are pruned, so an operand under another one still gets its warning. The test for an operand named .zfs now writes its record beneath .zfs straight into the database, since no scan stores one there any more. Judgement call: a path component named .zfs counts whatever it is, a symlink included. Model: opus-5-5	2026-10-26 15:15:37 +01:00
clawbot	aa99b37b99	Skip removed paths silently and fix the progress line (closes #133 ) check / check (push) Canceled after 0s Details A path removed while the scan runs gets no warning and no skipped count, in the walk or the hash phase. If the first of several hard links is gone, the next is read. Other errors are still warned about. On a terminal every display is the library's spinner, then the line. A phase with an exact total ends the padded line with its percent and a bar at a fixed column, drawn by sfdupes, as the library shows no spinner beside its own bar. A too-wide line loses its rate first. The time remaining is labelled remain, not eta. Judgement call: an operand removed mid-scan is skipped silently too. Deviation: README's stalled-line example keeps its words on 100 columns, not 80. Model: opus-5-5	2026-10-26 06:32:38 +01:00
clawbot	68881f04f2	Store each hash as its raw 32 bytes in a BLOB column (closes #134 ) check / check (push) Canceled after 0s Details The head, tail and content columns of the files table are BLOB NOT NULL and hold the raw 32-byte SHA-256, half the size of the hex text, in the table and in the files_signature index. The hash phase, the records, report and trees carry each hash as a [sha256.Size]byte and compare it with ==; nothing is hex-encoded. Reading a record whose hash column is not 32 bytes is an error naming its path. In the tests, testHash is a stand-in hash: a string's bytes padded with zeros, so fixtures keep the sort order their strings give. Judgement call: a hash of the wrong length is refused on reading, not copied silently. Model: opus-5-5	2026-10-26 03:15:38 +01:00
clawbot	f7a746f912	Hash every file's head, tail and content in the hash phase (closes #132 ) check / check (push) Canceled after 0s Details Every file the walk records now gets three hashes: head and tail, of its first and last 64 KiB, and content. The content hash covers the whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up, so a large file is never read in full. One 100 MiB constant replaces the 10 MiB and 50 MiB thresholds. The size census, the records stored without hashes and the content phase are gone, so a rescan reads only new and changed files. report and trees no longer special-case a missing content hash. README "Duplicate detection" and "Rationale" are rewritten. Judgement call: info drops its two hash-count lines, which now always equal the record count. Model: opus-5-5	2026-10-26 02:02:48 +01:00