1 Commits
Author SHA1 Message Date
clawbot b2af329e3c Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s
Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.

Judgement call: info drops its two hash-count lines, which now always
equal the record count.

Model: opus-5-5
2026-10-26 00:33:38 +00:00
+12 -12
View File
@@ -272,7 +272,7 @@ Four subcommands, all implemented:
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records.
schema version, the number of its records and their total size.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
@@ -427,8 +427,8 @@ The content hash depends on the file's size:
of identical content, barring a SHA-256 collision.
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
GiB and so on. The samples are fed into one hash in order, and the last one
stops at the end of the file.
GiB and so on. The samples are fed into one hash in order. A sample that would
run past the end of the file stops there.
The sampled hash is **deliberately probabilistic**. Bytes outside the samples
and outside the first and last 64 KiB are never read. Two large files that
@@ -912,15 +912,15 @@ Additional requirements:
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, a stalled line keeps its
words whenever the elapsed time and the ETA are both in minutes. It holds at
counts like the example above. Where the words and the ETA do not both fit, as
at millions of files and times in hours, the words give way. Only when those
four alone are too wide, as on a terminal narrower than about 70 columns hours
into a scan of millions of files, is the line cut short at the terminal's
edge, so the end of the ETA is lost first. A terminal made narrower while a
phase runs gets the same: the next draw erases the line within the new width
and fits the line to it, and the display goes on redrawing in place.
and the ETA are kept whole. On an 80-column terminal, at counts like those in
the example above, a stalled line keeps its words whenever the elapsed time
and the ETA are both in minutes. Where the words and the ETA do not both fit,
as at millions of files and times in hours, the words give way. Only when
those four alone are too wide, as on a terminal narrower than about 70 columns
hours into a scan of millions of files, is the line cut short at the
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
while a phase runs gets the same: the next draw erases the line within the new
width and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once