1 Commits
Author SHA1 Message Date
clawbot b2af329e3c Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s
Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.

Judgement call: info drops its two hash-count lines, which now always
equal the record count.

Model: opus-5-5
2026-10-26 00:33:38 +00:00
+12 -12
View File
@@ -272,7 +272,7 @@ Four subcommands, all implemented:
from the database records, compute a Merkle-style digest per directory, and from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees. report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its 4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records. schema version, the number of its records and their total size.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
@@ -427,8 +427,8 @@ The content hash depends on the file's size:
of identical content, barring a SHA-256 collision. of identical content, barring a SHA-256 collision.
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read - At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2 in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
GiB and so on. The samples are fed into one hash in order, and the last one GiB and so on. The samples are fed into one hash in order. A sample that would
stops at the end of the file. run past the end of the file stops there.
The sampled hash is **deliberately probabilistic**. Bytes outside the samples The sampled hash is **deliberately probabilistic**. Bytes outside the samples
and outside the first and last 64 KiB are never read. Two large files that and outside the first and last 64 KiB are never read. Two large files that
@@ -912,15 +912,15 @@ Additional requirements:
terminal is. A line too wide for the terminal first loses its bar, which gives terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, a stalled line keeps its and the ETA are kept whole. On an 80-column terminal, at counts like those in
words whenever the elapsed time and the ETA are both in minutes. It holds at the example above, a stalled line keeps its words whenever the elapsed time
counts like the example above. Where the words and the ETA do not both fit, as and the ETA are both in minutes. Where the words and the ETA do not both fit,
at millions of files and times in hours, the words give way. Only when those as at millions of files and times in hours, the words give way. Only when
four alone are too wide, as on a terminal narrower than about 70 columns hours those four alone are too wide, as on a terminal narrower than about 70 columns
into a scan of millions of files, is the line cut short at the terminal's hours into a scan of millions of files, is the line cut short at the
edge, so the end of the ETA is lost first. A terminal made narrower while a terminal's edge, so the end of the ETA is lost first. A terminal made narrower
phase runs gets the same: the next draw erases the line within the new width while a phase runs gets the same: the next draw erases the line within the new
and fits the line to it, and the display goes on redrawing in place. width and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once runs (whether or not an item has completed, and never more often), and once