1 Commits
Author SHA1 Message Date
clawbot 62b9cfc7ae Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s
Every file the walk records now gets three hashes in the hash phase:
head and tail, of its first and last 64 KiB, and content. The content
hash covers the whole file under 100 MiB, and 1 MiB at each gigabyte
from 100 MiB up, so a large file is never read in full. One 100 MiB
constant replaces the 10 MiB and 50 MiB thresholds. The size census,
the records stored without hashes and the content phase are gone, so a
scan has four phases and a rescan reads only new and changed files.
report and trees no longer special-case a missing content hash. README
"Duplicate detection" and "Rationale" are rewritten.

Judgement call: info drops its two hash-count lines, which would now
always equal the record count.

Model: opus-5-5
2026-10-25 23:37:09 +00:00
+12 -12
View File
@@ -272,7 +272,7 @@ Four subcommands, all implemented:
from the database records, compute a Merkle-style digest per directory, and from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees. report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its 4. `info` — the database's details: its path, the sizes of its files, its
schema version, the number of its records and their total size. schema version and the counts of its records.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
@@ -427,8 +427,8 @@ The content hash depends on the file's size:
of identical content, barring a SHA-256 collision. of identical content, barring a SHA-256 collision.
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read - At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2 in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
GiB and so on. The samples are fed into one hash in order. A sample that would GiB and so on. The samples are fed into one hash in order, and the last one
run past the end of the file stops there. stops at the end of the file.
The sampled hash is **deliberately probabilistic**. Bytes outside the samples The sampled hash is **deliberately probabilistic**. Bytes outside the samples
and outside the first and last 64 KiB are never read. Two large files that and outside the first and last 64 KiB are never read. Two large files that
@@ -912,15 +912,15 @@ Additional requirements:
terminal is. A line too wide for the terminal first loses its bar, which gives terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those in and the ETA are kept whole. On an 80-column terminal, a stalled line keeps its
the example above, a stalled line keeps its words whenever the elapsed time words whenever the elapsed time and the ETA are both in minutes. It holds at
and the ETA are both in minutes. Where the words and the ETA do not both fit, counts like the example above. Where the words and the ETA do not both fit, as
as at millions of files and times in hours, the words give way. Only when at millions of files and times in hours, the words give way. Only when those
those four alone are too wide, as on a terminal narrower than about 70 columns four alone are too wide, as on a terminal narrower than about 70 columns hours
hours into a scan of millions of files, is the line cut short at the into a scan of millions of files, is the line cut short at the terminal's
terminal's edge, so the end of the ETA is lost first. A terminal made narrower edge, so the end of the ETA is lost first. A terminal made narrower while a
while a phase runs gets the same: the next draw erases the line within the new phase runs gets the same: the next draw erases the line within the new width
width and fits the line to it, and the display goes on redrawing in place. and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once runs (whether or not an item has completed, and never more often), and once