Check every group member and show the content phase from its start
check / check (push) Successful in 57s

The content phase now checks every record sharing a size, head and
tail with lstat, including those that already have a content hash, and
reads those without one only when at least two pass. Only a missing
file is passed over silently; any other lstat error is warned about and
counted as skipped. The phase shows a running count while it queries
and checks, then its bar. The README states once when content is empty.
New tests cover these and hard links.

Model: opus-5-5
This commit is contained in:
2026-09-23 13:09:03 +00:00
parent 89fc9e4595
commit fff4409a24
6 changed files with 243 additions and 81 deletions
+34 -29
View File
@@ -184,11 +184,11 @@ All three subcommands operate on a single SQLite database file:
the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never
been hashed because its size was unique as of the last scan that
covered it. `content` alone is empty for a file of 10 MiB or more
that matches no other record on size, `head`, and `tail` yet, or
whose content read failed. A record with an empty `content` still
defines the file for tree reconstruction but is not a duplicate
until a later scan fills it in.
covered it. For a file of 10 MiB or more, `content` stays empty
until the content phase of a scan (see "`scan` mode" below) has
read the file. A record with an empty `content` is never part of a
duplicate group, though it still defines the file for tree
reconstruction.
### Duplicate detection
@@ -280,9 +280,9 @@ scanned operands:
removes records for deleted files. It also removes records for
paths that failed to stat or hash this run: the database only ever
contains signatures verified by the most recent scan that covered
them (a subsequent successful scan re-adds such files). A failed
content read in the content phase below removes nothing: the
record keeps its `head` and `tail`, with `content` empty.
them (a subsequent successful scan re-adds such files). A failure
in the content phase below removes nothing: the record is left as
it is.
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database. The one exception is the content phase below: a stored
@@ -333,16 +333,19 @@ during the hash phase:
`content` hash whose size, `head`, and `tail` equal another
record's, anywhere in the database: records from this scan and
records stored by earlier scans, inside or outside the scanned
operands. SQLite finds them, so only their records are loaded into
memory, never every file's hashes. Each such file is checked with
`lstat` first; one that is gone, is no longer a regular file, or
has changed (a different size, or an mtime newer than recorded)
keeps its record as it is and is not a duplicate. The files that
pass are read only if at least two records sharing their size,
`head`, and `tail` remain, counting those that already have a
`content` hash, so a file whose only matches are stale costs no
read. They are read by a worker pool as in the hash phase, in inode
order and once per inode, and their content hashes are committed in
operands. SQLite finds them, so only the records to be read are
kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a
`content` hash, has its file checked with `lstat` first. A file
that is gone, is no longer a regular file, or has changed (a
different size, or an mtime newer than recorded) keeps its record
as it is and is not a duplicate. Any other `lstat` error is warned
about and counted as skipped, with the same result. The files that
pass and have no `content` hash are read only if at least two of
those records pass, so a file whose only matches are stale costs no
read; a file that already has a `content` hash is never read again.
They are read by a worker pool as in the hash phase, in inode order
and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its
record keeps an empty `content`, so it is not a duplicate, and a
later scan tries again.
@@ -363,9 +366,9 @@ Rules for the walk:
path, and continue. Per-file errors never abort the run; the final
summary reports how many were skipped. As specified above, a
skipped path that has a database record from an earlier scan loses
that record, unless only its content read failed; an unreadable
directory subtree likewise loses its records (accepted: the
database mirrors what the latest scan could actually verify).
that record, unless it failed only in the content phase; an
unreadable directory subtree likewise loses its records (accepted:
the database mirrors what the latest scan could actually verify).
Concurrency: the walk phase (which also stats files), the hash phase,
and the content phase each use a worker pool of `--workers` workers
@@ -399,10 +402,9 @@ mounted.
Processing:
- Records without a `content` hash (size-unique when last scanned,
or 10 MiB or more and not yet matched on size, `head`, and `tail`)
are excluded: their content is unknown, so they are never reported
as duplicates.
- Records without a `content` hash (see "Database" above) are
excluded: their content is unknown, so they are never reported as
duplicates.
- Group the remaining records by the key
`(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group.
@@ -507,10 +509,13 @@ Each phase gets its own display, rendered the moment the phase
starts — a scan must never look hung. Loading the existing-record
index (`load`) and the walk have no known totals while running: show
a live count, rate, and elapsed time (spinner-style, no percentage or
ETA). The hash, update, and content phases
have exact totals — only files that actually need hashing appear in
the hash and content totals, so their ETAs are meaningful. Required
elements for the bars with known totals:
ETA). The content phase's display (`content`) starts the same way,
counting the records checked while SQLite finds the files to read and
`lstat` checks them, then shows a bar once reading starts. The hash
and update phases, and the content phase's reads, have exact totals
only files that actually need hashing appear in the hash and content
totals, so their ETAs are meaningful. Required elements for the bars
with known totals:
- elapsed time
- estimated time remaining