Check every group member and show the content phase from its start
check / check (push) Successful in 57s
check / check (push) Successful in 57s
The content phase now checks every record sharing a size, head and tail with lstat, including those that already have a content hash, and reads those without one only when at least two pass. Only a missing file is passed over silently; any other lstat error is warned about and counted as skipped. The phase shows a running count while it queries and checks, then its bar. The README states once when content is empty. New tests cover these and hard links. Model: opus-5-5
This commit is contained in:
@@ -184,11 +184,11 @@ All three subcommands operate on a single SQLite database file:
|
||||
the first- and last-64 KiB hashes and `content` the whole-file or
|
||||
sampled hash. All three are empty strings when the file has never
|
||||
been hashed because its size was unique as of the last scan that
|
||||
covered it. `content` alone is empty for a file of 10 MiB or more
|
||||
that matches no other record on size, `head`, and `tail` yet, or
|
||||
whose content read failed. A record with an empty `content` still
|
||||
defines the file for tree reconstruction but is not a duplicate
|
||||
until a later scan fills it in.
|
||||
covered it. For a file of 10 MiB or more, `content` stays empty
|
||||
until the content phase of a scan (see "`scan` mode" below) has
|
||||
read the file. A record with an empty `content` is never part of a
|
||||
duplicate group, though it still defines the file for tree
|
||||
reconstruction.
|
||||
|
||||
### Duplicate detection
|
||||
|
||||
@@ -280,9 +280,9 @@ scanned operands:
|
||||
removes records for deleted files. It also removes records for
|
||||
paths that failed to stat or hash this run: the database only ever
|
||||
contains signatures verified by the most recent scan that covered
|
||||
them (a subsequent successful scan re-adds such files). A failed
|
||||
content read in the content phase below removes nothing: the
|
||||
record keeps its `head` and `tail`, with `content` empty.
|
||||
them (a subsequent successful scan re-adds such files). A failure
|
||||
in the content phase below removes nothing: the record is left as
|
||||
it is.
|
||||
- Database records outside the scanned operands are untouched, so
|
||||
disjoint trees can be scanned on different schedules into the same
|
||||
database. The one exception is the content phase below: a stored
|
||||
@@ -333,16 +333,19 @@ during the hash phase:
|
||||
`content` hash whose size, `head`, and `tail` equal another
|
||||
record's, anywhere in the database: records from this scan and
|
||||
records stored by earlier scans, inside or outside the scanned
|
||||
operands. SQLite finds them, so only their records are loaded into
|
||||
memory, never every file's hashes. Each such file is checked with
|
||||
`lstat` first; one that is gone, is no longer a regular file, or
|
||||
has changed (a different size, or an mtime newer than recorded)
|
||||
keeps its record as it is and is not a duplicate. The files that
|
||||
pass are read only if at least two records sharing their size,
|
||||
`head`, and `tail` remain, counting those that already have a
|
||||
`content` hash, so a file whose only matches are stale costs no
|
||||
read. They are read by a worker pool as in the hash phase, in inode
|
||||
order and once per inode, and their content hashes are committed in
|
||||
operands. SQLite finds them, so only the records to be read are
|
||||
kept in memory, never every file's hashes. Every record sharing
|
||||
their size, `head`, and `tail`, including one that already has a
|
||||
`content` hash, has its file checked with `lstat` first. A file
|
||||
that is gone, is no longer a regular file, or has changed (a
|
||||
different size, or an mtime newer than recorded) keeps its record
|
||||
as it is and is not a duplicate. Any other `lstat` error is warned
|
||||
about and counted as skipped, with the same result. The files that
|
||||
pass and have no `content` hash are read only if at least two of
|
||||
those records pass, so a file whose only matches are stale costs no
|
||||
read; a file that already has a `content` hash is never read again.
|
||||
They are read by a worker pool as in the hash phase, in inode order
|
||||
and once per inode, and their content hashes are committed in
|
||||
batches. A failed read is warned about and counted as skipped; its
|
||||
record keeps an empty `content`, so it is not a duplicate, and a
|
||||
later scan tries again.
|
||||
@@ -363,9 +366,9 @@ Rules for the walk:
|
||||
path, and continue. Per-file errors never abort the run; the final
|
||||
summary reports how many were skipped. As specified above, a
|
||||
skipped path that has a database record from an earlier scan loses
|
||||
that record, unless only its content read failed; an unreadable
|
||||
directory subtree likewise loses its records (accepted: the
|
||||
database mirrors what the latest scan could actually verify).
|
||||
that record, unless it failed only in the content phase; an
|
||||
unreadable directory subtree likewise loses its records (accepted:
|
||||
the database mirrors what the latest scan could actually verify).
|
||||
|
||||
Concurrency: the walk phase (which also stats files), the hash phase,
|
||||
and the content phase each use a worker pool of `--workers` workers
|
||||
@@ -399,10 +402,9 @@ mounted.
|
||||
|
||||
Processing:
|
||||
|
||||
- Records without a `content` hash (size-unique when last scanned,
|
||||
or 10 MiB or more and not yet matched on size, `head`, and `tail`)
|
||||
are excluded: their content is unknown, so they are never reported
|
||||
as duplicates.
|
||||
- Records without a `content` hash (see "Database" above) are
|
||||
excluded: their content is unknown, so they are never reported as
|
||||
duplicates.
|
||||
- Group the remaining records by the key
|
||||
`(size, head, tail, content)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
@@ -507,10 +509,13 @@ Each phase gets its own display, rendered the moment the phase
|
||||
starts — a scan must never look hung. Loading the existing-record
|
||||
index (`load`) and the walk have no known totals while running: show
|
||||
a live count, rate, and elapsed time (spinner-style, no percentage or
|
||||
ETA). The hash, update, and content phases
|
||||
have exact totals — only files that actually need hashing appear in
|
||||
the hash and content totals, so their ETAs are meaningful. Required
|
||||
elements for the bars with known totals:
|
||||
ETA). The content phase's display (`content`) starts the same way,
|
||||
counting the records checked while SQLite finds the files to read and
|
||||
`lstat` checks them, then shows a bar once reading starts. The hash
|
||||
and update phases, and the content phase's reads, have exact totals —
|
||||
only files that actually need hashing appear in the hash and content
|
||||
totals, so their ETAs are meaningful. Required elements for the bars
|
||||
with known totals:
|
||||
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
|
||||
Reference in New Issue
Block a user