Hash in inode order, read hard links once, never open empty files

Sort the hash queue by (device, inode) so reads proceed in inode
order, which minimizes seeking on spinning disks. Paths that are hard
links to the same inode form one run: the run is read once and every
path shares the result, so link farms (rsync --link-dest backups)
cost one read per inode instead of one per path. A run that fails to
read skips all of its paths.

Zero-length files have constant head/tail hashes; return them without
opening the file.

The hash progress total now counts actual reads (runs, not paths).
Hard-linked paths still appear in reports as duplicates — their
content is identical — though they share storage; noted in README.
This commit is contained in:
2026-07-25 14:36:55 +07:00
parent 67bde6226d
commit b14b735c88
3 changed files with 247 additions and 57 deletions

View File

@@ -234,10 +234,14 @@ during the hash phase:
unchanged unhashed ones simply keep their records. Every file
with a shared size is hashed by the worker pool: read the first
`min(1024, size)` bytes and the last `min(1024, size)` bytes
(one read when `size <= 1024`, since the two windows coincide;
for `size == 0` hash the empty input) and compute the SHA-256 of
each. The phase total is exact, so progress and ETA are
meaningful. Completed records are committed in batched
(one read when `size <= 1024`, since the two windows coincide)
and compute the SHA-256 of each. Zero-length files have constant
hashes and are never opened. Files are hashed in **inode order**
(minimizing seeks on spinning disks), and paths that are hard
links to the same inode are **read once**, all sharing the one
result — a hard-link backup farm costs one read per inode, not
per path. The phase total counts actual reads, so progress and
ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**, so a scan interrupted after
hours keeps everything hashed so far and the next scan resumes
cheaply, skipping records already written.
@@ -354,6 +358,12 @@ Definitions:
equal. Equal digests imply equal recursive file count and equal
total byte size.
Known limitation (accepted): hard-linked paths are reported as
duplicates by `report` and count toward duplicate trees — their
content is genuinely identical — even though they share storage, so
removing one reclaims no space. Inode identity is used during the
scan to avoid redundant reads but is not persisted in the database.
Known limitation (accepted): only regular files that appear in the
database define a tree. Empty directories are invisible, and a file
skipped during the scan (e.g. permission error) in one copy but not the