Hash in inode order, read hard links once, never open empty files
Sort the hash queue by (device, inode) so reads proceed in inode order, which minimizes seeking on spinning disks. Paths that are hard links to the same inode form one run: the run is read once and every path shares the result, so link farms (rsync --link-dest backups) cost one read per inode instead of one per path. A run that fails to read skips all of its paths. Zero-length files have constant head/tail hashes; return them without opening the file. The hash progress total now counts actual reads (runs, not paths). Hard-linked paths still appear in reports as duplicates — their content is identical — though they share storage; noted in README.
This commit is contained in:
18
README.md
18
README.md
@@ -234,10 +234,14 @@ during the hash phase:
|
||||
unchanged unhashed ones simply keep their records. Every file
|
||||
with a shared size is hashed by the worker pool: read the first
|
||||
`min(1024, size)` bytes and the last `min(1024, size)` bytes
|
||||
(one read when `size <= 1024`, since the two windows coincide;
|
||||
for `size == 0` hash the empty input) and compute the SHA-256 of
|
||||
each. The phase total is exact, so progress and ETA are
|
||||
meaningful. Completed records are committed in batched
|
||||
(one read when `size <= 1024`, since the two windows coincide)
|
||||
and compute the SHA-256 of each. Zero-length files have constant
|
||||
hashes and are never opened. Files are hashed in **inode order**
|
||||
(minimizing seeks on spinning disks), and paths that are hard
|
||||
links to the same inode are **read once**, all sharing the one
|
||||
result — a hard-link backup farm costs one read per inode, not
|
||||
per path. The phase total counts actual reads, so progress and
|
||||
ETA are meaningful. Completed records are committed in batched
|
||||
transactions **while hashing runs**, so a scan interrupted after
|
||||
hours keeps everything hashed so far and the next scan resumes
|
||||
cheaply, skipping records already written.
|
||||
@@ -354,6 +358,12 @@ Definitions:
|
||||
equal. Equal digests imply equal recursive file count and equal
|
||||
total byte size.
|
||||
|
||||
Known limitation (accepted): hard-linked paths are reported as
|
||||
duplicates by `report` and count toward duplicate trees — their
|
||||
content is genuinely identical — even though they share storage, so
|
||||
removing one reclaims no space. Inode identity is used during the
|
||||
scan to avoid redundant reads but is not persisted in the database.
|
||||
|
||||
Known limitation (accepted): only regular files that appear in the
|
||||
database define a tree. Empty directories are invisible, and a file
|
||||
skipped during the scan (e.g. permission error) in one copy but not the
|
||||
|
||||
Reference in New Issue
Block a user