Record each file's size on disk and print it in trees (closes #147)
check / check (push) Waiting to run
check / check (push) Waiting to run
The files table gets a disk_size column: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes. When a rescan finds a file with its size and mtime unchanged but a different disk_size, it updates disk_size alone, without reading the file. trees adds disk_size up per directory and prints it as a column after size. On a compressed or sparse file it is less than size, so two copies of one tree can differ. Judgement calls: each row's disk_size is the dupe copy's, not the first's. A file whose record gets only a new disk_size counts as unchanged. Model: opus-5-5
This commit was merged in pull request #149.
This commit is contained in:
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
```
|
||||
|
||||
`scan` walks one or more filesystem trees and maintains one database record per
|
||||
regular file (path, size, mtime, head hash, tail hash, content hash). The
|
||||
database persists between runs. A rescan hashes only the files that are new or
|
||||
changed, and removes the records of files that no longer exist. `report` reads
|
||||
the database and prints the file-level duplicates report. `trees` reads the same
|
||||
database and prints the duplicate-tree report. `info` prints the database's
|
||||
details: its path, the sizes of its files and the number of its records. A
|
||||
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
|
||||
prints a usage message and exits 2.
|
||||
regular file (path, size, size on disk, mtime, head hash, tail hash, content
|
||||
hash). The database persists between runs. A rescan hashes only the files that
|
||||
are new or changed, and removes the records of files that no longer exist.
|
||||
`report` reads the database and prints the file-level duplicates report. `trees`
|
||||
reads the same database and prints the duplicate-tree report. `info` prints the
|
||||
database's details: its path, the sizes of its files and the number of its
|
||||
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||||
operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
@@ -226,12 +226,12 @@ Goals, in order:
|
||||
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
|
||||
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
|
||||
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
|
||||
under resilver). Holding one small record (path, size, mtime) per file in
|
||||
memory during a scan is acceptable; holding every file's hashes is not (they
|
||||
stay in the database). The reporting commands do not hold every file's
|
||||
hashes either: `report` reads the records in signature order and holds the
|
||||
duplicate paths of one size at a time, so its memory grows with the
|
||||
duplicates of the size that has the most of them, and `trees` reads the
|
||||
under resilver). Holding one small record (path, size, size on disk, mtime)
|
||||
per file in memory during a scan is acceptable; holding every file's hashes
|
||||
is not (they stay in the database). The reporting commands do not hold every
|
||||
file's hashes either: `report` reads the records in signature order and
|
||||
holds the duplicate paths of one size at a time, so its memory grows with
|
||||
the duplicates of the size that has the most of them, and `trees` reads the
|
||||
records in path order and keeps each directory's path, digest and totals,
|
||||
plus the hashes of only the files in the directories holding the record
|
||||
being read, so its memory grows with the number of directories and with the
|
||||
@@ -368,7 +368,8 @@ All four subcommands operate on a single SQLite database file:
|
||||
- While it loads the existing records (`load`) and walks: no records. A
|
||||
first scan's database holds none, however long its walk has run.
|
||||
- During the hash phase: the first records, committed 10,000 at a time.
|
||||
- In the update phase: the rest of the hashed records, and the deletions.
|
||||
- In the update phase: the rest of the hashed records, the new `disk_size`
|
||||
of unchanged files whose size on disk changed, and the deletions.
|
||||
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
||||
away, unless a report still has the database open (see above).
|
||||
- `report`, `trees` and `info` open the database read-only and need only read
|
||||
@@ -386,6 +387,7 @@ All four subcommands operate on a single SQLite database file:
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||||
size INTEGER NOT NULL, -- bytes, from lstat
|
||||
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
|
||||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||||
@@ -514,6 +516,11 @@ operands:
|
||||
- A file already in the database is **skipped without reading its contents**
|
||||
when its lstat size equals the recorded size and its lstat mtime is not newer
|
||||
than the recorded mtime. This is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from the recorded `disk_size`, the
|
||||
scan updates `disk_size` alone. It still does not read the file, and counts it
|
||||
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
|
||||
the write. A file scanned within those seconds gets its real `disk_size` at
|
||||
the next scan.
|
||||
- A file whose mtime is newer than recorded, or whose size differs, is hashed
|
||||
again. Change detection compares the mtime to the nanosecond, as finely as the
|
||||
filesystem records it, so a same-size rewrite counts as a change whenever the
|
||||
@@ -541,8 +548,8 @@ inside the walk and hash phases, and batched database writes begin during the
|
||||
hash phase:
|
||||
|
||||
1. **load** — read the existing records. Each record under the operands is kept
|
||||
in memory, by path, with its size and its mtime, for the walk to compare
|
||||
each file with.
|
||||
in memory, by path, with its size, its size on disk and its mtime, for the
|
||||
walk to compare each file with.
|
||||
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||||
with the walk worker pool: every operand seeds the shared queue, and each
|
||||
worker reads one directory at a time, handing discovered subdirectories back
|
||||
@@ -550,9 +557,10 @@ hash phase:
|
||||
(while the directory's metadata is still hot). Sequential directory
|
||||
enumeration is metadata-latency-bound and takes hours at tens of millions of
|
||||
files; per-directory parallelism is what makes the walk tractable on large
|
||||
or busy pools. The walk compares each file with its record as it goes, and
|
||||
an unchanged file needs nothing more. Every new or changed file is carried
|
||||
to the hash phase as a (path, size, mtime) record.
|
||||
or busy pools. The walk compares each file with its record as it goes. An
|
||||
unchanged file is not read again. When its size on disk has changed, the
|
||||
update phase writes its new `disk_size`. Every new or changed file is
|
||||
carried to the hash phase as a (path, size, size on disk, mtime) record.
|
||||
3. **hash** — the worker pool hashes every carried file as described in
|
||||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||||
`content`. Zero-length files have constant hashes and are never opened.
|
||||
@@ -564,7 +572,8 @@ hash phase:
|
||||
committed in batched transactions **while hashing runs**. A scan interrupted
|
||||
after hours therefore keeps everything hashed so far, and the next scan
|
||||
resumes cheaply, skipping records already written.
|
||||
4. **update** — commit the final partial batch of hashed records. Then delete
|
||||
4. **update** — commit the final partial batch of hashed records, and the new
|
||||
`disk_size` of each unchanged file whose size on disk changed. Then delete
|
||||
the records the scan did not verify: those of vanished files, and of paths
|
||||
that failed to stat or hash.
|
||||
|
||||
@@ -809,14 +818,26 @@ Processing:
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
group of N). `files` is the recursive regular-file count of one copy of the
|
||||
tree; `size` is the recursive total byte size of one copy:
|
||||
group of N). After the `first` and `dupe` paths come three numbers:
|
||||
|
||||
- `files`, the number of regular files in one copy of the tree, counted
|
||||
recursively;
|
||||
- `size`, the bytes of one copy, its files' sizes added up;
|
||||
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||||
added up.
|
||||
|
||||
```
|
||||
first dupe files size
|
||||
/srv/a/project /srv/backup/project 3417 104857600
|
||||
first dupe files size disk_size
|
||||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||
```
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It can be more than
|
||||
the file's size, because space is allocated in whole blocks. It is less than the
|
||||
file's size on a compressed or sparse file, so two copies of one tree can differ
|
||||
in `disk_size`. When a file in one copy is a hard link to one in the other, both
|
||||
copies count its blocks in full. Removing it from one copy then frees nothing,
|
||||
as the known limitation under "`trees` mode" says.
|
||||
|
||||
The `first` and `dupe` paths are escaped as described under "Report output
|
||||
format". The root directory's path is `/`.
|
||||
|
||||
@@ -1200,9 +1221,9 @@ All of the following, run in this directory, must pass:
|
||||
ordered by size descending.
|
||||
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||||
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
||||
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
||||
differs by name).
|
||||
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
|
||||
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
|
||||
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
|
||||
|
||||
Expected from the second `report` (after the modify/delete rescan):
|
||||
`one.bin` has left its group (its content changed), so
|
||||
|
||||
Reference in New Issue
Block a user