Record each file's size on disk and print it in trees (closes #147)
check / check (push) Waiting to run

The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. When a rescan finds a file with its size
and mtime unchanged but a different disk_size, it updates disk_size
alone, without reading the file. trees adds disk_size up per directory
and prints it as a column after size. On a compressed or sparse file it
is less than size, so two copies of one tree can differ.

Judgement calls: each row's disk_size is the dupe copy's, not the
first's. A file whose record gets only a new disk_size counts as
unchanged.

Model: opus-5-5
This commit was merged in pull request #149.
This commit is contained in:
2026-10-27 19:56:49 +01:00
parent 98193b2896
commit 4ed2e370b1
10 changed files with 452 additions and 145 deletions
+49 -28
View File
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
```
`scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash, content hash). The
database persists between runs. A rescan hashes only the files that are new or
changed, and removes the records of files that no longer exist. `report` reads
the database and prints the file-level duplicates report. `trees` reads the same
database and prints the duplicate-tree report. `info` prints the database's
details: its path, the sizes of its files and the number of its records. A
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
prints a usage message and exits 2.
regular file (path, size, size on disk, mtime, head hash, tail hash, content
hash). The database persists between runs. A rescan hashes only the files that
are new or changed, and removes the records of files that no longer exist.
`report` reads the database and prints the file-level duplicates report. `trees`
reads the same database and prints the duplicate-tree report. `info` prints the
database's details: its path, the sizes of its files and the number of its
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -226,12 +226,12 @@ Goals, in order:
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
stay in the database). The reporting commands do not hold every file's
hashes either: `report` reads the records in signature order and holds the
duplicate paths of one size at a time, so its memory grows with the
duplicates of the size that has the most of them, and `trees` reads the
under resilver). Holding one small record (path, size, size on disk, mtime)
per file in memory during a scan is acceptable; holding every file's hashes
is not (they stay in the database). The reporting commands do not hold every
file's hashes either: `report` reads the records in signature order and
holds the duplicate paths of one size at a time, so its memory grows with
the duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the
@@ -368,7 +368,8 @@ All four subcommands operate on a single SQLite database file:
- While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, and the deletions.
- In the update phase: the rest of the hashed records, the new `disk_size`
of unchanged files whose size on disk changed, and the deletions.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read
@@ -386,6 +387,7 @@ All four subcommands operate on a single SQLite database file:
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
@@ -514,6 +516,11 @@ operands:
- A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap.
- When such a file's size on disk differs from the recorded `disk_size`, the
scan updates `disk_size` alone. It still does not read the file, and counts it
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
the write. A file scanned within those seconds gets its real `disk_size` at
the next scan.
- A file whose mtime is newer than recorded, or whose size differs, is hashed
again. Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
@@ -541,8 +548,8 @@ inside the walk and hash phases, and batched database writes begin during the
hash phase:
1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size and its mtime, for the walk to compare
each file with.
in memory, by path, with its size, its size on disk and its mtime, for the
walk to compare each file with.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back
@@ -550,9 +557,10 @@ hash phase:
(while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes, and
an unchanged file needs nothing more. Every new or changed file is carried
to the hash phase as a (path, size, mtime) record.
or busy pools. The walk compares each file with its record as it goes. An
unchanged file is not read again. When its size on disk has changed, the
update phase writes its new `disk_size`. Every new or changed file is
carried to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened.
@@ -564,7 +572,8 @@ hash phase:
committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
4. **update** — commit the final partial batch of hashed records, and the new
`disk_size` of each unchanged file whose size on disk changed. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
@@ -809,14 +818,26 @@ Processing:
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
group of N). `files` is the recursive regular-file count of one copy of the
tree; `size` is the recursive total byte size of one copy:
group of N). After the `first` and `dupe` paths come three numbers:
- `files`, the number of regular files in one copy of the tree, counted
recursively;
- `size`, the bytes of one copy, its files' sizes added up;
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
added up.
```
first dupe files size
/srv/a/project /srv/backup/project 3417 104857600
first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984
```
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file, so two copies of one tree can differ
in `disk_size`. When a file in one copy is a hard link to one in the other, both
copies count its blocks in full. Removing it from one copy then frees nothing,
as the known limitation under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`.
@@ -1200,9 +1221,9 @@ All of the following, run in this directory, must pass:
ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
differs by name).
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so