Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s

The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. trees adds it up per directory and prints
it as a column after size. On a compressed or sparse file it is less
than size, so two copies of one tree can differ; each row shows the
dupe copy's.

Judgement call: a row's disk_size is the dupe copy's, not the first's.
Unchanged files keep the disk_size from their last hash: a change in
allocation that leaves size and mtime alone is not picked up.

Model: opus-5-5
This commit is contained in:
2026-10-27 18:08:27 +00:00
parent 98193b2896
commit dfbfdd0601
10 changed files with 230 additions and 88 deletions
+29 -16
View File
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
```
`scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash, content hash). The
database persists between runs. A rescan hashes only the files that are new or
changed, and removes the records of files that no longer exist. `report` reads
the database and prints the file-level duplicates report. `trees` reads the same
database and prints the duplicate-tree report. `info` prints the database's
details: its path, the sizes of its files and the number of its records. A
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
prints a usage message and exits 2.
regular file (path, size, size on disk, mtime, head hash, tail hash, content
hash). The database persists between runs. A rescan hashes only the files that
are new or changed, and removes the records of files that no longer exist.
`report` reads the database and prints the file-level duplicates report. `trees`
reads the same database and prints the duplicate-tree report. `info` prints the
database's details: its path, the sizes of its files and the number of its
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -386,6 +386,7 @@ All four subcommands operate on a single SQLite database file:
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
@@ -552,7 +553,7 @@ hash phase:
files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes, and
an unchanged file needs nothing more. Every new or changed file is carried
to the hash phase as a (path, size, mtime) record.
to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened.
@@ -809,14 +810,26 @@ Processing:
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
group of N). `files` is the recursive regular-file count of one copy of the
tree; `size` is the recursive total byte size of one copy:
group of N). After the `first` and `dupe` paths come three numbers:
- `files`, the number of regular files in one copy of the tree, counted
recursively;
- `size`, the bytes of one copy, its files' sizes added up;
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
added up.
```
first dupe files size
/srv/a/project /srv/backup/project 3417 104857600
first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984
```
A file's `disk_size` is the space its blocks take on disk. It is less than the
file's size on a compressed or sparse file, and a little more on most other
files. Two copies of one tree can therefore differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the known limitation
under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`.
@@ -1200,9 +1213,9 @@ All of the following, run in this directory, must pass:
ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
differs by name).
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so