Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The files table gets a disk_size column: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes. trees adds it up per directory and prints it as a column after size. On a compressed or sparse file it is less than size, so two copies of one tree can differ; each row shows the dupe copy's. Judgement call: a row's disk_size is the dupe copy's, not the first's. Unchanged files keep the disk_size from their last hash: a change in allocation that leaves size and mtime alone is not picked up. Model: opus-5-5
This commit is contained in:
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
```
|
||||
|
||||
`scan` walks one or more filesystem trees and maintains one database record per
|
||||
regular file (path, size, mtime, head hash, tail hash, content hash). The
|
||||
database persists between runs. A rescan hashes only the files that are new or
|
||||
changed, and removes the records of files that no longer exist. `report` reads
|
||||
the database and prints the file-level duplicates report. `trees` reads the same
|
||||
database and prints the duplicate-tree report. `info` prints the database's
|
||||
details: its path, the sizes of its files and the number of its records. A
|
||||
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
|
||||
prints a usage message and exits 2.
|
||||
regular file (path, size, size on disk, mtime, head hash, tail hash, content
|
||||
hash). The database persists between runs. A rescan hashes only the files that
|
||||
are new or changed, and removes the records of files that no longer exist.
|
||||
`report` reads the database and prints the file-level duplicates report. `trees`
|
||||
reads the same database and prints the duplicate-tree report. `info` prints the
|
||||
database's details: its path, the sizes of its files and the number of its
|
||||
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||||
operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
@@ -386,6 +386,7 @@ All four subcommands operate on a single SQLite database file:
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||||
size INTEGER NOT NULL, -- bytes, from lstat
|
||||
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
|
||||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||||
@@ -552,7 +553,7 @@ hash phase:
|
||||
files; per-directory parallelism is what makes the walk tractable on large
|
||||
or busy pools. The walk compares each file with its record as it goes, and
|
||||
an unchanged file needs nothing more. Every new or changed file is carried
|
||||
to the hash phase as a (path, size, mtime) record.
|
||||
to the hash phase as a (path, size, size on disk, mtime) record.
|
||||
3. **hash** — the worker pool hashes every carried file as described in
|
||||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||||
`content`. Zero-length files have constant hashes and are never opened.
|
||||
@@ -809,14 +810,26 @@ Processing:
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
group of N). `files` is the recursive regular-file count of one copy of the
|
||||
tree; `size` is the recursive total byte size of one copy:
|
||||
group of N). After the `first` and `dupe` paths come three numbers:
|
||||
|
||||
- `files`, the number of regular files in one copy of the tree, counted
|
||||
recursively;
|
||||
- `size`, the bytes of one copy, its files' sizes added up;
|
||||
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||||
added up.
|
||||
|
||||
```
|
||||
first dupe files size
|
||||
/srv/a/project /srv/backup/project 3417 104857600
|
||||
first dupe files size disk_size
|
||||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||
```
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It is less than the
|
||||
file's size on a compressed or sparse file, and a little more on most other
|
||||
files. Two copies of one tree can therefore differ in `disk_size`. When a file
|
||||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||
full. Removing it from one copy then frees nothing, as the known limitation
|
||||
under "`trees` mode" says.
|
||||
|
||||
The `first` and `dupe` paths are escaped as described under "Report output
|
||||
format". The root directory's path is `/`.
|
||||
|
||||
@@ -1200,9 +1213,9 @@ All of the following, run in this directory, must pass:
|
||||
ordered by size descending.
|
||||
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||||
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
||||
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
||||
differs by name).
|
||||
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
|
||||
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
|
||||
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
|
||||
|
||||
Expected from the second `report` (after the modify/delete rescan):
|
||||
`one.bin` has left its group (its content changed), so
|
||||
|
||||
Reference in New Issue
Block a user