Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The files table gets a disk_size column: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes. trees adds it up per directory and prints it as a column after size. On a compressed or sparse file it is less than size, so two copies of one tree can differ; each row shows the dupe copy's. Judgement call: a row's disk_size is the dupe copy's, not the first's. Unchanged files keep the disk_size from their last hash: a change in allocation that leaves size and mtime alone is not picked up. Model: opus-5-5
This commit is contained in:
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
|||||||
```
|
```
|
||||||
|
|
||||||
`scan` walks one or more filesystem trees and maintains one database record per
|
`scan` walks one or more filesystem trees and maintains one database record per
|
||||||
regular file (path, size, mtime, head hash, tail hash, content hash). The
|
regular file (path, size, size on disk, mtime, head hash, tail hash, content
|
||||||
database persists between runs. A rescan hashes only the files that are new or
|
hash). The database persists between runs. A rescan hashes only the files that
|
||||||
changed, and removes the records of files that no longer exist. `report` reads
|
are new or changed, and removes the records of files that no longer exist.
|
||||||
the database and prints the file-level duplicates report. `trees` reads the same
|
`report` reads the database and prints the file-level duplicates report. `trees`
|
||||||
database and prints the duplicate-tree report. `info` prints the database's
|
reads the same database and prints the duplicate-tree report. `info` prints the
|
||||||
details: its path, the sizes of its files and the number of its records. A
|
database's details: its path, the sizes of its files and the number of its
|
||||||
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
|
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||||||
prints a usage message and exits 2.
|
operand — prints a usage message and exits 2.
|
||||||
|
|
||||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||||
@@ -386,6 +386,7 @@ All four subcommands operate on a single SQLite database file:
|
|||||||
CREATE TABLE files (
|
CREATE TABLE files (
|
||||||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||||||
size INTEGER NOT NULL, -- bytes, from lstat
|
size INTEGER NOT NULL, -- bytes, from lstat
|
||||||
|
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
|
||||||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
||||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||||||
@@ -552,7 +553,7 @@ hash phase:
|
|||||||
files; per-directory parallelism is what makes the walk tractable on large
|
files; per-directory parallelism is what makes the walk tractable on large
|
||||||
or busy pools. The walk compares each file with its record as it goes, and
|
or busy pools. The walk compares each file with its record as it goes, and
|
||||||
an unchanged file needs nothing more. Every new or changed file is carried
|
an unchanged file needs nothing more. Every new or changed file is carried
|
||||||
to the hash phase as a (path, size, mtime) record.
|
to the hash phase as a (path, size, size on disk, mtime) record.
|
||||||
3. **hash** — the worker pool hashes every carried file as described in
|
3. **hash** — the worker pool hashes every carried file as described in
|
||||||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||||||
`content`. Zero-length files have constant hashes and are never opened.
|
`content`. Zero-length files have constant hashes and are never opened.
|
||||||
@@ -809,14 +810,26 @@ Processing:
|
|||||||
#### Trees output format
|
#### Trees output format
|
||||||
|
|
||||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||||
group of N). `files` is the recursive regular-file count of one copy of the
|
group of N). After the `first` and `dupe` paths come three numbers:
|
||||||
tree; `size` is the recursive total byte size of one copy:
|
|
||||||
|
- `files`, the number of regular files in one copy of the tree, counted
|
||||||
|
recursively;
|
||||||
|
- `size`, the bytes of one copy, its files' sizes added up;
|
||||||
|
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||||||
|
added up.
|
||||||
|
|
||||||
```
|
```
|
||||||
first dupe files size
|
first dupe files size disk_size
|
||||||
/srv/a/project /srv/backup/project 3417 104857600
|
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||||
```
|
```
|
||||||
|
|
||||||
|
A file's `disk_size` is the space its blocks take on disk. It is less than the
|
||||||
|
file's size on a compressed or sparse file, and a little more on most other
|
||||||
|
files. Two copies of one tree can therefore differ in `disk_size`. When a file
|
||||||
|
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||||
|
full. Removing it from one copy then frees nothing, as the known limitation
|
||||||
|
under "`trees` mode" says.
|
||||||
|
|
||||||
The `first` and `dupe` paths are escaped as described under "Report output
|
The `first` and `dupe` paths are escaped as described under "Report output
|
||||||
format". The root directory's path is `/`.
|
format". The root directory's path is `/`.
|
||||||
|
|
||||||
@@ -1200,9 +1213,9 @@ All of the following, run in this directory, must pass:
|
|||||||
ordered by size descending.
|
ordered by size descending.
|
||||||
|
|
||||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||||||
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
|
||||||
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
|
||||||
differs by name).
|
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
|
||||||
|
|
||||||
Expected from the second `report` (after the modify/delete rescan):
|
Expected from the second `report` (after the modify/delete rescan):
|
||||||
`one.bin` has left its group (its content changed), so
|
`one.bin` has left its group (its content changed), so
|
||||||
|
|||||||
@@ -29,6 +29,12 @@
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- each record holds `disk_size`, the bytes the file takes on disk: its allocated
|
||||||
|
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
|
||||||
|
prints a `disk_size` column after `size`, the sum over the `dupe` copy's
|
||||||
|
files; on a compressed or sparse file it differs from `size` (2026-10-27,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||||
|
|
||||||
- the database keeps the number of records and the sum of their sizes in the one
|
- the database keeps the number of records and the sum of their sizes in the one
|
||||||
row of a `totals` table, which three triggers on `files` keep current. `info`
|
row of a `totals` table, which three triggers on `files` keep current. `info`
|
||||||
reads that row, and so do `scan`'s `load` phase and the `read` phases of
|
reads that row, and so do `scan`'s `load` phase and the `read` phases of
|
||||||
|
|||||||
@@ -47,14 +47,15 @@ const dbFilePerm = 0o644
|
|||||||
const lockFilePerm = 0o600
|
const lockFilePerm = 0o600
|
||||||
|
|
||||||
// createTableSQL is the schema applied to a fresh database. Paths are
|
// createTableSQL is the schema applied to a fresh database. Paths are
|
||||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
|
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
|
||||||
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
|
// disk_size holds the bytes the file takes on disk. mtime holds whole
|
||||||
// second. head, tail and content each hold the raw 32 bytes of a
|
// Unix seconds and mtime_nsec the nanoseconds within that second. head,
|
||||||
// SHA-256.
|
// tail and content each hold the raw 32 bytes of a SHA-256.
|
||||||
const createTableSQL = `
|
const createTableSQL = `
|
||||||
CREATE TABLE files (
|
CREATE TABLE files (
|
||||||
path BLOB PRIMARY KEY,
|
path BLOB PRIMARY KEY,
|
||||||
size INTEGER NOT NULL,
|
size INTEGER NOT NULL,
|
||||||
|
disk_size INTEGER NOT NULL,
|
||||||
mtime INTEGER NOT NULL,
|
mtime INTEGER NOT NULL,
|
||||||
mtime_nsec INTEGER NOT NULL,
|
mtime_nsec INTEGER NOT NULL,
|
||||||
head BLOB NOT NULL,
|
head BLOB NOT NULL,
|
||||||
@@ -115,11 +116,13 @@ SELECT records FROM totals
|
|||||||
// upsertSQL inserts one file record, replacing any existing record for
|
// upsertSQL inserts one file record, replacing any existing record for
|
||||||
// the same path.
|
// the same path.
|
||||||
const upsertSQL = `
|
const upsertSQL = `
|
||||||
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
|
INSERT INTO files (
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
path, size, disk_size, mtime, mtime_nsec, head, tail, content
|
||||||
|
)
|
||||||
|
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||||
ON CONFLICT (path) DO UPDATE SET
|
ON CONFLICT (path) DO UPDATE SET
|
||||||
size = excluded.size, mtime = excluded.mtime,
|
size = excluded.size, disk_size = excluded.disk_size,
|
||||||
mtime_nsec = excluded.mtime_nsec,
|
mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
|
||||||
head = excluded.head, tail = excluded.tail,
|
head = excluded.head, tail = excluded.tail,
|
||||||
content = excluded.content
|
content = excluded.content
|
||||||
`
|
`
|
||||||
@@ -536,8 +539,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
total(n)
|
total(n)
|
||||||
|
|
||||||
rows, err := tx.QueryContext(ctx,
|
rows, err := tx.QueryContext(ctx,
|
||||||
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
|
"SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
|
||||||
"FROM files ORDER BY path")
|
"content FROM files ORDER BY path")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read records: %w", err)
|
return fmt.Errorf("read records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -551,7 +554,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
r scanRec
|
r scanRec
|
||||||
)
|
)
|
||||||
|
|
||||||
err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content)
|
err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
|
||||||
|
&head, &tail, &content)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read record: %w", err)
|
return fmt.Errorf("read record: %w", err)
|
||||||
}
|
}
|
||||||
@@ -586,10 +590,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
|
|||||||
`
|
`
|
||||||
|
|
||||||
// loadSignatureRows gives total the number of records, then streams
|
// loadSignatureRows gives total the number of records, then streams
|
||||||
// every record's signature and path to fn, without its mtime, in the
|
// every record's signature and path to fn, without its mtime or its
|
||||||
// order of signatureRowsSQL. The count and the records are read in one
|
// disk_size, in the order of signatureRowsSQL. The count and the
|
||||||
// transaction, so they agree while a scan is committing. An error from
|
// records are read in one transaction, so they agree while a scan is
|
||||||
// fn stops the reading and is returned as it is.
|
// committing. An error from fn stops the reading and is returned as it
|
||||||
|
// is.
|
||||||
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||||
total func(n int64), fn func(r scanRec) error,
|
total func(n int64), fn func(r scanRec) error,
|
||||||
) error {
|
) error {
|
||||||
@@ -833,7 +838,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
|||||||
defer func() { _ = st.Close() }()
|
defer func() { _ = st.Close() }()
|
||||||
|
|
||||||
for _, r := range upserts {
|
for _, r := range upserts {
|
||||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size,
|
_, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
|
||||||
r.mtime.Unix(), r.mtime.Nanosecond(),
|
r.mtime.Unix(), r.mtime.Nanosecond(),
|
||||||
r.head[:], r.tail[:], r.content[:])
|
r.head[:], r.tail[:], r.content[:])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+9
-6
@@ -1141,8 +1141,9 @@ func TestRunTreesSucceeds(t *testing.T) {
|
|||||||
|
|
||||||
// The two directories holding the duplicate pair are duplicate
|
// The two directories holding the duplicate pair are duplicate
|
||||||
// trees of each other.
|
// trees of each other.
|
||||||
want := "first\tdupe\tfiles\tsize\n" +
|
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||||
|
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -1176,9 +1177,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
|
|||||||
cases := map[string]string{
|
cases := map[string]string{
|
||||||
cmdReport: "first\tdupe\tsize\n" +
|
cmdReport: "first\tdupe\tsize\n" +
|
||||||
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
||||||
cmdTrees: "first\tdupe\tfiles\tsize\n" +
|
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
||||||
"\t1\t300\n",
|
"\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
|
||||||
|
"\n",
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, want := range cases {
|
for name, want := range cases {
|
||||||
@@ -1213,8 +1215,9 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
|||||||
t.Errorf("report stdout = %q, want %q", got, want)
|
t.Errorf("report stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
want = "first\tdupe\tfiles\tsize\n" +
|
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||||
|
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||||
if got := runStdout(t, cmdTrees); got != want {
|
if got := runStdout(t, cmdTrees); got != want {
|
||||||
t.Errorf("trees stdout = %q, want %q", got, want)
|
t.Errorf("trees stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,14 +21,16 @@ const minGroupSize = 2
|
|||||||
|
|
||||||
// scanRec is one file record from the database. The signature (size,
|
// scanRec is one file record from the database. The signature (size,
|
||||||
// head, tail, content) is the duplicate key; mtime is informational
|
// head, tail, content) is the duplicate key; mtime is informational
|
||||||
// only and used by scan for change detection.
|
// only and used by scan for change detection. diskSize is the bytes the
|
||||||
|
// file takes on disk, which trees adds up.
|
||||||
type scanRec struct {
|
type scanRec struct {
|
||||||
size int64
|
size int64
|
||||||
mtime time.Time
|
diskSize int64
|
||||||
head [sha256.Size]byte
|
mtime time.Time
|
||||||
tail [sha256.Size]byte
|
head [sha256.Size]byte
|
||||||
content [sha256.Size]byte
|
tail [sha256.Size]byte
|
||||||
path string
|
content [sha256.Size]byte
|
||||||
|
path string
|
||||||
}
|
}
|
||||||
|
|
||||||
// reportPhase names one phase of report with its place among all of
|
// reportPhase names one phase of report with its place among all of
|
||||||
|
|||||||
+2
-2
@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
|
|||||||
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
||||||
|
|
||||||
return []scanRec{
|
return []scanRec{
|
||||||
{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"},
|
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||||
{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -52,15 +52,17 @@ const workQueueDepth = 1024
|
|||||||
// has already printed its line, so run prints nothing more.
|
// has already printed its line, so run prints nothing more.
|
||||||
var errInterrupted = errors.New("scan interrupted")
|
var errInterrupted = errors.New("scan interrupted")
|
||||||
|
|
||||||
// fileRec carries one statted file between the scan phases. dev and
|
// fileRec carries one statted file between the scan phases. diskSize is
|
||||||
// ino identify the underlying inode so hard-linked paths can share
|
// the bytes the file takes on disk. dev and ino identify the underlying
|
||||||
// one read; both are zero when the platform exposes no inode.
|
// inode so hard-linked paths can share one read; both are zero when the
|
||||||
|
// platform exposes no inode.
|
||||||
type fileRec struct {
|
type fileRec struct {
|
||||||
path string
|
path string
|
||||||
size int64
|
size int64
|
||||||
mtime time.Time
|
diskSize int64
|
||||||
dev uint64
|
mtime time.Time
|
||||||
ino uint64
|
dev uint64
|
||||||
|
ino uint64
|
||||||
}
|
}
|
||||||
|
|
||||||
// fileMeta is the in-memory index entry for one existing database
|
// fileMeta is the in-memory index entry for one existing database
|
||||||
@@ -695,12 +697,13 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
|
|||||||
s.resolve(rec.path)
|
s.resolve(rec.path)
|
||||||
|
|
||||||
s.batch = append(s.batch, scanRec{
|
s.batch = append(s.batch, scanRec{
|
||||||
size: rec.size,
|
size: rec.size,
|
||||||
mtime: rec.mtime,
|
diskSize: rec.diskSize,
|
||||||
head: r.head,
|
mtime: rec.mtime,
|
||||||
tail: r.tail,
|
head: r.head,
|
||||||
content: r.content,
|
tail: r.tail,
|
||||||
path: rec.path,
|
content: r.content,
|
||||||
|
path: rec.path,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -961,11 +964,12 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
|||||||
|
|
||||||
dev, ino := inodeOfInfo(fi)
|
dev, ino := inodeOfInfo(fi)
|
||||||
rec := fileRec{
|
rec := fileRec{
|
||||||
path: root,
|
path: root,
|
||||||
size: fi.Size(),
|
size: fi.Size(),
|
||||||
mtime: fi.ModTime(),
|
diskSize: diskSizeOfInfo(fi),
|
||||||
dev: dev,
|
mtime: fi.ModTime(),
|
||||||
ino: ino,
|
dev: dev,
|
||||||
|
ino: ino,
|
||||||
}
|
}
|
||||||
|
|
||||||
if !isDatabaseFile(rec, rules) {
|
if !isDatabaseFile(rec, rules) {
|
||||||
@@ -1145,11 +1149,12 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
|
|
||||||
dev, ino := inodeOfInfo(info)
|
dev, ino := inodeOfInfo(info)
|
||||||
rec := fileRec{
|
rec := fileRec{
|
||||||
path: p,
|
path: p,
|
||||||
size: info.Size(),
|
size: info.Size(),
|
||||||
mtime: info.ModTime(),
|
diskSize: diskSizeOfInfo(info),
|
||||||
dev: dev,
|
mtime: info.ModTime(),
|
||||||
ino: ino,
|
dev: dev,
|
||||||
|
ino: ino,
|
||||||
}
|
}
|
||||||
|
|
||||||
if !isDatabaseFile(rec, rules) {
|
if !isDatabaseFile(rec, rules) {
|
||||||
@@ -1241,6 +1246,23 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
|
|||||||
return statDev(st), st.Ino
|
return statDev(st), st.Ino
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// statBlockSize is the unit of st_blocks in an lstat result on Linux and
|
||||||
|
// macOS, whatever the filesystem's own block size.
|
||||||
|
const statBlockSize = 512
|
||||||
|
|
||||||
|
// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
|
||||||
|
// blocks, st_blocks × 512. On a compressed or sparse file it is less
|
||||||
|
// than the file's size. It is 0 when the platform exposes no block
|
||||||
|
// count.
|
||||||
|
func diskSizeOfInfo(fi fs.FileInfo) int64 {
|
||||||
|
st, ok := fi.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
return st.Blocks * statBlockSize
|
||||||
|
}
|
||||||
|
|
||||||
// hashResult carries the hashes computed for one inode run (or the
|
// hashResult carries the hashes computed for one inode run (or the
|
||||||
// error that prevented computing them) from the pool's workers to the
|
// error that prevented computing them) from the pool's workers to the
|
||||||
// hash phase. Its run leaves out the paths found removed.
|
// hash phase. Its run leaves out the paths found removed.
|
||||||
|
|||||||
@@ -42,6 +42,21 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
|||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||||
|
// own lstat: st_blocks counts 512-byte units.
|
||||||
|
func diskSizeOf(t *testing.T, path string) int64 {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
var st syscall.Stat_t
|
||||||
|
|
||||||
|
err := syscall.Lstat(path, &st)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return st.Blocks * 512
|
||||||
|
}
|
||||||
|
|
||||||
// pattern returns n bytes of deterministic content seeded by tag.
|
// pattern returns n bytes of deterministic content seeded by tag.
|
||||||
func pattern(tag byte, n int) []byte {
|
func pattern(tag byte, n int) []byte {
|
||||||
data := make([]byte, n)
|
data := make([]byte, n)
|
||||||
@@ -1416,6 +1431,46 @@ func TestScanStoresRawHashes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestScanStoresDiskSize scans a sparse file, which takes less space on
|
||||||
|
// disk than its length, and checks the disk_size stored for it against
|
||||||
|
// the file's own lstat.
|
||||||
|
func TestScanStoresDiskSize(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
db := openTestDB(t)
|
||||||
|
p := writeFile(t, dir, "sparse", []byte("x"))
|
||||||
|
|
||||||
|
// Truncate makes the file longer without giving it blocks for the
|
||||||
|
// new bytes.
|
||||||
|
const length = 1024 * 1024
|
||||||
|
|
||||||
|
err := os.Truncate(p, length)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
syncTree(t, db, dir)
|
||||||
|
|
||||||
|
want := diskSizeOf(t, p)
|
||||||
|
if want >= length {
|
||||||
|
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
|
||||||
|
"length %d", want, length)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got int64
|
||||||
|
|
||||||
|
err = db.QueryRowContext(t.Context(),
|
||||||
|
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("disk_size = %d, want %d", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
||||||
// size with an mtime later in the same second as the recorded one: the
|
// size with an mtime later in the same second as the recorded one: the
|
||||||
// next scan must notice the change and re-hash the file. The operand is
|
// next scan must notice the change and re-hash the file. The operand is
|
||||||
|
|||||||
@@ -22,6 +22,10 @@ type treeNode struct {
|
|||||||
digest [sha256.Size]byte
|
digest [sha256.Size]byte
|
||||||
fileCount int64
|
fileCount int64
|
||||||
totalSize int64
|
totalSize int64
|
||||||
|
// diskSize is the bytes the directory's files take on disk. Two
|
||||||
|
// copies of one tree have the same totalSize, but their diskSize
|
||||||
|
// can differ.
|
||||||
|
diskSize int64
|
||||||
}
|
}
|
||||||
|
|
||||||
// treesPhase names one phase of trees with its place among all of them,
|
// treesPhase names one phase of trees with its place among all of them,
|
||||||
@@ -139,15 +143,16 @@ func readTree(ctx context.Context, db *sql.DB,
|
|||||||
|
|
||||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||||
// a row for every member of each group but its first, counting each
|
// a row for every member of each group but its first, counting each
|
||||||
// group on prog once its rows are written. stdout may be the terminal
|
// group on prog once its rows are written. The disk_size column is the
|
||||||
// prog is drawn on, so the lines go through prog's writer. It returns
|
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
||||||
// the number of rows (dupe trees) and the bytes they could reclaim.
|
// go through prog's writer. It returns the number of rows (dupe trees)
|
||||||
|
// and the bytes they could reclaim.
|
||||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
) (int, int64, error) {
|
) (int, int64, error) {
|
||||||
out := prog.writer(stdout)
|
out := prog.writer(stdout)
|
||||||
|
|
||||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
@@ -159,9 +164,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
|||||||
for _, g := range dupes {
|
for _, g := range dupes {
|
||||||
first := g[0]
|
first := g[0]
|
||||||
for _, n := range g[1:] {
|
for _, n := range g[1:] {
|
||||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
|
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
|
||||||
escapePath(first.path), escapePath(n.path),
|
escapePath(first.path), escapePath(n.path),
|
||||||
first.fileCount, first.totalSize)
|
first.fileCount, first.totalSize, n.diskSize)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
@@ -232,6 +237,7 @@ func (b *treeBuilder) add(r scanRec) {
|
|||||||
dir.entries = append(dir.entries, fileEntry(name, r))
|
dir.entries = append(dir.entries, fileEntry(name, r))
|
||||||
dir.fileCount++
|
dir.fileCount++
|
||||||
dir.totalSize += r.size
|
dir.totalSize += r.size
|
||||||
|
dir.diskSize += r.diskSize
|
||||||
}
|
}
|
||||||
|
|
||||||
// openDir starts the directory called name inside the innermost open
|
// openDir starts the directory called name inside the innermost open
|
||||||
@@ -270,6 +276,7 @@ func (b *treeBuilder) closeTo(n int) {
|
|||||||
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
||||||
dir.parent.fileCount += dir.fileCount
|
dir.parent.fileCount += dir.fileCount
|
||||||
dir.parent.totalSize += dir.totalSize
|
dir.parent.totalSize += dir.totalSize
|
||||||
|
dir.parent.diskSize += dir.diskSize
|
||||||
b.dirs = append(b.dirs, dir)
|
b.dirs = append(b.dirs, dir)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+40
-11
@@ -17,18 +17,24 @@ const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
|||||||
|
|
||||||
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
|
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
|
||||||
// /d/t2 are identical, /d/t3 differs from them only by one filename.
|
// /d/t2 are identical, /d/t3 differs from them only by one filename.
|
||||||
|
// Each file takes one 4 KiB block on disk.
|
||||||
func smokeTreeRecs() []scanRec {
|
func smokeTreeRecs() []scanRec {
|
||||||
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
||||||
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
||||||
|
|
||||||
return []scanRec{
|
return []scanRec{
|
||||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
|
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
|
content: f1Content, path: "/d/t1/f1"},
|
||||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
|
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
|
content: f2Content, path: "/d/t1/sub/f2"},
|
||||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
|
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
|
content: f1Content, path: "/d/t2/f1"},
|
||||||
path: "/d/t3/sub/f2renamed"},
|
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||||
|
content: f2Content, path: "/d/t2/sub/f2"},
|
||||||
|
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||||
|
content: f1Content, path: "/d/t3/f1"},
|
||||||
|
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||||
|
content: f2Content, path: "/d/t3/sub/f2renamed"},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -198,13 +204,33 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
|||||||
code, exitOK, stderr.String())
|
code, exitOK, stderr.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
want := "first\tdupe\tfiles\tsize\n" +
|
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
|
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||||
|
// take different space on disk. The row's disk_size is the sum over the
|
||||||
|
// dupe copy's files, those in its subdirectory included.
|
||||||
|
func TestRunTreesDiskSize(t *testing.T) {
|
||||||
|
f, g := testHash("f"), testHash("g")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
|
||||||
|
{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
|
||||||
|
{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
|
||||||
|
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
|
want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
|
||||||
|
if got := runStdout(t, cmdTrees); got != want {
|
||||||
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
||||||
// records with stderr not a terminal, and checks every line it prints
|
// records with stderr not a terminal, and checks every line it prints
|
||||||
// there. After the line naming the database, each phase prints a line
|
// there. After the line naming the database, each phase prints a line
|
||||||
@@ -226,7 +252,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
|||||||
code, exitOK, stderr())
|
code, exitOK, stderr())
|
||||||
}
|
}
|
||||||
|
|
||||||
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
|
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
|
"/d/t1\t/d/t2\t2\t3100\t8192\n"
|
||||||
if got := stdout.String(); got != report {
|
if got := stdout.String(); got != report {
|
||||||
t.Errorf("stdout = %q, want %q", got, report)
|
t.Errorf("stdout = %q, want %q", got, report)
|
||||||
}
|
}
|
||||||
@@ -291,7 +318,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
|||||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||||
}
|
}
|
||||||
|
|
||||||
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
|
want := []string{
|
||||||
|
"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
|
||||||
|
}
|
||||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user