Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s

The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. trees adds it up per directory and prints
it as a column after size. On a compressed or sparse file it is less
than size, so two copies of one tree can differ; each row shows the
dupe copy's.

Judgement call: a row's disk_size is the dupe copy's, not the first's.
Unchanged files keep the disk_size from their last hash: a change in
allocation that leaves size and mtime alone is not picked up.

Model: opus-5-5
This commit is contained in:
2026-10-27 18:08:27 +00:00
parent 98193b2896
commit dfbfdd0601
10 changed files with 230 additions and 88 deletions
+29 -16
View File
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
``` ```
`scan` walks one or more filesystem trees and maintains one database record per `scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash, content hash). The regular file (path, size, size on disk, mtime, head hash, tail hash, content
database persists between runs. A rescan hashes only the files that are new or hash). The database persists between runs. A rescan hashes only the files that
changed, and removes the records of files that no longer exist. `report` reads are new or changed, and removes the records of files that no longer exist.
the database and prints the file-level duplicates report. `trees` reads the same `report` reads the database and prints the file-level duplicates report. `trees`
database and prints the duplicate-tree report. `info` prints the database's reads the same database and prints the duplicate-tree report. `info` prints the
details: its path, the sizes of its files and the number of its records. A database's details: its path, the sizes of its files and the number of its
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand — records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
prints a usage message and exits 2. operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan` by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -386,6 +386,7 @@ All four subcommands operate on a single SQLite database file:
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat size INTEGER NOT NULL, -- bytes, from lstat
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999 mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
@@ -552,7 +553,7 @@ hash phase:
files; per-directory parallelism is what makes the walk tractable on large files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes, and or busy pools. The walk compares each file with its record as it goes, and
an unchanged file needs nothing more. Every new or changed file is carried an unchanged file needs nothing more. Every new or changed file is carried
to the hash phase as a (path, size, mtime) record. to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in 3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and "Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened. `content`. Zero-length files have constant hashes and are never opened.
@@ -809,14 +810,26 @@ Processing:
#### Trees output format #### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
group of N). `files` is the recursive regular-file count of one copy of the group of N). After the `first` and `dupe` paths come three numbers:
tree; `size` is the recursive total byte size of one copy:
- `files`, the number of regular files in one copy of the tree, counted
recursively;
- `size`, the bytes of one copy, its files' sizes added up;
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
added up.
``` ```
first dupe files size first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 /srv/a/project /srv/backup/project 3417 104857600 61865984
``` ```
A file's `disk_size` is the space its blocks take on disk. It is less than the
file's size on a compressed or sparse file, and a little more on most other
files. Two copies of one tree can therefore differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the known limitation
under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`. format". The root directory's path is `/`.
@@ -1200,9 +1213,9 @@ All of the following, run in this directory, must pass:
ordered by size descending. ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2 Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal files, 3100 bytes, and a `disk_size` that depends on the filesystem.
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
differs by name). `t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
Expected from the second `report` (after the modify/delete rescan): Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so `one.bin` has left its group (its content changed), so
+6
View File
@@ -29,6 +29,12 @@
# Completed Steps # Completed Steps
- each record holds `disk_size`, the bytes the file takes on disk: its allocated
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
prints a `disk_size` column after `size`, the sum over the `dupe` copy's
files; on a compressed or sparse file it differs from `size` (2026-10-27,
https://git.eeqj.de/sneak/sfdupes/issues/147)
- the database keeps the number of records and the sum of their sizes in the one - the database keeps the number of records and the sum of their sizes in the one
row of a `totals` table, which three triggers on `files` keep current. `info` row of a `totals` table, which three triggers on `files` keep current. `info`
reads that row, and so do `scan`'s `load` phase and the `read` phases of reads that row, and so do `scan`'s `load` phase and the `read` phases of
+21 -16
View File
@@ -47,14 +47,15 @@ const dbFilePerm = 0o644
const lockFilePerm = 0o600 const lockFilePerm = 0o600
// createTableSQL is the schema applied to a fresh database. Paths are // createTableSQL is the schema applied to a fresh database. Paths are
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime // BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
// holds whole Unix seconds and mtime_nsec the nanoseconds within that // disk_size holds the bytes the file takes on disk. mtime holds whole
// second. head, tail and content each hold the raw 32 bytes of a // Unix seconds and mtime_nsec the nanoseconds within that second. head,
// SHA-256. // tail and content each hold the raw 32 bytes of a SHA-256.
const createTableSQL = ` const createTableSQL = `
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, path BLOB PRIMARY KEY,
size INTEGER NOT NULL, size INTEGER NOT NULL,
disk_size INTEGER NOT NULL,
mtime INTEGER NOT NULL, mtime INTEGER NOT NULL,
mtime_nsec INTEGER NOT NULL, mtime_nsec INTEGER NOT NULL,
head BLOB NOT NULL, head BLOB NOT NULL,
@@ -115,11 +116,13 @@ SELECT records FROM totals
// upsertSQL inserts one file record, replacing any existing record for // upsertSQL inserts one file record, replacing any existing record for
// the same path. // the same path.
const upsertSQL = ` const upsertSQL = `
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content) INSERT INTO files (
VALUES (?, ?, ?, ?, ?, ?, ?) path, size, disk_size, mtime, mtime_nsec, head, tail, content
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (path) DO UPDATE SET ON CONFLICT (path) DO UPDATE SET
size = excluded.size, mtime = excluded.mtime, size = excluded.size, disk_size = excluded.disk_size,
mtime_nsec = excluded.mtime_nsec, mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
head = excluded.head, tail = excluded.tail, head = excluded.head, tail = excluded.tail,
content = excluded.content content = excluded.content
` `
@@ -536,8 +539,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
total(n) total(n)
rows, err := tx.QueryContext(ctx, rows, err := tx.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+ "SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
"FROM files ORDER BY path") "content FROM files ORDER BY path")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -551,7 +554,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
r scanRec r scanRec
) )
err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content) err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
&head, &tail, &content)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
@@ -586,10 +590,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
` `
// loadSignatureRows gives total the number of records, then streams // loadSignatureRows gives total the number of records, then streams
// every record's signature and path to fn, without its mtime, in the // every record's signature and path to fn, without its mtime or its
// order of signatureRowsSQL. The count and the records are read in one // disk_size, in the order of signatureRowsSQL. The count and the
// transaction, so they agree while a scan is committing. An error from // records are read in one transaction, so they agree while a scan is
// fn stops the reading and is returned as it is. // committing. An error from fn stops the reading and is returned as it
// is.
func loadSignatureRows(ctx context.Context, db *sql.DB, func loadSignatureRows(ctx context.Context, db *sql.DB,
total func(n int64), fn func(r scanRec) error, total func(n int64), fn func(r scanRec) error,
) error { ) error {
@@ -833,7 +838,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
defer func() { _ = st.Close() }() defer func() { _ = st.Close() }()
for _, r := range upserts { for _, r := range upserts {
_, err = st.ExecContext(ctx, []byte(r.path), r.size, _, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
r.mtime.Unix(), r.mtime.Nanosecond(), r.mtime.Unix(), r.mtime.Nanosecond(),
r.head[:], r.tail[:], r.content[:]) r.head[:], r.tail[:], r.content[:])
if err != nil { if err != nil {
+9 -6
View File
@@ -1141,8 +1141,9 @@ func TestRunTreesSucceeds(t *testing.T) {
// The two directories holding the duplicate pair are duplicate // The two directories holding the duplicate pair are duplicate
// trees of each other. // trees of each other.
want := "first\tdupe\tfiles\tsize\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n" filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -1176,9 +1177,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
cases := map[string]string{ cases := map[string]string{
cmdReport: "first\tdupe\tsize\n" + cmdReport: "first\tdupe\tsize\n" +
dupes[0] + "\t" + dupes[1] + "\t300\n", dupes[0] + "\t" + dupes[1] + "\t300\n",
cmdTrees: "first\tdupe\tfiles\tsize\n" + cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
"\t1\t300\n", "\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
"\n",
} }
for name, want := range cases { for name, want := range cases {
@@ -1213,8 +1215,9 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Errorf("report stdout = %q, want %q", got, want) t.Errorf("report stdout = %q, want %q", got, want)
} }
want = "first\tdupe\tfiles\tsize\n" + want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n" filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
if got := runStdout(t, cmdTrees); got != want { if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want) t.Errorf("trees stdout = %q, want %q", got, want)
} }
+9 -7
View File
@@ -21,14 +21,16 @@ const minGroupSize = 2
// scanRec is one file record from the database. The signature (size, // scanRec is one file record from the database. The signature (size,
// head, tail, content) is the duplicate key; mtime is informational // head, tail, content) is the duplicate key; mtime is informational
// only and used by scan for change detection. // only and used by scan for change detection. diskSize is the bytes the
// file takes on disk, which trees adds up.
type scanRec struct { type scanRec struct {
size int64 size int64
mtime time.Time diskSize int64
head [sha256.Size]byte mtime time.Time
tail [sha256.Size]byte head [sha256.Size]byte
content [sha256.Size]byte tail [sha256.Size]byte
path string content [sha256.Size]byte
path string
} }
// reportPhase names one phase of report with its place among all of // reportPhase names one phase of report with its place among all of
+2 -2
View File
@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
h, tl, c := testHash("h"), testHash("t"), testHash("c") h, tl, c := testHash("h"), testHash("t"), testHash("c")
return []scanRec{ return []scanRec{
{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"}, {size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"}, {size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
} }
} }
+46 -24
View File
@@ -52,15 +52,17 @@ const workQueueDepth = 1024
// has already printed its line, so run prints nothing more. // has already printed its line, so run prints nothing more.
var errInterrupted = errors.New("scan interrupted") var errInterrupted = errors.New("scan interrupted")
// fileRec carries one statted file between the scan phases. dev and // fileRec carries one statted file between the scan phases. diskSize is
// ino identify the underlying inode so hard-linked paths can share // the bytes the file takes on disk. dev and ino identify the underlying
// one read; both are zero when the platform exposes no inode. // inode so hard-linked paths can share one read; both are zero when the
// platform exposes no inode.
type fileRec struct { type fileRec struct {
path string path string
size int64 size int64
mtime time.Time diskSize int64
dev uint64 mtime time.Time
ino uint64 dev uint64
ino uint64
} }
// fileMeta is the in-memory index entry for one existing database // fileMeta is the in-memory index entry for one existing database
@@ -695,12 +697,13 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
s.resolve(rec.path) s.resolve(rec.path)
s.batch = append(s.batch, scanRec{ s.batch = append(s.batch, scanRec{
size: rec.size, size: rec.size,
mtime: rec.mtime, diskSize: rec.diskSize,
head: r.head, mtime: rec.mtime,
tail: r.tail, head: r.head,
content: r.content, tail: r.tail,
path: rec.path, content: r.content,
path: rec.path,
}) })
} }
@@ -961,11 +964,12 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
rec := fileRec{ rec := fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime(), diskSize: diskSizeOfInfo(fi),
dev: dev, mtime: fi.ModTime(),
ino: ino, dev: dev,
ino: ino,
} }
if !isDatabaseFile(rec, rules) { if !isDatabaseFile(rec, rules) {
@@ -1145,11 +1149,12 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
dev, ino := inodeOfInfo(info) dev, ino := inodeOfInfo(info)
rec := fileRec{ rec := fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), diskSize: diskSizeOfInfo(info),
dev: dev, mtime: info.ModTime(),
ino: ino, dev: dev,
ino: ino,
} }
if !isDatabaseFile(rec, rules) { if !isDatabaseFile(rec, rules) {
@@ -1241,6 +1246,23 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
return statDev(st), st.Ino return statDev(st), st.Ino
} }
// statBlockSize is the unit of st_blocks in an lstat result on Linux and
// macOS, whatever the filesystem's own block size.
const statBlockSize = 512
// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
// blocks, st_blocks × 512. On a compressed or sparse file it is less
// than the file's size. It is 0 when the platform exposes no block
// count.
func diskSizeOfInfo(fi fs.FileInfo) int64 {
st, ok := fi.Sys().(*syscall.Stat_t)
if !ok {
return 0
}
return st.Blocks * statBlockSize
}
// hashResult carries the hashes computed for one inode run (or the // hashResult carries the hashes computed for one inode run (or the
// error that prevented computing them) from the pool's workers to the // error that prevented computing them) from the pool's workers to the
// hash phase. Its run leaves out the paths found removed. // hash phase. Its run leaves out the paths found removed.
+55
View File
@@ -42,6 +42,21 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
return p return p
} }
// diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units.
func diskSizeOf(t *testing.T, path string) int64 {
t.Helper()
var st syscall.Stat_t
err := syscall.Lstat(path, &st)
if err != nil {
t.Fatal(err)
}
return st.Blocks * 512
}
// pattern returns n bytes of deterministic content seeded by tag. // pattern returns n bytes of deterministic content seeded by tag.
func pattern(tag byte, n int) []byte { func pattern(tag byte, n int) []byte {
data := make([]byte, n) data := make([]byte, n)
@@ -1416,6 +1431,46 @@ func TestScanStoresRawHashes(t *testing.T) {
} }
} }
// TestScanStoresDiskSize scans a sparse file, which takes less space on
// disk than its length, and checks the disk_size stored for it against
// the file's own lstat.
func TestScanStoresDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "sparse", []byte("x"))
// Truncate makes the file longer without giving it blocks for the
// new bytes.
const length = 1024 * 1024
err := os.Truncate(p, length)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
want := diskSizeOf(t, p)
if want >= length {
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
"length %d", want, length)
}
var got int64
err = db.QueryRowContext(t.Context(),
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
if err != nil {
t.Fatal(err)
}
if got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same // TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the // size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file. The operand is // next scan must notice the change and re-hash the file. The operand is
+13 -6
View File
@@ -22,6 +22,10 @@ type treeNode struct {
digest [sha256.Size]byte digest [sha256.Size]byte
fileCount int64 fileCount int64
totalSize int64 totalSize int64
// diskSize is the bytes the directory's files take on disk. Two
// copies of one tree have the same totalSize, but their diskSize
// can differ.
diskSize int64
} }
// treesPhase names one phase of trees with its place among all of them, // treesPhase names one phase of trees with its place among all of them,
@@ -139,15 +143,16 @@ func readTree(ctx context.Context, db *sql.DB,
// writeTreeGroups writes dupes to stdout as TSV, a header line and then // writeTreeGroups writes dupes to stdout as TSV, a header line and then
// a row for every member of each group but its first, counting each // a row for every member of each group but its first, counting each
// group on prog once its rows are written. stdout may be the terminal // group on prog once its rows are written. The disk_size column is the
// prog is drawn on, so the lines go through prog's writer. It returns // dupe's own. stdout may be the terminal prog is drawn on, so the lines
// the number of rows (dupe trees) and the bytes they could reclaim. // go through prog's writer. It returns the number of rows (dupe trees)
// and the bytes they could reclaim.
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode, func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
prog *progress, prog *progress,
) (int, int64, error) { ) (int, int64, error) {
out := prog.writer(stdout) out := prog.writer(stdout)
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize") _, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
if err != nil { if err != nil {
return 0, 0, fmt.Errorf("write stdout: %w", err) return 0, 0, fmt.Errorf("write stdout: %w", err)
} }
@@ -159,9 +164,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
for _, g := range dupes { for _, g := range dupes {
first := g[0] first := g[0]
for _, n := range g[1:] { for _, n := range g[1:] {
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n", _, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
escapePath(first.path), escapePath(n.path), escapePath(first.path), escapePath(n.path),
first.fileCount, first.totalSize) first.fileCount, first.totalSize, n.diskSize)
if err != nil { if err != nil {
return 0, 0, fmt.Errorf("write stdout: %w", err) return 0, 0, fmt.Errorf("write stdout: %w", err)
} }
@@ -232,6 +237,7 @@ func (b *treeBuilder) add(r scanRec) {
dir.entries = append(dir.entries, fileEntry(name, r)) dir.entries = append(dir.entries, fileEntry(name, r))
dir.fileCount++ dir.fileCount++
dir.totalSize += r.size dir.totalSize += r.size
dir.diskSize += r.diskSize
} }
// openDir starts the directory called name inside the innermost open // openDir starts the directory called name inside the innermost open
@@ -270,6 +276,7 @@ func (b *treeBuilder) closeTo(n int) {
"d\x00"+name+"\x00"+string(dir.digest[:])) "d\x00"+name+"\x00"+string(dir.digest[:]))
dir.parent.fileCount += dir.fileCount dir.parent.fileCount += dir.fileCount
dir.parent.totalSize += dir.totalSize dir.parent.totalSize += dir.totalSize
dir.parent.diskSize += dir.diskSize
b.dirs = append(b.dirs, dir) b.dirs = append(b.dirs, dir)
} }
} }
+40 -11
View File
@@ -17,18 +17,24 @@ const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and // smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename. // /d/t2 are identical, /d/t3 differs from them only by one filename.
// Each file takes one 4 KiB block on disk.
func smokeTreeRecs() []scanRec { func smokeTreeRecs() []scanRec {
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c") f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c") f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
return []scanRec{ return []scanRec{
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"}, {size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"}, content: f1Content, path: "/d/t1/f1"},
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"}, {size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"}, content: f2Content, path: "/d/t1/sub/f2"},
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"}, {size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, content: f1Content, path: "/d/t2/f1"},
path: "/d/t3/sub/f2renamed"}, {size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
content: f2Content, path: "/d/t2/sub/f2"},
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
content: f1Content, path: "/d/t3/f1"},
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
content: f2Content, path: "/d/t3/sub/f2renamed"},
} }
} }
@@ -198,13 +204,33 @@ func TestRunTreesEscapesPaths(t *testing.T) {
code, exitOK, stderr.String()) code, exitOK, stderr.String())
} }
want := "first\tdupe\tfiles\tsize\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n" `/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
} }
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
// take different space on disk. The row's disk_size is the sum over the
// dupe copy's files, those in its subdirectory included.
func TestRunTreesDiskSize(t *testing.T) {
f, g := testHash("f"), testHash("g")
recs := []scanRec{
{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
}
t.Setenv(databaseEnv, seedDatabase(t, recs))
want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
if got := runStdout(t, cmdTrees); got != want {
t.Errorf("stdout = %q, want %q", got, want)
}
}
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test // TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
// records with stderr not a terminal, and checks every line it prints // records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line // there. After the line naming the database, each phase prints a line
@@ -226,7 +252,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
code, exitOK, stderr()) code, exitOK, stderr())
} }
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n" report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
"/d/t1\t/d/t2\t2\t3100\t8192\n"
if got := stdout.String(); got != report { if got := stdout.String(); got != report {
t.Errorf("stdout = %q, want %q", got, report) t.Errorf("stdout = %q, want %q", got, report)
} }
@@ -291,7 +318,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:]) shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
} }
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"} want := []string{
"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
}
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) { if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want) t.Errorf("terminal shows lines %q, want %q first", shown, want)
} }