Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The files table gets a disk_size column: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes. trees adds it up per directory and prints it as a column after size. On a compressed or sparse file it is less than size, so two copies of one tree can differ; each row shows the dupe copy's. Judgement call: a row's disk_size is the dupe copy's, not the first's. Unchanged files keep the disk_size from their last hash: a change in allocation that leaves size and mtime alone is not picked up. Model: opus-5-5
This commit is contained in:
@@ -33,14 +33,14 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
```
|
||||
|
||||
`scan` walks one or more filesystem trees and maintains one database record per
|
||||
regular file (path, size, mtime, head hash, tail hash, content hash). The
|
||||
database persists between runs. A rescan hashes only the files that are new or
|
||||
changed, and removes the records of files that no longer exist. `report` reads
|
||||
the database and prints the file-level duplicates report. `trees` reads the same
|
||||
database and prints the duplicate-tree report. `info` prints the database's
|
||||
details: its path, the sizes of its files and the number of its records. A
|
||||
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
|
||||
prints a usage message and exits 2.
|
||||
regular file (path, size, size on disk, mtime, head hash, tail hash, content
|
||||
hash). The database persists between runs. A rescan hashes only the files that
|
||||
are new or changed, and removes the records of files that no longer exist.
|
||||
`report` reads the database and prints the file-level duplicates report. `trees`
|
||||
reads the same database and prints the duplicate-tree report. `info` prints the
|
||||
database's details: its path, the sizes of its files and the number of its
|
||||
records. A missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||||
operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
@@ -386,6 +386,7 @@ All four subcommands operate on a single SQLite database file:
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||||
size INTEGER NOT NULL, -- bytes, from lstat
|
||||
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512), from lstat
|
||||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
|
||||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||||
@@ -552,7 +553,7 @@ hash phase:
|
||||
files; per-directory parallelism is what makes the walk tractable on large
|
||||
or busy pools. The walk compares each file with its record as it goes, and
|
||||
an unchanged file needs nothing more. Every new or changed file is carried
|
||||
to the hash phase as a (path, size, mtime) record.
|
||||
to the hash phase as a (path, size, size on disk, mtime) record.
|
||||
3. **hash** — the worker pool hashes every carried file as described in
|
||||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||||
`content`. Zero-length files have constant hashes and are never opened.
|
||||
@@ -809,14 +810,26 @@ Processing:
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
group of N). `files` is the recursive regular-file count of one copy of the
|
||||
tree; `size` is the recursive total byte size of one copy:
|
||||
group of N). After the `first` and `dupe` paths come three numbers:
|
||||
|
||||
- `files`, the number of regular files in one copy of the tree, counted
|
||||
recursively;
|
||||
- `size`, the bytes of one copy, its files' sizes added up;
|
||||
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||||
added up.
|
||||
|
||||
```
|
||||
first dupe files size
|
||||
/srv/a/project /srv/backup/project 3417 104857600
|
||||
first dupe files size disk_size
|
||||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||
```
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It is less than the
|
||||
file's size on a compressed or sparse file, and a little more on most other
|
||||
files. Two copies of one tree can therefore differ in `disk_size`. When a file
|
||||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||
full. Removing it from one copy then frees nothing, as the known limitation
|
||||
under "`trees` mode" says.
|
||||
|
||||
The `first` and `dupe` paths are escaped as described under "Report output
|
||||
format". The root directory's path is `/`.
|
||||
|
||||
@@ -1200,9 +1213,9 @@ All of the following, run in this directory, must pass:
|
||||
ordered by size descending.
|
||||
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||||
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
||||
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
||||
differs by name).
|
||||
files, 3100 bytes, and a `disk_size` that depends on the filesystem.
|
||||
`$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the
|
||||
`t1`/`t2` group), and `t3` appears nowhere (its file set differs by name).
|
||||
|
||||
Expected from the second `report` (after the modify/delete rescan):
|
||||
`one.bin` has left its group (its content changed), so
|
||||
|
||||
@@ -29,6 +29,12 @@
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- each record holds `disk_size`, the bytes the file takes on disk: its allocated
|
||||
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
|
||||
prints a `disk_size` column after `size`, the sum over the `dupe` copy's
|
||||
files; on a compressed or sparse file it differs from `size` (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||
|
||||
- the database keeps the number of records and the sum of their sizes in the one
|
||||
row of a `totals` table, which three triggers on `files` keep current. `info`
|
||||
reads that row, and so do `scan`'s `load` phase and the `read` phases of
|
||||
|
||||
@@ -47,14 +47,15 @@ const dbFilePerm = 0o644
|
||||
const lockFilePerm = 0o600
|
||||
|
||||
// createTableSQL is the schema applied to a fresh database. Paths are
|
||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
|
||||
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
|
||||
// second. head, tail and content each hold the raw 32 bytes of a
|
||||
// SHA-256.
|
||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
|
||||
// disk_size holds the bytes the file takes on disk. mtime holds whole
|
||||
// Unix seconds and mtime_nsec the nanoseconds within that second. head,
|
||||
// tail and content each hold the raw 32 bytes of a SHA-256.
|
||||
const createTableSQL = `
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY,
|
||||
size INTEGER NOT NULL,
|
||||
disk_size INTEGER NOT NULL,
|
||||
mtime INTEGER NOT NULL,
|
||||
mtime_nsec INTEGER NOT NULL,
|
||||
head BLOB NOT NULL,
|
||||
@@ -115,11 +116,13 @@ SELECT records FROM totals
|
||||
// upsertSQL inserts one file record, replacing any existing record for
|
||||
// the same path.
|
||||
const upsertSQL = `
|
||||
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO files (
|
||||
path, size, disk_size, mtime, mtime_nsec, head, tail, content
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (path) DO UPDATE SET
|
||||
size = excluded.size, mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size, disk_size = excluded.disk_size,
|
||||
mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
|
||||
head = excluded.head, tail = excluded.tail,
|
||||
content = excluded.content
|
||||
`
|
||||
@@ -536,8 +539,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
total(n)
|
||||
|
||||
rows, err := tx.QueryContext(ctx,
|
||||
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
|
||||
"FROM files ORDER BY path")
|
||||
"SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
|
||||
"content FROM files ORDER BY path")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
@@ -551,7 +554,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
r scanRec
|
||||
)
|
||||
|
||||
err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content)
|
||||
err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
|
||||
&head, &tail, &content)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read record: %w", err)
|
||||
}
|
||||
@@ -586,10 +590,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
|
||||
`
|
||||
|
||||
// loadSignatureRows gives total the number of records, then streams
|
||||
// every record's signature and path to fn, without its mtime, in the
|
||||
// order of signatureRowsSQL. The count and the records are read in one
|
||||
// transaction, so they agree while a scan is committing. An error from
|
||||
// fn stops the reading and is returned as it is.
|
||||
// every record's signature and path to fn, without its mtime or its
|
||||
// disk_size, in the order of signatureRowsSQL. The count and the
|
||||
// records are read in one transaction, so they agree while a scan is
|
||||
// committing. An error from fn stops the reading and is returned as it
|
||||
// is.
|
||||
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||
total func(n int64), fn func(r scanRec) error,
|
||||
) error {
|
||||
@@ -833,7 +838,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
||||
defer func() { _ = st.Close() }()
|
||||
|
||||
for _, r := range upserts {
|
||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size,
|
||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
|
||||
r.mtime.Unix(), r.mtime.Nanosecond(),
|
||||
r.head[:], r.tail[:], r.content[:])
|
||||
if err != nil {
|
||||
|
||||
+9
-6
@@ -1141,8 +1141,9 @@ func TestRunTreesSucceeds(t *testing.T) {
|
||||
|
||||
// The two directories holding the duplicate pair are duplicate
|
||||
// trees of each other.
|
||||
want := "first\tdupe\tfiles\tsize\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
@@ -1176,9 +1177,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
cmdReport: "first\tdupe\tsize\n" +
|
||||
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
||||
cmdTrees: "first\tdupe\tfiles\tsize\n" +
|
||||
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
||||
"\t1\t300\n",
|
||||
"\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
|
||||
"\n",
|
||||
}
|
||||
|
||||
for name, want := range cases {
|
||||
@@ -1213,8 +1215,9 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
||||
t.Errorf("report stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
want = "first\tdupe\tfiles\tsize\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||
if got := runStdout(t, cmdTrees); got != want {
|
||||
t.Errorf("trees stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
@@ -21,14 +21,16 @@ const minGroupSize = 2
|
||||
|
||||
// scanRec is one file record from the database. The signature (size,
|
||||
// head, tail, content) is the duplicate key; mtime is informational
|
||||
// only and used by scan for change detection.
|
||||
// only and used by scan for change detection. diskSize is the bytes the
|
||||
// file takes on disk, which trees adds up.
|
||||
type scanRec struct {
|
||||
size int64
|
||||
mtime time.Time
|
||||
head [sha256.Size]byte
|
||||
tail [sha256.Size]byte
|
||||
content [sha256.Size]byte
|
||||
path string
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
head [sha256.Size]byte
|
||||
tail [sha256.Size]byte
|
||||
content [sha256.Size]byte
|
||||
path string
|
||||
}
|
||||
|
||||
// reportPhase names one phase of report with its place among all of
|
||||
|
||||
+2
-2
@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
|
||||
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
||||
|
||||
return []scanRec{
|
||||
{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||
{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -52,15 +52,17 @@ const workQueueDepth = 1024
|
||||
// has already printed its line, so run prints nothing more.
|
||||
var errInterrupted = errors.New("scan interrupted")
|
||||
|
||||
// fileRec carries one statted file between the scan phases. dev and
|
||||
// ino identify the underlying inode so hard-linked paths can share
|
||||
// one read; both are zero when the platform exposes no inode.
|
||||
// fileRec carries one statted file between the scan phases. diskSize is
|
||||
// the bytes the file takes on disk. dev and ino identify the underlying
|
||||
// inode so hard-linked paths can share one read; both are zero when the
|
||||
// platform exposes no inode.
|
||||
type fileRec struct {
|
||||
path string
|
||||
size int64
|
||||
mtime time.Time
|
||||
dev uint64
|
||||
ino uint64
|
||||
path string
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
dev uint64
|
||||
ino uint64
|
||||
}
|
||||
|
||||
// fileMeta is the in-memory index entry for one existing database
|
||||
@@ -695,12 +697,13 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
|
||||
s.resolve(rec.path)
|
||||
|
||||
s.batch = append(s.batch, scanRec{
|
||||
size: rec.size,
|
||||
mtime: rec.mtime,
|
||||
head: r.head,
|
||||
tail: r.tail,
|
||||
content: r.content,
|
||||
path: rec.path,
|
||||
size: rec.size,
|
||||
diskSize: rec.diskSize,
|
||||
mtime: rec.mtime,
|
||||
head: r.head,
|
||||
tail: r.tail,
|
||||
content: r.content,
|
||||
path: rec.path,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -961,11 +964,12 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
||||
|
||||
dev, ino := inodeOfInfo(fi)
|
||||
rec := fileRec{
|
||||
path: root,
|
||||
size: fi.Size(),
|
||||
mtime: fi.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
path: root,
|
||||
size: fi.Size(),
|
||||
diskSize: diskSizeOfInfo(fi),
|
||||
mtime: fi.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
}
|
||||
|
||||
if !isDatabaseFile(rec, rules) {
|
||||
@@ -1145,11 +1149,12 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
|
||||
dev, ino := inodeOfInfo(info)
|
||||
rec := fileRec{
|
||||
path: p,
|
||||
size: info.Size(),
|
||||
mtime: info.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
path: p,
|
||||
size: info.Size(),
|
||||
diskSize: diskSizeOfInfo(info),
|
||||
mtime: info.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
}
|
||||
|
||||
if !isDatabaseFile(rec, rules) {
|
||||
@@ -1241,6 +1246,23 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
|
||||
return statDev(st), st.Ino
|
||||
}
|
||||
|
||||
// statBlockSize is the unit of st_blocks in an lstat result on Linux and
|
||||
// macOS, whatever the filesystem's own block size.
|
||||
const statBlockSize = 512
|
||||
|
||||
// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
|
||||
// blocks, st_blocks × 512. On a compressed or sparse file it is less
|
||||
// than the file's size. It is 0 when the platform exposes no block
|
||||
// count.
|
||||
func diskSizeOfInfo(fi fs.FileInfo) int64 {
|
||||
st, ok := fi.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
|
||||
return st.Blocks * statBlockSize
|
||||
}
|
||||
|
||||
// hashResult carries the hashes computed for one inode run (or the
|
||||
// error that prevented computing them) from the pool's workers to the
|
||||
// hash phase. Its run leaves out the paths found removed.
|
||||
|
||||
@@ -42,6 +42,21 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
||||
return p
|
||||
}
|
||||
|
||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||
// own lstat: st_blocks counts 512-byte units.
|
||||
func diskSizeOf(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
var st syscall.Stat_t
|
||||
|
||||
err := syscall.Lstat(path, &st)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return st.Blocks * 512
|
||||
}
|
||||
|
||||
// pattern returns n bytes of deterministic content seeded by tag.
|
||||
func pattern(tag byte, n int) []byte {
|
||||
data := make([]byte, n)
|
||||
@@ -1416,6 +1431,46 @@ func TestScanStoresRawHashes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestScanStoresDiskSize scans a sparse file, which takes less space on
|
||||
// disk than its length, and checks the disk_size stored for it against
|
||||
// the file's own lstat.
|
||||
func TestScanStoresDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
p := writeFile(t, dir, "sparse", []byte("x"))
|
||||
|
||||
// Truncate makes the file longer without giving it blocks for the
|
||||
// new bytes.
|
||||
const length = 1024 * 1024
|
||||
|
||||
err := os.Truncate(p, length)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
|
||||
"length %d", want, length)
|
||||
}
|
||||
|
||||
var got int64
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
||||
// size with an mtime later in the same second as the recorded one: the
|
||||
// next scan must notice the change and re-hash the file. The operand is
|
||||
|
||||
@@ -22,6 +22,10 @@ type treeNode struct {
|
||||
digest [sha256.Size]byte
|
||||
fileCount int64
|
||||
totalSize int64
|
||||
// diskSize is the bytes the directory's files take on disk. Two
|
||||
// copies of one tree have the same totalSize, but their diskSize
|
||||
// can differ.
|
||||
diskSize int64
|
||||
}
|
||||
|
||||
// treesPhase names one phase of trees with its place among all of them,
|
||||
@@ -139,15 +143,16 @@ func readTree(ctx context.Context, db *sql.DB,
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
// a row for every member of each group but its first, counting each
|
||||
// group on prog once its rows are written. stdout may be the terminal
|
||||
// prog is drawn on, so the lines go through prog's writer. It returns
|
||||
// the number of rows (dupe trees) and the bytes they could reclaim.
|
||||
// group on prog once its rows are written. The disk_size column is the
|
||||
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
||||
// go through prog's writer. It returns the number of rows (dupe trees)
|
||||
// and the bytes they could reclaim.
|
||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
prog *progress,
|
||||
) (int, int64, error) {
|
||||
out := prog.writer(stdout)
|
||||
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
@@ -159,9 +164,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
for _, g := range dupes {
|
||||
first := g[0]
|
||||
for _, n := range g[1:] {
|
||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
|
||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
|
||||
escapePath(first.path), escapePath(n.path),
|
||||
first.fileCount, first.totalSize)
|
||||
first.fileCount, first.totalSize, n.diskSize)
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
@@ -232,6 +237,7 @@ func (b *treeBuilder) add(r scanRec) {
|
||||
dir.entries = append(dir.entries, fileEntry(name, r))
|
||||
dir.fileCount++
|
||||
dir.totalSize += r.size
|
||||
dir.diskSize += r.diskSize
|
||||
}
|
||||
|
||||
// openDir starts the directory called name inside the innermost open
|
||||
@@ -270,6 +276,7 @@ func (b *treeBuilder) closeTo(n int) {
|
||||
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
||||
dir.parent.fileCount += dir.fileCount
|
||||
dir.parent.totalSize += dir.totalSize
|
||||
dir.parent.diskSize += dir.diskSize
|
||||
b.dirs = append(b.dirs, dir)
|
||||
}
|
||||
}
|
||||
|
||||
+40
-11
@@ -17,18 +17,24 @@ const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
||||
|
||||
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
|
||||
// /d/t2 are identical, /d/t3 differs from them only by one filename.
|
||||
// Each file takes one 4 KiB block on disk.
|
||||
func smokeTreeRecs() []scanRec {
|
||||
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
||||
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
||||
|
||||
return []scanRec{
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
|
||||
path: "/d/t3/sub/f2renamed"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t1/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t1/sub/f2"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t2/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t2/sub/f2"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t3/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t3/sub/f2renamed"},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -198,13 +204,33 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
want := "first\tdupe\tfiles\tsize\n" +
|
||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
func TestRunTreesDiskSize(t *testing.T) {
|
||||
f, g := testHash("f"), testHash("g")
|
||||
recs := []scanRec{
|
||||
{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
|
||||
{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
|
||||
{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
|
||||
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
||||
}
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
|
||||
if got := runStdout(t, cmdTrees); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
||||
// records with stderr not a terminal, and checks every line it prints
|
||||
// there. After the line naming the database, each phase prints a line
|
||||
@@ -226,7 +252,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
|
||||
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
"/d/t1\t/d/t2\t2\t3100\t8192\n"
|
||||
if got := stdout.String(); got != report {
|
||||
t.Errorf("stdout = %q, want %q", got, report)
|
||||
}
|
||||
@@ -291,7 +318,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||
}
|
||||
|
||||
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
|
||||
want := []string{
|
||||
"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
|
||||
}
|
||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user