1 Commits
Author SHA1 Message Date
sneak dfbfdd0601 Record each file's size on disk and print it in trees (closes #147)
check / check (push) Canceled after 0s
The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. trees adds it up per directory and prints
it as a column after size. On a compressed or sparse file it is less
than size, so two copies of one tree can differ; each row shows the
dupe copy's.

Judgement call: a row's disk_size is the dupe copy's, not the first's.
Unchanged files keep the disk_size from their last hash: a change in
allocation that leaves size and mtime alone is not picked up.

Model: opus-5-5
2026-10-27 18:08:27 +00:00
6 changed files with 90 additions and 255 deletions
+19 -27
View File
@@ -226,12 +226,12 @@ Goals, in order:
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below). its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
Therefore no file of 100 MiB or more is read in full. Scale target: tens of Therefore no file of 100 MiB or more is read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
under resilver). Holding one small record (path, size, size on disk, mtime) under resilver). Holding one small record (path, size, mtime) per file in
per file in memory during a scan is acceptable; holding every file's hashes memory during a scan is acceptable; holding every file's hashes is not (they
is not (they stay in the database). The reporting commands do not hold every stay in the database). The reporting commands do not hold every file's
file's hashes either: `report` reads the records in signature order and hashes either: `report` reads the records in signature order and holds the
holds the duplicate paths of one size at a time, so its memory grows with duplicate paths of one size at a time, so its memory grows with the
the duplicates of the size that has the most of them, and `trees` reads the duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals, records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the being read, so its memory grows with the number of directories and with the
@@ -368,8 +368,7 @@ All four subcommands operate on a single SQLite database file:
- While it loads the existing records (`load`) and walks: no records. A - While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run. first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time. - During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, the new `disk_size` - In the update phase: the rest of the hashed records, and the deletions.
of unchanged files whose size on disk changed, and the deletions.
- At the end, as it closes the database: the `-wal` and `-shm` files go - At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above). away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read - `report`, `trees` and `info` open the database read-only and need only read
@@ -516,11 +515,6 @@ operands:
- A file already in the database is **skipped without reading its contents** - A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap. than the recorded mtime. This is what makes a daily rescan cheap.
- When such a file's size on disk differs from the recorded `disk_size`, the
scan updates `disk_size` alone. It still does not read the file, and counts it
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
the write. A file scanned within those seconds gets its real `disk_size` at
the next scan.
- A file whose mtime is newer than recorded, or whose size differs, is hashed - A file whose mtime is newer than recorded, or whose size differs, is hashed
again. Change detection compares the mtime to the nanosecond, as finely as the again. Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the filesystem records it, so a same-size rewrite counts as a change whenever the
@@ -548,8 +542,8 @@ inside the walk and hash phases, and batched database writes begin during the
hash phase: hash phase:
1. **load** — read the existing records. Each record under the operands is kept 1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size, its size on disk and its mtime, for the in memory, by path, with its size and its mtime, for the walk to compare
walk to compare each file with. each file with.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently 2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back worker reads one directory at a time, handing discovered subdirectories back
@@ -557,10 +551,9 @@ hash phase:
(while the directory's metadata is still hot). Sequential directory (while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes. An or busy pools. The walk compares each file with its record as it goes, and
unchanged file is not read again. When its size on disk has changed, the an unchanged file needs nothing more. Every new or changed file is carried
update phase writes its new `disk_size`. Every new or changed file is to the hash phase as a (path, size, size on disk, mtime) record.
carried to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in 3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and "Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened. `content`. Zero-length files have constant hashes and are never opened.
@@ -572,8 +565,7 @@ hash phase:
committed in batched transactions **while hashing runs**. A scan interrupted committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written. resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records, and the new 4. **update** — commit the final partial batch of hashed records. Then delete
`disk_size` of each unchanged file whose size on disk changed. Then delete
the records the scan did not verify: those of vanished files, and of paths the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash. that failed to stat or hash.
@@ -831,12 +823,12 @@ first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984 /srv/a/project /srv/backup/project 3417 104857600 61865984
``` ```
A file's `disk_size` is the space its blocks take on disk. It can be more than A file's `disk_size` is the space its blocks take on disk. It is less than the
the file's size, because space is allocated in whole blocks. It is less than the file's size on a compressed or sparse file, and a little more on most other
file's size on a compressed or sparse file, so two copies of one tree can differ files. Two copies of one tree can therefore differ in `disk_size`. When a file
in `disk_size`. When a file in one copy is a hard link to one in the other, both in one copy is a hard link to one in the other, both copies count its blocks in
copies count its blocks in full. Removing it from one copy then frees nothing, full. Removing it from one copy then frees nothing, as the known limitation
as the known limitation under "`trees` mode" says. under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`. format". The root directory's path is `/`.
+2 -3
View File
@@ -32,9 +32,8 @@
- each record holds `disk_size`, the bytes the file takes on disk: its allocated - each record holds `disk_size`, the bytes the file takes on disk: its allocated
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees` blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
prints a `disk_size` column after `size`, the sum over the `dupe` copy's prints a `disk_size` column after `size`, the sum over the `dupe` copy's
files; on a compressed or sparse file it differs from `size`. A rescan updates files; on a compressed or sparse file it differs from `size` (2026-10-27,
the `disk_size` of an unchanged file whose size on disk changed, without https://git.eeqj.de/sneak/sfdupes/issues/147)
reading the file (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/147)
- the database keeps the number of records and the sum of their sizes in the one - the database keeps the number of records and the sum of their sizes in the one
row of a `totals` table, which three triggers on `files` keep current. `info` row of a `totals` table, which three triggers on `files` keep current. `info`
+18 -67
View File
@@ -729,15 +729,15 @@ func readDatabaseCounts(ctx context.Context,
return c, nil return c, nil
} }
// loadFileMeta streams every record's path, size, disk_size and mtime // loadFileMeta streams every record's path, size and mtime to fn. Scan
// to fn. Scan change detection needs no hash values, and skipping the // change detection needs no hash values, and skipping the hash columns
// hash columns keeps the scan's in-memory index small on // keeps the scan's in-memory index small on multi-million-file
// multi-million-file databases. // databases.
func loadFileMeta(ctx context.Context, db *sql.DB, func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, m fileMeta), fn func(path string, size int64, mtime time.Time),
) error { ) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, disk_size, mtime, mtime_nsec FROM files") "SELECT path, size, mtime, mtime_nsec FROM files")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -746,19 +746,16 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
m fileMeta size, sec, nsec int64
sec, nsec int64
) )
err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec) err = rows.Scan(&path, &size, &sec, &nsec)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
m.mtime = time.Unix(sec, nsec) fn(string(path), size, time.Unix(sec, nsec))
fn(string(path), m)
} }
err = rows.Err() err = rows.Err()
@@ -776,21 +773,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// WAL small and let concurrent reports observe progress. // WAL small and let concurrent reports observe progress.
const updateBatchSize = 10000 const updateBatchSize = 10000
// applyChanges writes upserts for new and changed files and deletes for // applyChanges writes one scan's database changes — upserts for new and
// vanished ones, in batched transactions. Progress is rendered on prog // changed files, deletes for vanished ones — in batched transactions.
// (one increment per change). // Progress is rendered on prog (one increment per change).
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec, func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
deletes []string, prog *progress, deletes []string, prog *progress,
) error { ) error {
for batch := range slices.Chunk(upserts, updateBatchSize) { for batch := range slices.Chunk(upserts, updateBatchSize) {
err := applyBatch(ctx, db, batch, nil, nil, prog) err := applyBatch(ctx, db, batch, nil, prog)
if err != nil { if err != nil {
return err return err
} }
} }
for batch := range slices.Chunk(deletes, updateBatchSize) { for batch := range slices.Chunk(deletes, updateBatchSize) {
err := applyBatch(ctx, db, nil, nil, batch, prog) err := applyBatch(ctx, db, nil, batch, prog)
if err != nil { if err != nil {
return err return err
} }
@@ -799,26 +796,10 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
return nil return nil
} }
// applyDiskSizes writes the disk_size of each of recs into its record, // applyBatch commits one batch of upserts and deletes in a single
// in batched transactions, and leaves the rest of the record alone. // transaction.
// Progress is rendered on prog (one increment per record).
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
prog *progress,
) error {
for batch := range slices.Chunk(recs, updateBatchSize) {
err := applyBatch(ctx, db, nil, batch, nil, prog)
if err != nil {
return err
}
}
return nil
}
// applyBatch commits one batch of upserts, disk_size updates and
// deletes in a single transaction.
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec, func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
diskSizes []fileRec, deletes []string, prog *progress, deletes []string, prog *progress,
) error { ) error {
tx, err := db.BeginTx(ctx, nil) tx, err := db.BeginTx(ctx, nil)
if err != nil { if err != nil {
@@ -832,11 +813,6 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
return err return err
} }
err = execDiskSizes(ctx, tx, diskSizes, prog)
if err != nil {
return err
}
err = execDeletes(ctx, tx, deletes, prog) err = execDeletes(ctx, tx, deletes, prog)
if err != nil { if err != nil {
return err return err
@@ -875,31 +851,6 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
return nil return nil
} }
// execDiskSizes sets the disk_size of the record of each of recs. It
// does not touch size, so the totals triggers do not fire.
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
prog *progress,
) error {
st, err := tx.PrepareContext(ctx,
"UPDATE files SET disk_size = ? WHERE path = ?")
if err != nil {
return fmt.Errorf("prepare disk_size update: %w", err)
}
defer func() { _ = st.Close() }()
for _, r := range recs {
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
if err != nil {
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
}
prog.increment()
}
return nil
}
// execDeletes removes the records for paths no longer present. // execDeletes removes the records for paths no longer present.
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string, func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
prog *progress, prog *progress,
+4 -23
View File
@@ -818,23 +818,6 @@ func scanFixture(t *testing.T) []string {
return dupes return dupes
} }
// storedDiskSize returns the disk_size of the record for path in the
// database at dbPath. A test of a trees run compares with this rather
// than with a new lstat, which on ZFS can differ from the scan's (see
// diskSizeOf).
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
t.Helper()
db, err := openReportDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
return recordByPath(t, dbRecords(t, db), path).diskSize
}
// scanOK runs scan over operands, fails the test unless it exits 0 with // scanOK runs scan over operands, fails the test unless it exits 0 with
// nothing on stdout, and returns everything it printed to stderr. // nothing on stdout, and returns everything it printed to stderr.
func scanOK(t *testing.T, operands ...string) string { func scanOK(t *testing.T, operands ...string) string {
@@ -1160,7 +1143,7 @@ func TestRunTreesSucceeds(t *testing.T) {
// trees of each other. // trees of each other.
want := "first\tdupe\tfiles\tsize\tdisk_size\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n" strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -1196,8 +1179,8 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
dupes[0] + "\t" + dupes[1] + "\t300\n", dupes[0] + "\t" + dupes[1] + "\t300\n",
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" + cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
"\t1\t300\t" + "\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n", "\n",
} }
for name, want := range cases { for name, want := range cases {
@@ -1232,11 +1215,9 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Errorf("report stdout = %q, want %q", got, want) t.Errorf("report stdout = %q, want %q", got, want)
} }
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
want = "first\tdupe\tfiles\tsize\tdisk_size\n" + want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSize, 10) + "\n" strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
if got := runStdout(t, cmdTrees); got != want { if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want) t.Errorf("trees stdout = %q, want %q", got, want)
} }
+27 -42
View File
@@ -66,12 +66,11 @@ type fileRec struct {
} }
// fileMeta is the in-memory index entry for one existing database // fileMeta is the in-memory index entry for one existing database
// record: what the walk compares the file with. Hashes stay on disk; at // record: just enough for change detection. Hashes stay on disk; at
// tens of millions of records they would dominate the scan's memory. // tens of millions of records they would dominate the scan's memory.
type fileMeta struct { type fileMeta struct {
size int64 size int64
diskSize int64 mtime time.Time
mtime time.Time
} }
// scanUnit is what every scan phase counts, as its progress display // scanUnit is what every scan phase counts, as its progress display
@@ -342,12 +341,11 @@ type scanStats struct {
// whatever remains after the walk and hash phases is deleted by the // whatever remains after the walk and hash phases is deleted by the
// update phase. // update phase.
type scanState struct { type scanState struct {
db *sql.DB db *sql.DB
existing map[string]fileMeta existing map[string]fileMeta
toHash []fileRec // new and changed files: must be read toHash []fileRec // new and changed files: must be read
diskSizes []fileRec // unchanged files whose disk_size changed batch []scanRec
batch []scanRec st scanStats
st scanStats
} }
// syncScan synchronizes the database with the filesystem under roots; // syncScan synchronizes the database with the filesystem under roots;
@@ -476,24 +474,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta) s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db, func(path string, m fileMeta) { return loadFileMeta(ctx, s.db,
prog.increment() func(path string, size int64, mtime time.Time) {
prog.increment()
if underAnyRoot(path, roots) { if underAnyRoot(path, roots) {
s.existing[path] = m s.existing[path] = fileMeta{size: size, mtime: mtime}
} }
}) })
} }
// walkPhase drains the walk and queues every new or changed file for // walkPhase drains the walk and queues every new or changed file for
// the hash phase. An unchanged file is not read, and its record counts // the hash phase. An unchanged file needs nothing more, and its record
// as verified. When its disk_size differs from the record's, it is // counts as verified. A record at or below a path the walk left out,
// queued for the update phase, which writes the new disk_size. A record // because -x stopped at it, it is a default exclude, or it is not on a
// at or below a path the walk left out, because -x stopped at it, it is // persistent filesystem, counts as outside the roots: it is not
// a default exclude, or it is not on a persistent filesystem, counts as // deleted. That does not hold below a root at or inside that path,
// outside the roots: it is not deleted. That does not hold below a root // since that root's own walk verifies the records below it; nor is a
// at or inside that path, since that root's own walk verifies the // line printed for such a root.
// records below it; nor is a line printed for such a root.
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) { func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The records the last scan left under the roots estimate how many // The records the last scan left under the roots estimate how many
// files the walk will find. A first scan has none, and no estimate. // files the walk will find. A first scan has none, and no estimate.
@@ -551,13 +549,6 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
delete(s.existing, ev.rec.path) delete(s.existing, ev.rec.path)
s.st.unchanged++ s.st.unchanged++
// The space a file takes on disk can change while its size and
// mtime stay the same. ZFS, for one, counts a new file's blocks
// only seconds after the write.
if old.diskSize != ev.rec.diskSize {
s.diskSizes = append(s.diskSizes, ev.rec)
}
} }
prog.finish() prog.finish()
@@ -728,7 +719,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
return nil return nil
} }
err := applyBatch(ctx, s.db, s.batch, nil, nil, nil) err := applyBatch(ctx, s.db, s.batch, nil, nil)
if err != nil { if err != nil {
return err return err
} }
@@ -739,10 +730,9 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
} }
// updatePhase writes the scan's tail under one progress display: the // updatePhase writes the scan's tail under one progress display: the
// final partial batch of hashed records, the new disk_size of every // final partial batch of hashed records, and deletions for every record
// unchanged file whose disk_size changed, and deletions for every // the scan did not verify (vanished files, plus paths that failed to
// record the scan did not verify (vanished files, plus paths that // stat or hash).
// failed to stat or hash).
func (s *scanState) updatePhase(ctx context.Context) error { func (s *scanState) updatePhase(ctx context.Context) error {
deletes := make([]string, 0, len(s.existing)) deletes := make([]string, 0, len(s.existing))
for path := range s.existing { for path := range s.existing {
@@ -754,7 +744,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.st.removed = len(deletes) s.st.removed = len(deletes)
total := len(s.batch) + len(s.diskSizes) + len(deletes) total := len(s.batch) + len(deletes)
prog := newProgress(progressParams{ prog := newProgress(progressParams{
label: scanPhase("update"), unit: scanUnit, total: int64(total), label: scanPhase("update"), unit: scanUnit, total: int64(total),
stalledOn: "db", stalledOn: "db",
@@ -769,11 +759,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.batch = nil s.batch = nil
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
if err != nil {
return err
}
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
+20 -93
View File
@@ -43,10 +43,7 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
} }
// diskSizeOf returns the bytes the file at path takes on disk, from its // diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's // own lstat: st_blocks counts 512-byte units.
// blocks only seconds after the write, and fsync does not hasten that.
// A test therefore calls unix.Sync, which waits for it, before the scan
// whose disk_size it compares with this.
func diskSizeOf(t *testing.T, path string) int64 { func diskSizeOf(t *testing.T, path string) int64 {
t.Helper() t.Helper()
@@ -1435,115 +1432,45 @@ func TestScanStoresRawHashes(t *testing.T) {
} }
// TestScanStoresDiskSize scans a sparse file, which takes less space on // TestScanStoresDiskSize scans a sparse file, which takes less space on
// disk than its length, once through its directory and once as the // disk than its length, and checks the disk_size stored for it against
// operand itself. Each scan stores the disk_size of the file's lstat. // the file's own lstat.
func TestScanStoresDiskSize(t *testing.T) { func TestScanStoresDiskSize(t *testing.T) {
t.Parallel() t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "sparse", []byte("x"))
// Truncate makes the file longer without giving it blocks for the
// new bytes.
const length = 1024 * 1024 const length = 1024 * 1024
p := sparseFile(t, t.TempDir(), "sparse", length) err := os.Truncate(p, length)
pokeAt(t, p, 0, []byte("x"))
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == 0 || want >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", want, length)
}
for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t)
syncTree(t, db, operand)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, want)
}
}
}
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
// sparse file and moves its mtime on. The rescan hashes the file again
// and rewrites its record with the new disk_size.
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := sparseFile(t, dir, "sparse", 1024*1024)
pokeAt(t, p, 0, []byte("x"))
syncTree(t, db, dir)
before := recordByPath(t, dbRecords(t, db), p).diskSize
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
// The write may leave the mtime the scan recorded; a later one makes
// the rescan read the file again.
later := time.Now().Add(time.Hour)
err := os.Chtimes(p, later, later)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
unix.Sync() // see diskSizeOf syncTree(t, db, dir)
want := diskSizeOf(t, p) want := diskSizeOf(t, p)
if want == before { if want >= length {
t.Fatalf("the file still takes %d bytes on disk after the write "+ t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
"into its hole", want) "length %d", want, length)
} }
st := syncTree(t, db, dir) var got int64
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st) err = db.QueryRowContext(t.Context(),
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
if err != nil {
t.Fatal(err)
} }
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want { if got != want {
t.Errorf("disk_size = %d, want %d", got, want) t.Errorf("disk_size = %d, want %d", got, want)
} }
} }
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
// disk_size the file does not have, as when ZFS had not yet counted the
// file's blocks at the last scan. The rescan writes the file's own
// disk_size into the record without reading the file again.
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "a.bin", pattern(1, 500))
syncTree(t, db, dir)
// lstat counts 512-byte units, so no file takes 1 byte on disk.
_, err := db.ExecContext(t.Context(),
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
if err != nil {
t.Fatal(err)
}
unix.Sync() // see diskSizeOf
// A file read again would count as updated.
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same // TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the // size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file. The operand is // next scan must notice the change and re-hash the file. The operand is