1 Commits
Author SHA1 Message Date
clawbot 4ed2e370b1 Record each file's size on disk and print it in trees (closes #147)
check / check (push) Waiting to run
The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. When a rescan finds a file with its size
and mtime unchanged but a different disk_size, it updates disk_size
alone, without reading the file. trees adds disk_size up per directory
and prints it as a column after size. On a compressed or sparse file it
is less than size, so two copies of one tree can differ.

Judgement calls: each row's disk_size is the dupe copy's, not the
first's. A file whose record gets only a new disk_size counts as
unchanged.

Model: opus-5-5
2026-10-27 19:56:49 +01:00
6 changed files with 256 additions and 91 deletions
+27 -19
View File
@@ -226,12 +226,12 @@ Goals, in order:
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below). its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
Therefore no file of 100 MiB or more is read in full. Scale target: tens of Therefore no file of 100 MiB or more is read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
under resilver). Holding one small record (path, size, mtime) per file in under resilver). Holding one small record (path, size, size on disk, mtime)
memory during a scan is acceptable; holding every file's hashes is not (they per file in memory during a scan is acceptable; holding every file's hashes
stay in the database). The reporting commands do not hold every file's is not (they stay in the database). The reporting commands do not hold every
hashes either: `report` reads the records in signature order and holds the file's hashes either: `report` reads the records in signature order and
duplicate paths of one size at a time, so its memory grows with the holds the duplicate paths of one size at a time, so its memory grows with
duplicates of the size that has the most of them, and `trees` reads the the duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals, records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the being read, so its memory grows with the number of directories and with the
@@ -368,7 +368,8 @@ All four subcommands operate on a single SQLite database file:
- While it loads the existing records (`load`) and walks: no records. A - While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run. first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time. - During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, and the deletions. - In the update phase: the rest of the hashed records, the new `disk_size`
of unchanged files whose size on disk changed, and the deletions.
- At the end, as it closes the database: the `-wal` and `-shm` files go - At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above). away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read - `report`, `trees` and `info` open the database read-only and need only read
@@ -515,6 +516,11 @@ operands:
- A file already in the database is **skipped without reading its contents** - A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap. than the recorded mtime. This is what makes a daily rescan cheap.
- When such a file's size on disk differs from the recorded `disk_size`, the
scan updates `disk_size` alone. It still does not read the file, and counts it
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
the write. A file scanned within those seconds gets its real `disk_size` at
the next scan.
- A file whose mtime is newer than recorded, or whose size differs, is hashed - A file whose mtime is newer than recorded, or whose size differs, is hashed
again. Change detection compares the mtime to the nanosecond, as finely as the again. Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the filesystem records it, so a same-size rewrite counts as a change whenever the
@@ -542,8 +548,8 @@ inside the walk and hash phases, and batched database writes begin during the
hash phase: hash phase:
1. **load** — read the existing records. Each record under the operands is kept 1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size and its mtime, for the walk to compare in memory, by path, with its size, its size on disk and its mtime, for the
each file with. walk to compare each file with.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently 2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back worker reads one directory at a time, handing discovered subdirectories back
@@ -551,9 +557,10 @@ hash phase:
(while the directory's metadata is still hot). Sequential directory (while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes, and or busy pools. The walk compares each file with its record as it goes. An
an unchanged file needs nothing more. Every new or changed file is carried unchanged file is not read again. When its size on disk has changed, the
to the hash phase as a (path, size, size on disk, mtime) record. update phase writes its new `disk_size`. Every new or changed file is
carried to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in 3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and "Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened. `content`. Zero-length files have constant hashes and are never opened.
@@ -565,7 +572,8 @@ hash phase:
committed in batched transactions **while hashing runs**. A scan interrupted committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written. resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete 4. **update** — commit the final partial batch of hashed records, and the new
`disk_size` of each unchanged file whose size on disk changed. Then delete
the records the scan did not verify: those of vanished files, and of paths the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash. that failed to stat or hash.
@@ -823,12 +831,12 @@ first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984 /srv/a/project /srv/backup/project 3417 104857600 61865984
``` ```
A file's `disk_size` is the space its blocks take on disk. It is less than the A file's `disk_size` is the space its blocks take on disk. It can be more than
file's size on a compressed or sparse file, and a little more on most other the file's size, because space is allocated in whole blocks. It is less than the
files. Two copies of one tree can therefore differ in `disk_size`. When a file file's size on a compressed or sparse file, so two copies of one tree can differ
in one copy is a hard link to one in the other, both copies count its blocks in in `disk_size`. When a file in one copy is a hard link to one in the other, both
full. Removing it from one copy then frees nothing, as the known limitation copies count its blocks in full. Removing it from one copy then frees nothing,
under "`trees` mode" says. as the known limitation under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`. format". The root directory's path is `/`.
+3 -2
View File
@@ -32,8 +32,9 @@
- each record holds `disk_size`, the bytes the file takes on disk: its allocated - each record holds `disk_size`, the bytes the file takes on disk: its allocated
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees` blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
prints a `disk_size` column after `size`, the sum over the `dupe` copy's prints a `disk_size` column after `size`, the sum over the `dupe` copy's
files; on a compressed or sparse file it differs from `size` (2026-10-27, files; on a compressed or sparse file it differs from `size`. A rescan updates
https://git.eeqj.de/sneak/sfdupes/issues/147) the `disk_size` of an unchanged file whose size on disk changed, without
reading the file (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/147)
- the database keeps the number of records and the sum of their sizes in the one - the database keeps the number of records and the sum of their sizes in the one
row of a `totals` table, which three triggers on `files` keep current. `info` row of a `totals` table, which three triggers on `files` keep current. `info`
+67 -18
View File
@@ -729,15 +729,15 @@ func readDatabaseCounts(ctx context.Context,
return c, nil return c, nil
} }
// loadFileMeta streams every record's path, size and mtime to fn. Scan // loadFileMeta streams every record's path, size, disk_size and mtime
// change detection needs no hash values, and skipping the hash columns // to fn. Scan change detection needs no hash values, and skipping the
// keeps the scan's in-memory index small on multi-million-file // hash columns keeps the scan's in-memory index small on
// databases. // multi-million-file databases.
func loadFileMeta(ctx context.Context, db *sql.DB, func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, size int64, mtime time.Time), fn func(path string, m fileMeta),
) error { ) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec FROM files") "SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -746,16 +746,19 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
size, sec, nsec int64 m fileMeta
sec, nsec int64
) )
err = rows.Scan(&path, &size, &sec, &nsec) err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
fn(string(path), size, time.Unix(sec, nsec)) m.mtime = time.Unix(sec, nsec)
fn(string(path), m)
} }
err = rows.Err() err = rows.Err()
@@ -773,21 +776,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// WAL small and let concurrent reports observe progress. // WAL small and let concurrent reports observe progress.
const updateBatchSize = 10000 const updateBatchSize = 10000
// applyChanges writes one scan's database changes — upserts for new and // applyChanges writes upserts for new and changed files and deletes for
// changed files, deletes for vanished ones — in batched transactions. // vanished ones, in batched transactions. Progress is rendered on prog
// Progress is rendered on prog (one increment per change). // (one increment per change).
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec, func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
deletes []string, prog *progress, deletes []string, prog *progress,
) error { ) error {
for batch := range slices.Chunk(upserts, updateBatchSize) { for batch := range slices.Chunk(upserts, updateBatchSize) {
err := applyBatch(ctx, db, batch, nil, prog) err := applyBatch(ctx, db, batch, nil, nil, prog)
if err != nil { if err != nil {
return err return err
} }
} }
for batch := range slices.Chunk(deletes, updateBatchSize) { for batch := range slices.Chunk(deletes, updateBatchSize) {
err := applyBatch(ctx, db, nil, batch, prog) err := applyBatch(ctx, db, nil, nil, batch, prog)
if err != nil { if err != nil {
return err return err
} }
@@ -796,10 +799,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
return nil return nil
} }
// applyBatch commits one batch of upserts and deletes in a single // applyDiskSizes writes the disk_size of each of recs into its record,
// transaction. // in batched transactions, and leaves the rest of the record alone.
// Progress is rendered on prog (one increment per record).
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
prog *progress,
) error {
for batch := range slices.Chunk(recs, updateBatchSize) {
err := applyBatch(ctx, db, nil, batch, nil, prog)
if err != nil {
return err
}
}
return nil
}
// applyBatch commits one batch of upserts, disk_size updates and
// deletes in a single transaction.
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec, func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
deletes []string, prog *progress, diskSizes []fileRec, deletes []string, prog *progress,
) error { ) error {
tx, err := db.BeginTx(ctx, nil) tx, err := db.BeginTx(ctx, nil)
if err != nil { if err != nil {
@@ -813,6 +832,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
return err return err
} }
err = execDiskSizes(ctx, tx, diskSizes, prog)
if err != nil {
return err
}
err = execDeletes(ctx, tx, deletes, prog) err = execDeletes(ctx, tx, deletes, prog)
if err != nil { if err != nil {
return err return err
@@ -851,6 +875,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
return nil return nil
} }
// execDiskSizes sets the disk_size of the record of each of recs. It
// does not touch size, so the totals triggers do not fire.
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
prog *progress,
) error {
st, err := tx.PrepareContext(ctx,
"UPDATE files SET disk_size = ? WHERE path = ?")
if err != nil {
return fmt.Errorf("prepare disk_size update: %w", err)
}
defer func() { _ = st.Close() }()
for _, r := range recs {
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
if err != nil {
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
}
prog.increment()
}
return nil
}
// execDeletes removes the records for paths no longer present. // execDeletes removes the records for paths no longer present.
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string, func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
prog *progress, prog *progress,
+23 -4
View File
@@ -818,6 +818,23 @@ func scanFixture(t *testing.T) []string {
return dupes return dupes
} }
// storedDiskSize returns the disk_size of the record for path in the
// database at dbPath. A test of a trees run compares with this rather
// than with a new lstat, which on ZFS can differ from the scan's (see
// diskSizeOf).
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
t.Helper()
db, err := openReportDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
return recordByPath(t, dbRecords(t, db), path).diskSize
}
// scanOK runs scan over operands, fails the test unless it exits 0 with // scanOK runs scan over operands, fails the test unless it exits 0 with
// nothing on stdout, and returns everything it printed to stderr. // nothing on stdout, and returns everything it printed to stderr.
func scanOK(t *testing.T, operands ...string) string { func scanOK(t *testing.T, operands ...string) string {
@@ -1143,7 +1160,7 @@ func TestRunTreesSucceeds(t *testing.T) {
// trees of each other. // trees of each other.
want := "first\tdupe\tfiles\tsize\tdisk_size\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n" strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -1179,8 +1196,8 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
dupes[0] + "\t" + dupes[1] + "\t300\n", dupes[0] + "\t" + dupes[1] + "\t300\n",
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" + cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
"\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\t1\t300\t" +
"\n", strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
} }
for name, want := range cases { for name, want := range cases {
@@ -1215,9 +1232,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Errorf("report stdout = %q, want %q", got, want) t.Errorf("report stdout = %q, want %q", got, want)
} }
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
want = "first\tdupe\tfiles\tsize\tdisk_size\n" + want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n" strconv.FormatInt(diskSize, 10) + "\n"
if got := runStdout(t, cmdTrees); got != want { if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want) t.Errorf("trees stdout = %q, want %q", got, want)
} }
+42 -27
View File
@@ -66,11 +66,12 @@ type fileRec struct {
} }
// fileMeta is the in-memory index entry for one existing database // fileMeta is the in-memory index entry for one existing database
// record: just enough for change detection. Hashes stay on disk; at // record: what the walk compares the file with. Hashes stay on disk; at
// tens of millions of records they would dominate the scan's memory. // tens of millions of records they would dominate the scan's memory.
type fileMeta struct { type fileMeta struct {
size int64 size int64
mtime time.Time diskSize int64
mtime time.Time
} }
// scanUnit is what every scan phase counts, as its progress display // scanUnit is what every scan phase counts, as its progress display
@@ -341,11 +342,12 @@ type scanStats struct {
// whatever remains after the walk and hash phases is deleted by the // whatever remains after the walk and hash phases is deleted by the
// update phase. // update phase.
type scanState struct { type scanState struct {
db *sql.DB db *sql.DB
existing map[string]fileMeta existing map[string]fileMeta
toHash []fileRec // new and changed files: must be read toHash []fileRec // new and changed files: must be read
batch []scanRec diskSizes []fileRec // unchanged files whose disk_size changed
st scanStats batch []scanRec
st scanStats
} }
// syncScan synchronizes the database with the filesystem under roots; // syncScan synchronizes the database with the filesystem under roots;
@@ -474,24 +476,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta) s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db, return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
func(path string, size int64, mtime time.Time) { prog.increment()
prog.increment()
if underAnyRoot(path, roots) { if underAnyRoot(path, roots) {
s.existing[path] = fileMeta{size: size, mtime: mtime} s.existing[path] = m
} }
}) })
} }
// walkPhase drains the walk and queues every new or changed file for // walkPhase drains the walk and queues every new or changed file for
// the hash phase. An unchanged file needs nothing more, and its record // the hash phase. An unchanged file is not read, and its record counts
// counts as verified. A record at or below a path the walk left out, // as verified. When its disk_size differs from the record's, it is
// because -x stopped at it, it is a default exclude, or it is not on a // queued for the update phase, which writes the new disk_size. A record
// persistent filesystem, counts as outside the roots: it is not // at or below a path the walk left out, because -x stopped at it, it is
// deleted. That does not hold below a root at or inside that path, // a default exclude, or it is not on a persistent filesystem, counts as
// since that root's own walk verifies the records below it; nor is a // outside the roots: it is not deleted. That does not hold below a root
// line printed for such a root. // at or inside that path, since that root's own walk verifies the
// records below it; nor is a line printed for such a root.
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) { func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The records the last scan left under the roots estimate how many // The records the last scan left under the roots estimate how many
// files the walk will find. A first scan has none, and no estimate. // files the walk will find. A first scan has none, and no estimate.
@@ -549,6 +551,13 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
delete(s.existing, ev.rec.path) delete(s.existing, ev.rec.path)
s.st.unchanged++ s.st.unchanged++
// The space a file takes on disk can change while its size and
// mtime stay the same. ZFS, for one, counts a new file's blocks
// only seconds after the write.
if old.diskSize != ev.rec.diskSize {
s.diskSizes = append(s.diskSizes, ev.rec)
}
} }
prog.finish() prog.finish()
@@ -719,7 +728,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
return nil return nil
} }
err := applyBatch(ctx, s.db, s.batch, nil, nil) err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
if err != nil { if err != nil {
return err return err
} }
@@ -730,9 +739,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
} }
// updatePhase writes the scan's tail under one progress display: the // updatePhase writes the scan's tail under one progress display: the
// final partial batch of hashed records, and deletions for every record // final partial batch of hashed records, the new disk_size of every
// the scan did not verify (vanished files, plus paths that failed to // unchanged file whose disk_size changed, and deletions for every
// stat or hash). // record the scan did not verify (vanished files, plus paths that
// failed to stat or hash).
func (s *scanState) updatePhase(ctx context.Context) error { func (s *scanState) updatePhase(ctx context.Context) error {
deletes := make([]string, 0, len(s.existing)) deletes := make([]string, 0, len(s.existing))
for path := range s.existing { for path := range s.existing {
@@ -744,7 +754,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.st.removed = len(deletes) s.st.removed = len(deletes)
total := len(s.batch) + len(deletes) total := len(s.batch) + len(s.diskSizes) + len(deletes)
prog := newProgress(progressParams{ prog := newProgress(progressParams{
label: scanPhase("update"), unit: scanUnit, total: int64(total), label: scanPhase("update"), unit: scanUnit, total: int64(total),
stalledOn: "db", stalledOn: "db",
@@ -759,6 +769,11 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.batch = nil s.batch = nil
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
if err != nil {
return err
}
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
+94 -21
View File
@@ -43,7 +43,10 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
} }
// diskSizeOf returns the bytes the file at path takes on disk, from its // diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. // own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
// blocks only seconds after the write, and fsync does not hasten that.
// A test therefore calls unix.Sync, which waits for it, before the scan
// whose disk_size it compares with this.
func diskSizeOf(t *testing.T, path string) int64 { func diskSizeOf(t *testing.T, path string) int64 {
t.Helper() t.Helper()
@@ -1432,45 +1435,115 @@ func TestScanStoresRawHashes(t *testing.T) {
} }
// TestScanStoresDiskSize scans a sparse file, which takes less space on // TestScanStoresDiskSize scans a sparse file, which takes less space on
// disk than its length, and checks the disk_size stored for it against // disk than its length, once through its directory and once as the
// the file's own lstat. // operand itself. Each scan stores the disk_size of the file's lstat.
func TestScanStoresDiskSize(t *testing.T) { func TestScanStoresDiskSize(t *testing.T) {
t.Parallel() t.Parallel()
const length = 1024 * 1024
p := sparseFile(t, t.TempDir(), "sparse", length)
pokeAt(t, p, 0, []byte("x"))
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == 0 || want >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", want, length)
}
for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t)
syncTree(t, db, operand)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, want)
}
}
}
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
// sparse file and moves its mtime on. The rescan hashes the file again
// and rewrites its record with the new disk_size.
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir() dir := t.TempDir()
db := openTestDB(t) db := openTestDB(t)
p := writeFile(t, dir, "sparse", []byte("x")) p := sparseFile(t, dir, "sparse", 1024*1024)
pokeAt(t, p, 0, []byte("x"))
// Truncate makes the file longer without giving it blocks for the
// new bytes.
const length = 1024 * 1024
err := os.Truncate(p, length)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir) syncTree(t, db, dir)
want := diskSizeOf(t, p) before := recordByPath(t, dbRecords(t, db), p).diskSize
if want >= length {
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
"length %d", want, length)
}
var got int64 pokeAt(t, p, 512*1024, pattern(7, 64*1024))
err = db.QueryRowContext(t.Context(), // The write may leave the mtime the scan recorded; a later one makes
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got) // the rescan read the file again.
later := time.Now().Add(time.Hour)
err := os.Chtimes(p, later, later)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want { if got != want {
t.Errorf("disk_size = %d, want %d", got, want) t.Errorf("disk_size = %d, want %d", got, want)
} }
} }
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
// disk_size the file does not have, as when ZFS had not yet counted the
// file's blocks at the last scan. The rescan writes the file's own
// disk_size into the record without reading the file again.
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "a.bin", pattern(1, 500))
syncTree(t, db, dir)
// lstat counts 512-byte units, so no file takes 1 byte on disk.
_, err := db.ExecContext(t.Context(),
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
if err != nil {
t.Fatal(err)
}
unix.Sync() // see diskSizeOf
// A file read again would count as updated.
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same // TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the // size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file. The operand is // next scan must notice the change and re-hash the file. The operand is