Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4ae7274bc0 |
@@ -226,12 +226,12 @@ Goals, in order:
|
||||
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
|
||||
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
|
||||
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
|
||||
under resilver). Holding one small record (path, size, mtime) per file in
|
||||
memory during a scan is acceptable; holding every file's hashes is not (they
|
||||
stay in the database). The reporting commands do not hold every file's
|
||||
hashes either: `report` reads the records in signature order and holds the
|
||||
duplicate paths of one size at a time, so its memory grows with the
|
||||
duplicates of the size that has the most of them, and `trees` reads the
|
||||
under resilver). Holding one small record (path, size, size on disk, mtime)
|
||||
per file in memory during a scan is acceptable; holding every file's hashes
|
||||
is not (they stay in the database). The reporting commands do not hold every
|
||||
file's hashes either: `report` reads the records in signature order and
|
||||
holds the duplicate paths of one size at a time, so its memory grows with
|
||||
the duplicates of the size that has the most of them, and `trees` reads the
|
||||
records in path order and keeps each directory's path, digest and totals,
|
||||
plus the hashes of only the files in the directories holding the record
|
||||
being read, so its memory grows with the number of directories and with the
|
||||
@@ -368,7 +368,8 @@ All four subcommands operate on a single SQLite database file:
|
||||
- While it loads the existing records (`load`) and walks: no records. A
|
||||
first scan's database holds none, however long its walk has run.
|
||||
- During the hash phase: the first records, committed 10,000 at a time.
|
||||
- In the update phase: the rest of the hashed records, and the deletions.
|
||||
- In the update phase: the rest of the hashed records, the new `disk_size`
|
||||
of unchanged files whose size on disk changed, and the deletions.
|
||||
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
||||
away, unless a report still has the database open (see above).
|
||||
- `report`, `trees` and `info` open the database read-only and need only read
|
||||
@@ -515,6 +516,11 @@ operands:
|
||||
- A file already in the database is **skipped without reading its contents**
|
||||
when its lstat size equals the recorded size and its lstat mtime is not newer
|
||||
than the recorded mtime. This is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from the recorded `disk_size`, the
|
||||
scan updates `disk_size` alone. It still does not read the file, and counts it
|
||||
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
|
||||
the write. A file scanned within those seconds gets its real `disk_size` at
|
||||
the next scan.
|
||||
- A file whose mtime is newer than recorded, or whose size differs, is hashed
|
||||
again. Change detection compares the mtime to the nanosecond, as finely as the
|
||||
filesystem records it, so a same-size rewrite counts as a change whenever the
|
||||
@@ -542,8 +548,8 @@ inside the walk and hash phases, and batched database writes begin during the
|
||||
hash phase:
|
||||
|
||||
1. **load** — read the existing records. Each record under the operands is kept
|
||||
in memory, by path, with its size and its mtime, for the walk to compare
|
||||
each file with.
|
||||
in memory, by path, with its size, its size on disk and its mtime, for the
|
||||
walk to compare each file with.
|
||||
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||||
with the walk worker pool: every operand seeds the shared queue, and each
|
||||
worker reads one directory at a time, handing discovered subdirectories back
|
||||
@@ -551,9 +557,10 @@ hash phase:
|
||||
(while the directory's metadata is still hot). Sequential directory
|
||||
enumeration is metadata-latency-bound and takes hours at tens of millions of
|
||||
files; per-directory parallelism is what makes the walk tractable on large
|
||||
or busy pools. The walk compares each file with its record as it goes, and
|
||||
an unchanged file needs nothing more. Every new or changed file is carried
|
||||
to the hash phase as a (path, size, size on disk, mtime) record.
|
||||
or busy pools. The walk compares each file with its record as it goes. An
|
||||
unchanged file is not read again. When its size on disk has changed, the
|
||||
update phase writes its new `disk_size`. Every new or changed file is
|
||||
carried to the hash phase as a (path, size, size on disk, mtime) record.
|
||||
3. **hash** — the worker pool hashes every carried file as described in
|
||||
"Duplicate detection" above, which gives the file its `head`, `tail` and
|
||||
`content`. Zero-length files have constant hashes and are never opened.
|
||||
@@ -565,7 +572,8 @@ hash phase:
|
||||
committed in batched transactions **while hashing runs**. A scan interrupted
|
||||
after hours therefore keeps everything hashed so far, and the next scan
|
||||
resumes cheaply, skipping records already written.
|
||||
4. **update** — commit the final partial batch of hashed records. Then delete
|
||||
4. **update** — commit the final partial batch of hashed records, and the new
|
||||
`disk_size` of each unchanged file whose size on disk changed. Then delete
|
||||
the records the scan did not verify: those of vanished files, and of paths
|
||||
that failed to stat or hash.
|
||||
|
||||
@@ -823,12 +831,12 @@ first dupe files size disk_size
|
||||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||
```
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It is less than the
|
||||
file's size on a compressed or sparse file, and a little more on most other
|
||||
files. Two copies of one tree can therefore differ in `disk_size`. When a file
|
||||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||
full. Removing it from one copy then frees nothing, as the known limitation
|
||||
under "`trees` mode" says.
|
||||
A file's `disk_size` is the space its blocks take on disk. It can be more than
|
||||
the file's size, because space is allocated in whole blocks. It is less than the
|
||||
file's size on a compressed or sparse file, so two copies of one tree can differ
|
||||
in `disk_size`. When a file in one copy is a hard link to one in the other, both
|
||||
copies count its blocks in full. Removing it from one copy then frees nothing,
|
||||
as the known limitation under "`trees` mode" says.
|
||||
|
||||
The `first` and `dupe` paths are escaped as described under "Report output
|
||||
format". The root directory's path is `/`.
|
||||
|
||||
@@ -32,8 +32,9 @@
|
||||
- each record holds `disk_size`, the bytes the file takes on disk: its allocated
|
||||
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
|
||||
prints a `disk_size` column after `size`, the sum over the `dupe` copy's
|
||||
files; on a compressed or sparse file it differs from `size` (2026-10-27,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||
files; on a compressed or sparse file it differs from `size`. A rescan updates
|
||||
the `disk_size` of an unchanged file whose size on disk changed, without
|
||||
reading the file (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/147)
|
||||
|
||||
- the database keeps the number of records and the sum of their sizes in the one
|
||||
row of a `totals` table, which three triggers on `files` keep current. `info`
|
||||
|
||||
@@ -729,15 +729,15 @@ func readDatabaseCounts(ctx context.Context,
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// loadFileMeta streams every record's path, size and mtime to fn. Scan
|
||||
// change detection needs no hash values, and skipping the hash columns
|
||||
// keeps the scan's in-memory index small on multi-million-file
|
||||
// databases.
|
||||
// loadFileMeta streams every record's path, size, disk_size and mtime
|
||||
// to fn. Scan change detection needs no hash values, and skipping the
|
||||
// hash columns keeps the scan's in-memory index small on
|
||||
// multi-million-file databases.
|
||||
func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
fn func(path string, size int64, mtime time.Time),
|
||||
fn func(path string, m fileMeta),
|
||||
) error {
|
||||
rows, err := db.QueryContext(ctx,
|
||||
"SELECT path, size, mtime, mtime_nsec FROM files")
|
||||
"SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
@@ -746,16 +746,19 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
path []byte
|
||||
size, sec, nsec int64
|
||||
path []byte
|
||||
m fileMeta
|
||||
sec, nsec int64
|
||||
)
|
||||
|
||||
err = rows.Scan(&path, &size, &sec, &nsec)
|
||||
err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read record: %w", err)
|
||||
}
|
||||
|
||||
fn(string(path), size, time.Unix(sec, nsec))
|
||||
m.mtime = time.Unix(sec, nsec)
|
||||
|
||||
fn(string(path), m)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
@@ -773,21 +776,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
// WAL small and let concurrent reports observe progress.
|
||||
const updateBatchSize = 10000
|
||||
|
||||
// applyChanges writes one scan's database changes — upserts for new and
|
||||
// changed files, deletes for vanished ones — in batched transactions.
|
||||
// Progress is rendered on prog (one increment per change).
|
||||
// applyChanges writes upserts for new and changed files and deletes for
|
||||
// vanished ones, in batched transactions. Progress is rendered on prog
|
||||
// (one increment per change).
|
||||
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
deletes []string, prog *progress,
|
||||
) error {
|
||||
for batch := range slices.Chunk(upserts, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, batch, nil, prog)
|
||||
err := applyBatch(ctx, db, batch, nil, nil, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
for batch := range slices.Chunk(deletes, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, nil, batch, prog)
|
||||
err := applyBatch(ctx, db, nil, nil, batch, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -796,10 +799,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyBatch commits one batch of upserts and deletes in a single
|
||||
// transaction.
|
||||
// applyDiskSizes writes the disk_size of each of recs into its record,
|
||||
// in batched transactions, and leaves the rest of the record alone.
|
||||
// Progress is rendered on prog (one increment per record).
|
||||
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
|
||||
prog *progress,
|
||||
) error {
|
||||
for batch := range slices.Chunk(recs, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, nil, batch, nil, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyBatch commits one batch of upserts, disk_size updates and
|
||||
// deletes in a single transaction.
|
||||
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
deletes []string, prog *progress,
|
||||
diskSizes []fileRec, deletes []string, prog *progress,
|
||||
) error {
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
@@ -813,6 +832,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
return err
|
||||
}
|
||||
|
||||
err = execDiskSizes(ctx, tx, diskSizes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = execDeletes(ctx, tx, deletes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -851,6 +875,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
||||
return nil
|
||||
}
|
||||
|
||||
// execDiskSizes sets the disk_size of the record of each of recs. It
|
||||
// does not touch size, so the totals triggers do not fire.
|
||||
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
|
||||
prog *progress,
|
||||
) error {
|
||||
st, err := tx.PrepareContext(ctx,
|
||||
"UPDATE files SET disk_size = ? WHERE path = ?")
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare disk_size update: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = st.Close() }()
|
||||
|
||||
for _, r := range recs {
|
||||
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
|
||||
if err != nil {
|
||||
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
|
||||
}
|
||||
|
||||
prog.increment()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// execDeletes removes the records for paths no longer present.
|
||||
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
|
||||
prog *progress,
|
||||
|
||||
+23
-4
@@ -818,6 +818,23 @@ func scanFixture(t *testing.T) []string {
|
||||
return dupes
|
||||
}
|
||||
|
||||
// storedDiskSize returns the disk_size of the record for path in the
|
||||
// database at dbPath. A test of a trees run compares with this rather
|
||||
// than with a new lstat, which on ZFS can differ from the scan's (see
|
||||
// diskSizeOf).
|
||||
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
db, err := openReportDatabase(t.Context(), dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
return recordByPath(t, dbRecords(t, db), path).diskSize
|
||||
}
|
||||
|
||||
// scanOK runs scan over operands, fails the test unless it exits 0 with
|
||||
// nothing on stdout, and returns everything it printed to stderr.
|
||||
func scanOK(t *testing.T, operands ...string) string {
|
||||
@@ -1143,7 +1160,7 @@ func TestRunTreesSucceeds(t *testing.T) {
|
||||
// trees of each other.
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
@@ -1179,8 +1196,8 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
|
||||
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
||||
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
||||
"\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
|
||||
"\n",
|
||||
"\t1\t300\t" +
|
||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
|
||||
}
|
||||
|
||||
for name, want := range cases {
|
||||
@@ -1215,9 +1232,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
||||
t.Errorf("report stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
|
||||
|
||||
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
|
||||
strconv.FormatInt(diskSize, 10) + "\n"
|
||||
if got := runStdout(t, cmdTrees); got != want {
|
||||
t.Errorf("trees stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
@@ -66,11 +66,12 @@ type fileRec struct {
|
||||
}
|
||||
|
||||
// fileMeta is the in-memory index entry for one existing database
|
||||
// record: just enough for change detection. Hashes stay on disk; at
|
||||
// record: what the walk compares the file with. Hashes stay on disk; at
|
||||
// tens of millions of records they would dominate the scan's memory.
|
||||
type fileMeta struct {
|
||||
size int64
|
||||
mtime time.Time
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
}
|
||||
|
||||
// scanUnit is what every scan phase counts, as its progress display
|
||||
@@ -341,11 +342,12 @@ type scanStats struct {
|
||||
// whatever remains after the walk and hash phases is deleted by the
|
||||
// update phase.
|
||||
type scanState struct {
|
||||
db *sql.DB
|
||||
existing map[string]fileMeta
|
||||
toHash []fileRec // new and changed files: must be read
|
||||
batch []scanRec
|
||||
st scanStats
|
||||
db *sql.DB
|
||||
existing map[string]fileMeta
|
||||
toHash []fileRec // new and changed files: must be read
|
||||
diskSizes []fileRec // unchanged files whose disk_size changed
|
||||
batch []scanRec
|
||||
st scanStats
|
||||
}
|
||||
|
||||
// syncScan synchronizes the database with the filesystem under roots;
|
||||
@@ -474,24 +476,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
|
||||
s.existing = make(map[string]fileMeta)
|
||||
|
||||
return loadFileMeta(ctx, s.db,
|
||||
func(path string, size int64, mtime time.Time) {
|
||||
prog.increment()
|
||||
return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
|
||||
prog.increment()
|
||||
|
||||
if underAnyRoot(path, roots) {
|
||||
s.existing[path] = fileMeta{size: size, mtime: mtime}
|
||||
}
|
||||
})
|
||||
if underAnyRoot(path, roots) {
|
||||
s.existing[path] = m
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// walkPhase drains the walk and queues every new or changed file for
|
||||
// the hash phase. An unchanged file needs nothing more, and its record
|
||||
// counts as verified. A record at or below a path the walk left out,
|
||||
// because -x stopped at it, it is a default exclude, or it is not on a
|
||||
// persistent filesystem, counts as outside the roots: it is not
|
||||
// deleted. That does not hold below a root at or inside that path,
|
||||
// since that root's own walk verifies the records below it; nor is a
|
||||
// line printed for such a root.
|
||||
// the hash phase. An unchanged file is not read, and its record counts
|
||||
// as verified. When its disk_size differs from the record's, it is
|
||||
// queued for the update phase, which writes the new disk_size. A record
|
||||
// at or below a path the walk left out, because -x stopped at it, it is
|
||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
||||
// outside the roots: it is not deleted. That does not hold below a root
|
||||
// at or inside that path, since that root's own walk verifies the
|
||||
// records below it; nor is a line printed for such a root.
|
||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
// The records the last scan left under the roots estimate how many
|
||||
// files the walk will find. A first scan has none, and no estimate.
|
||||
@@ -549,6 +551,13 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
delete(s.existing, ev.rec.path)
|
||||
|
||||
s.st.unchanged++
|
||||
|
||||
// The space a file takes on disk can change while its size and
|
||||
// mtime stay the same. ZFS, for one, counts a new file's blocks
|
||||
// only seconds after the write.
|
||||
if old.diskSize != ev.rec.diskSize {
|
||||
s.diskSizes = append(s.diskSizes, ev.rec)
|
||||
}
|
||||
}
|
||||
|
||||
prog.finish()
|
||||
@@ -719,7 +728,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
err := applyBatch(ctx, s.db, s.batch, nil, nil)
|
||||
err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -730,9 +739,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// updatePhase writes the scan's tail under one progress display: the
|
||||
// final partial batch of hashed records, and deletions for every record
|
||||
// the scan did not verify (vanished files, plus paths that failed to
|
||||
// stat or hash).
|
||||
// final partial batch of hashed records, the new disk_size of every
|
||||
// unchanged file whose disk_size changed, and deletions for every
|
||||
// record the scan did not verify (vanished files, plus paths that
|
||||
// failed to stat or hash).
|
||||
func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
deletes := make([]string, 0, len(s.existing))
|
||||
for path := range s.existing {
|
||||
@@ -744,7 +754,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
|
||||
s.st.removed = len(deletes)
|
||||
|
||||
total := len(s.batch) + len(deletes)
|
||||
total := len(s.batch) + len(s.diskSizes) + len(deletes)
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
||||
stalledOn: "db",
|
||||
@@ -759,6 +769,11 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
|
||||
s.batch = nil
|
||||
|
||||
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||
}
|
||||
|
||||
|
||||
+94
-21
@@ -43,7 +43,10 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
||||
}
|
||||
|
||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||
// own lstat: st_blocks counts 512-byte units.
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
|
||||
// blocks only seconds after the write, and fsync does not hasten that.
|
||||
// A test therefore calls unix.Sync, which waits for it, before the scan
|
||||
// whose disk_size it compares with this.
|
||||
func diskSizeOf(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
@@ -1432,45 +1435,115 @@ func TestScanStoresRawHashes(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestScanStoresDiskSize scans a sparse file, which takes less space on
|
||||
// disk than its length, and checks the disk_size stored for it against
|
||||
// the file's own lstat.
|
||||
// disk than its length, once through its directory and once as the
|
||||
// operand itself. Each scan stores the disk_size of the file's lstat.
|
||||
func TestScanStoresDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const length = 1024 * 1024
|
||||
|
||||
p := sparseFile(t, t.TempDir(), "sparse", length)
|
||||
pokeAt(t, p, 0, []byte("x"))
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == 0 || want >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want more than none "+
|
||||
"and fewer than its length %d", want, length)
|
||||
}
|
||||
|
||||
for _, operand := range []string{filepath.Dir(p), p} {
|
||||
db := openTestDB(t)
|
||||
syncTree(t, db, operand)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d",
|
||||
operand, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
|
||||
// sparse file and moves its mtime on. The rescan hashes the file again
|
||||
// and rewrites its record with the new disk_size.
|
||||
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
p := writeFile(t, dir, "sparse", []byte("x"))
|
||||
|
||||
// Truncate makes the file longer without giving it blocks for the
|
||||
// new bytes.
|
||||
const length = 1024 * 1024
|
||||
|
||||
err := os.Truncate(p, length)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := sparseFile(t, dir, "sparse", 1024*1024)
|
||||
pokeAt(t, p, 0, []byte("x"))
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
|
||||
"length %d", want, length)
|
||||
}
|
||||
before := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
|
||||
var got int64
|
||||
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
|
||||
// The write may leave the mtime the scan recorded; a later one makes
|
||||
// the rescan read the file again.
|
||||
later := time.Now().Add(time.Hour)
|
||||
|
||||
err := os.Chtimes(p, later, later)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == before {
|
||||
t.Fatalf("the file still takes %d bytes on disk after the write "+
|
||||
"into its hole", want)
|
||||
}
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
|
||||
// disk_size the file does not have, as when ZFS had not yet counted the
|
||||
// file's blocks at the last scan. The rescan writes the file's own
|
||||
// disk_size into the record without reading the file again.
|
||||
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
p := writeFile(t, dir, "a.bin", pattern(1, 500))
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
// lstat counts 512-byte units, so no file takes 1 byte on disk.
|
||||
_, err := db.ExecContext(t.Context(),
|
||||
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
// A file read again would count as updated.
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, unchanged: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
|
||||
}
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if want := diskSizeOf(t, p); got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
||||
// size with an mtime later in the same second as the recorded one: the
|
||||
// next scan must notice the change and re-hash the file. The operand is
|
||||
|
||||
Reference in New Issue
Block a user