Author SHA1 Message Date
sneak a00b97e299 Say ZFS counts a new file's blocks up to seconds after the write (closes #152)
check / check (push) Canceled after 0s
`README.md` and a comment in `scan.go` said "only seconds after the write". That is false, because the count can land in under a second. Both now agree with the comment on `diskSizeOf` in `scan_test.go`.

Model: opus-5-5
2026-11-03 03:30:33 +00:00
clawbot 30dae35c51 Stop the disk size tests failing at random on ZFS (closes #150)
check / check (push) Canceled after 0s
The three disk size tests in `scan_test.go` failed at random on ZFS, which failed the build on `next`. ZFS counts a write's blocks up to seconds later, sometimes through several sizes, and `unix.Sync` does not wait for that. The rewrite test now waits for the size to change, then rescans until the size is the same on both sides of a rescan, and compares exactly, all within 30 seconds. The other two tests fail only on a `disk_size` outside the reads around their scan.

Limit: those two can pass on a wrong 512 when ZFS has not yet counted the blocks.

Model: opus-5-5
2026-11-03 04:29:19 +01:00
4 changed files with 64 additions and 28 deletions

No files matched your search

+3 -3
View File
@@ -407,9 +407,9 @@ operands:
is what makes a daily rescan cheap. is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the unchanged. ZFS, for example, counts a new file's blocks up to seconds after
write. A block is the unit in which a filesystem allocates disk space. A file the write. A block is the unit in which a filesystem allocates disk space. A
scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later same-size rewrite counts as a change whenever the filesystem gives it a later
+11 -4
View File
@@ -29,11 +29,18 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
blocks only seconds after the write. The count can land in under a second.
Both now say up to seconds after the write, as the comment on `diskSizeOf` in
`scan_test.go` does (2026-11-03, https://git.eeqj.de/sneak/sfdupes/issues/152)
- The three disk size tests in `scan_test.go` no longer fail at random on ZFS. - The three disk size tests in `scan_test.go` no longer fail at random on ZFS.
`unix.Sync` can return before ZFS has counted a file's blocks. The test that `unix.Sync` can return before ZFS has counted a file's blocks, and one ZFS
needs the size to change waits for it, for at most 30 seconds. Each test reads commit can take a file through several sizes. The test that needs the size to
the size before and after its scan and accepts either value (2026-11-02, change waits for it, for at most 30 seconds. Each test reads the size before
https://git.eeqj.de/sneak/sfdupes/issues/150) and after its scan. The rewrite test rescans until the two reads match, then
compares exactly. The other two fail only on a `disk_size` outside the two
reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by - `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
sentence. Every completed step is kept, with its date and its issue or branch. sentence. Every completed step is kept, with its date and its issue or branch.
+1 -1
View File
@@ -554,7 +554,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The space a file takes on disk can change while its size and // The space a file takes on disk can change while its size and
// mtime stay the same. ZFS, for one, counts a new file's blocks // mtime stay the same. ZFS, for one, counts a new file's blocks
// only seconds after the write. // up to seconds after the write.
if old.diskSize != ev.rec.diskSize { if old.diskSize != ev.rec.diskSize {
s.diskSizes = append(s.diskSizes, ev.rec) s.diskSizes = append(s.diskSizes, ev.rec)
} }
+49 -20
View File
@@ -44,12 +44,15 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
// diskSizeOf returns the bytes the file at path takes on disk, from its // diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's // own lstat: st_blocks counts 512-byte units. ZFS counts a write's
// blocks only when it commits the write, seconds later and sometimes in // blocks only when it commits the write, up to seconds later, and one
// steps, so this value can change at any moment. fsync does not hasten // commit can take the file through several growing sizes. So this value
// that. unix.Sync usually does, but it can return before the blocks are // can change at any moment. Neither fsync nor unix.Sync hastens that,
// counted. A test that needs the value to change therefore waits for // but a unix.Sync call often takes long enough that the blocks are
// that, and a test that compares it with a scan's disk_size reads it // counted when it returns. A test that needs the value to change waits
// before and after the scan and accepts either. // for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
func diskSizeOf(t *testing.T, path string) int64 { func diskSizeOf(t *testing.T, path string) int64 {
t.Helper() t.Helper()
@@ -1449,6 +1452,8 @@ func TestScanStoresDiskSize(t *testing.T) {
p := sparseFile(t, t.TempDir(), "sparse", length) p := sparseFile(t, t.TempDir(), "sparse", length)
pokeAt(t, p, 0, []byte("x")) pokeAt(t, p, 0, []byte("x"))
unix.Sync() // see diskSizeOf
size := diskSizeOf(t, p) size := diskSizeOf(t, p)
if size == 0 || size >= length { if size == 0 || size >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+ t.Fatalf("the file takes %d bytes on disk, want more than none "+
@@ -1466,8 +1471,8 @@ func TestScanStoresDiskSize(t *testing.T) {
after := diskSizeOf(t, p) after := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != before && got != after { if got < before || got > after {
t.Errorf("scan of %s: disk_size = %d, want %d or %d", t.Errorf("scan of %s: disk_size = %d, want %d to %d",
operand, got, before, after) operand, got, before, after)
} }
} }
@@ -1510,20 +1515,42 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
unix.Sync() unix.Sync()
} }
// ZFS may still be counting, so the size may change during the rescan. // ZFS may still be counting, so the size may change during a rescan.
before := diskSizeOf(t, p) // Only a rescan with the same size on both sides of it can be compared
// exactly. A rescan that keeps the old size still fails, because the
// wait made before differ from it.
for {
before := diskSizeOf(t, p)
st := syncTree(t, db, dir) st := syncTree(t, db, dir)
after := diskSizeOf(t, p) after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, updated: 1}) { if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st) t.Fatalf("rescan stats = %+v, want 1 updated", st)
} }
got := recordByPath(t, dbRecords(t, db), p).diskSize if before == after {
if got != before && got != after { got := recordByPath(t, dbRecords(t, db), p).diskSize
t.Errorf("disk_size = %d, want %d or %d", got, before, after) if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
} }
} }
@@ -1547,6 +1574,8 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
unix.Sync() // see diskSizeOf
// ZFS may count the blocks during the rescan; see diskSizeOf. // ZFS may count the blocks during the rescan; see diskSizeOf.
before := diskSizeOf(t, p) before := diskSizeOf(t, p)
@@ -1560,8 +1589,8 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
} }
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != before && got != after { if got < before || got > after {
t.Errorf("disk_size = %d, want %d or %d", got, before, after) t.Errorf("disk_size = %d, want %d to %d", got, before, after)
} }
} }