Author SHA1 Message Date
sneak ac2f4416fa Stop the disk size tests failing at random on ZFS (closes #150)
check / check (push) Canceled after 0s
One ZFS commit can take a file through several sizes, and unix.Sync can
return before the blocks are counted. A test that expected the size on
disk to be settled therefore failed at random. The test that needs the
size to change now calls unix.Sync in a loop until it does, for at most
30 seconds. All three tests read the size before and after the scan and
fail only on a disk_size outside those two reads. The two that do not
wait call unix.Sync once first, so a wrong 512 usually fails them.

Model: opus-5-5
2026-11-03 00:07:03 +00:00
4 changed files with 20 additions and 47 deletions
+3 -3
View File
@@ -407,9 +407,9 @@ operands:
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan.
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A block is the unit in which a filesystem allocates disk space. A file
scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later
+2 -8
View File
@@ -29,18 +29,12 @@
Newest first. An entry ends with its date and its issue or branch, where those
are known.
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
blocks only seconds after the write. The count can land in under a second.
Both now say up to seconds after the write, as the comment on `diskSizeOf` in
`scan_test.go` does (2026-11-03, https://git.eeqj.de/sneak/sfdupes/issues/152)
- The three disk size tests in `scan_test.go` no longer fail at random on ZFS.
`unix.Sync` can return before ZFS has counted a file's blocks, and one ZFS
commit can take a file through several sizes. The test that needs the size to
change waits for it, for at most 30 seconds. Each test reads the size before
and after its scan. The rewrite test rescans until the two reads match, then
compares exactly. The other two fail only on a `disk_size` outside the two
reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
and after its scan. It fails only on a `disk_size` outside those two reads
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
sentence. Every completed step is kept, with its date and its issue or branch.
+1 -1
View File
@@ -554,7 +554,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The space a file takes on disk can change while its size and
// mtime stay the same. ZFS, for one, counts a new file's blocks
// up to seconds after the write.
// only seconds after the write.
if old.diskSize != ev.rec.diskSize {
s.diskSizes = append(s.diskSizes, ev.rec)
}
+14 -35
View File
@@ -50,9 +50,8 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
// but a unix.Sync call often takes long enough that the blocks are
// counted when it returns. A test that needs the value to change waits
// for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
// it before and after the scan, and fails only on a disk_size outside
// those two reads.
func diskSizeOf(t *testing.T, path string) int64 {
t.Helper()
@@ -1515,42 +1514,22 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
unix.Sync()
}
// ZFS may still be counting, so the size may change during a rescan.
// Only a rescan with the same size on both sides of it can be compared
// exactly. A rescan that keeps the old size still fails, because the
// wait made before differ from it.
for {
before := diskSizeOf(t, p)
// ZFS may still be counting, so the size may change during the rescan.
// A rescan that keeps the old size still fails, because the wait made
// before differ from it.
before := diskSizeOf(t, p)
st := syncTree(t, db, dir)
st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if before == after {
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got < before || got > after {
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
}
}