Compare commits

..
Author SHA1 Message Date
sneak 5f51a4da78 Stop the disk size tests failing at random on ZFS (closes #150)
check / check (push) Canceled after 0s
One ZFS commit can take a file through several sizes, and unix.Sync can
return before the blocks are counted. A test that expected the size on
disk to be settled therefore failed at random. The test that needs the
size to change now calls unix.Sync in a loop until it does, for at most
30 seconds. It then rescans until the size reads the same before and
after a rescan, and compares disk_size exactly, within the same 30
seconds. The other two tests fail only on a disk_size outside their two
reads. They call unix.Sync once first, so a wrong 512 usually fails them.

Model: opus-5-5
2026-11-03 02:55:48 +00:00
clawbot 4ecf8dc484 Rewrite TODO.md sentence by sentence (closes #135)
check / check (push) Canceled after 0s
Rewrites `TODO.md` from scratch for #135, by the rules applied to `README.md`. Every completed step keeps its date, its issue or branch and its facts, in short sentences, with lists as bullets. An entry superseded later says so. Entries record history, so they were not corrected against the code.

Known stale: the 2026-10-17 release-rule entry.
Changed: one product name in the 2026-10-15 entry is replaced by a description.

Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
2026-11-02 23:29:22 +01:00
clawbot 2322f8bb02 Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
Rewrites `README.md` from scratch for #135, so that a newcomer can follow each sentence on one reading. It leads with what the program is and how to use it, then the reference, then development. Where the old text and the code disagreed, it states what the code does; #140 lists each point. Test comments that cited moved text cite the new headings.

Review scope: full reviews through the 16th, then only text changed since.
Dropped: the permission to add `github.com/spf13/viper`.

Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
2026-11-02 23:12:27 +01:00
3 changed files with 693 additions and 475 deletions

No files matched your search

+16 -13
View File
@@ -408,8 +408,8 @@ operands:
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A file scanned within those seconds gets its real `disk_size` at the write. A block is the unit in which a filesystem allocates disk space. A file
next scan. scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -777,10 +777,11 @@ first dupe files size disk_size
A file's `disk_size` is the space its blocks take on disk. It can be more than A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. So two copies of one tree can differ file's size on a compressed or sparse file. A sparse file has gaps that take no
in `disk_size`. When a file in one copy is a hard link to one in the other, both space on disk. So two copies of one tree can differ in `disk_size`. When a file
copies count its blocks in full. Removing it from one copy then frees nothing, in one copy is a hard link to one in the other, both copies count its blocks in
as the limitation under "`trees` mode" says. full. Removing it from one copy then frees nothing, as the limitation under
"`trees` mode" says.
The paths are escaped as described under "Report output format". The root The paths are escaped as described under "Report output format". The root
directory's path is `/`. directory's path is `/`.
@@ -941,7 +942,8 @@ Only one `scan` at a time can hold the lock.
#### Schema #### Schema
The schema is the database's layout: its tables, its index and its triggers. The schema is the database's layout: its tables, its index and its triggers. A
trigger is a statement SQLite runs by itself when a table changes.
`PRAGMA user_version` holds the schema version, currently 1. An empty database `PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error. is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal `report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
@@ -982,9 +984,10 @@ END;
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not - `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text. necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk, `st_blocks` × 512 - `disk_size` is the space the file's blocks take on disk. `lstat` is the system
from lstat. It is less than `size` on a compressed or sparse file (see "Trees call that stats a file without following a symlink. Its `st_blocks` field
output format"). counts that space in units of 512 bytes. `disk_size` is less than `size` on a
compressed or sparse file (see "Trees output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and - `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any `mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -1002,9 +1005,9 @@ END;
`scan` and the `read` phase of `report` and `trees` read the number of records `scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers record. That takes minutes on a database of tens of millions. The triggers
make each write to `files` slower, so a first scan of many small files takes make each insert, each delete and each change of a record's `size` slower, so
about two fifths longer. On large files, reading them takes most of the time, a first scan of many small files takes about two fifths longer. On large
so the difference is small. files, reading them takes most of the time, so the difference is small.
### Progress ### Progress
+603 -437
View File
File diff suppressed because it is too large. Load diff
+74 -25
View File
@@ -43,10 +43,16 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
} }
// diskSizeOf returns the bytes the file at path takes on disk, from its // diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's // own lstat: st_blocks counts 512-byte units. ZFS counts a write's
// blocks only seconds after the write, and fsync does not hasten that. // blocks only when it commits the write, up to seconds later, and one
// A test therefore calls unix.Sync, which waits for it, before the scan // commit can take the file through several growing sizes. So this value
// whose disk_size it compares with this. // can change at any moment. Neither fsync nor unix.Sync hastens that,
// but a unix.Sync call often takes long enough that the blocks are
// counted when it returns. A test that needs the value to change waits
// for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
func diskSizeOf(t *testing.T, path string) int64 { func diskSizeOf(t *testing.T, path string) int64 {
t.Helper() t.Helper()
@@ -1448,20 +1454,26 @@ func TestScanStoresDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p) size := diskSizeOf(t, p)
if want == 0 || want >= length { if size == 0 || size >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+ t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", want, length) "and fewer than its length %d", size, length)
} }
for _, operand := range []string{filepath.Dir(p), p} { for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t) db := openTestDB(t)
// ZFS may count the blocks during the scan; see diskSizeOf.
before := diskSizeOf(t, p)
syncTree(t, db, operand) syncTree(t, db, operand)
after := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want { if got < before || got > after {
t.Errorf("scan of %s: disk_size = %d, want %d", t.Errorf("scan of %s: disk_size = %d, want %d to %d",
operand, got, want) operand, got, before, after)
} }
} }
} }
@@ -1479,7 +1491,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
syncTree(t, db, dir) syncTree(t, db, dir)
before := recordByPath(t, dbRecords(t, db), p).diskSize recorded := recordByPath(t, dbRecords(t, db), p).diskSize
pokeAt(t, p, 512*1024, pattern(7, 64*1024)) pokeAt(t, p, 512*1024, pattern(7, 64*1024))
@@ -1492,22 +1504,53 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
unix.Sync() // see diskSizeOf // Wait for ZFS to count the write; see diskSizeOf.
deadline := time.Now().Add(30 * time.Second)
for diskSizeOf(t, p) == recorded {
if time.Now().After(deadline) {
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
"after the write into its hole", recorded)
}
want := diskSizeOf(t, p) unix.Sync()
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
} }
st := syncTree(t, db, dir) // ZFS may still be counting, so the size may change during a rescan.
if st != (scanStats{walked: 1, updated: 1}) { // Only a rescan with the same size on both sides of it can be compared
t.Fatalf("rescan stats = %+v, want 1 updated", st) // exactly. A rescan that keeps the old size still fails, because the
} // wait made before differ from it.
for {
before := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize st := syncTree(t, db, dir)
if got != want {
t.Errorf("disk_size = %d, want %d", got, want) after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if before == after {
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
} }
} }
@@ -1533,15 +1576,21 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf unix.Sync() // see diskSizeOf
// ZFS may count the blocks during the rescan; see diskSizeOf.
before := diskSizeOf(t, p)
// A file read again would count as updated. // A file read again would count as updated.
st := syncTree(t, db, dir) st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, unchanged: 1}) { if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st) t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
} }
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if want := diskSizeOf(t, p); got != want { if got < before || got > after {
t.Errorf("disk_size = %d, want %d", got, want) t.Errorf("disk_size = %d, want %d to %d", got, before, after)
} }
} }