Compare commits

..
Author SHA1 Message Date
sneak 5f51a4da78 Stop the disk size tests failing at random on ZFS (closes #150)
check / check (push) Canceled after 0s
One ZFS commit can take a file through several sizes, and unix.Sync can
return before the blocks are counted. A test that expected the size on
disk to be settled therefore failed at random. The test that needs the
size to change now calls unix.Sync in a loop until it does, for at most
30 seconds. It then rescans until the size reads the same before and
after a rescan, and compares disk_size exactly, within the same 30
seconds. The other two tests fail only on a disk_size outside their two
reads. They call unix.Sync once first, so a wrong 512 usually fails them.

Model: opus-5-5
2026-11-03 02:55:48 +00:00
clawbot 4ecf8dc484 Rewrite TODO.md sentence by sentence (closes #135)
check / check (push) Canceled after 0s
Rewrites `TODO.md` from scratch for #135, by the rules applied to `README.md`. Every completed step keeps its date, its issue or branch and its facts, in short sentences, with lists as bullets. An entry superseded later says so. Entries record history, so they were not corrected against the code.

Known stale: the 2026-10-17 release-rule entry.
Changed: one product name in the 2026-10-15 entry is replaced by a description.

Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
2026-11-02 23:29:22 +01:00
clawbot 2322f8bb02 Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
Rewrites `README.md` from scratch for #135, so that a newcomer can follow each sentence on one reading. It leads with what the program is and how to use it, then the reference, then development. Where the old text and the code disagreed, it states what the code does; #140 lists each point. Test comments that cited moved text cite the new headings.

Review scope: full reviews through the 16th, then only text changed since.
Dropped: the permission to add `github.com/spf13/viper`.

Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
2026-11-02 23:12:27 +01:00
3 changed files with 699 additions and 481 deletions

No files matched your search

+22 -19
View File
@@ -408,8 +408,8 @@ operands:
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A file scanned within those seconds gets its real `disk_size` at the
next scan.
write. A block is the unit in which a filesystem allocates disk space. A file
scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -777,10 +777,11 @@ first dupe files size disk_size
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. So two copies of one tree can differ
in `disk_size`. When a file in one copy is a hard link to one in the other, both
copies count its blocks in full. Removing it from one copy then frees nothing,
as the limitation under "`trees` mode" says.
file's size on a compressed or sparse file. A sparse file has gaps that take no
space on disk. So two copies of one tree can differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the limitation under
"`trees` mode" says.
The paths are escaped as described under "Report output format". The root
directory's path is `/`.
@@ -941,16 +942,17 @@ Only one `scan` at a time can hold the lock.
#### Schema
The schema is the database's layout, its table and its index.
The schema is the database's layout: its tables, its index and its triggers. A
trigger is a statement SQLite runs by itself when a table changes.
`PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
error too. `scan` creates the table and the index and sets the version in one
transaction (see "`info` mode"). So the three land together or not at all. A
first scan stopped while doing so leaves an empty database, which the next scan
sets up. A database at version 0 that already has a `files` table was not made
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
the file and rescan.
error too. `scan` creates the tables, the index and the triggers and sets the
version in one transaction (see "`info` mode"). So they all land together or not
at all. A first scan stopped while doing so leaves an empty database, which the
next scan sets up. A database at version 0 that already has a `files` table was
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
remove the file and rescan.
```sql
CREATE TABLE files (
@@ -982,9 +984,10 @@ END;
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk, `st_blocks` × 512
from lstat. It is less than `size` on a compressed or sparse file (see "Trees
output format").
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
call that stats a file without following a symlink. Its `st_blocks` field
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
compressed or sparse file (see "Trees output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -1002,9 +1005,9 @@ END;
`scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers
make each write to `files` slower, so a first scan of many small files takes
about two fifths longer. On large files, reading them takes most of the time,
so the difference is small.
make each insert, each delete and each change of a record's `size` slower, so
a first scan of many small files takes about two fifths longer. On large
files, reading them takes most of the time, so the difference is small.
### Progress
+603 -437
View File
File diff suppressed because it is too large. Load diff
+74 -25
View File
@@ -43,10 +43,16 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
}
// diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
// blocks only seconds after the write, and fsync does not hasten that.
// A test therefore calls unix.Sync, which waits for it, before the scan
// whose disk_size it compares with this.
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
// blocks only when it commits the write, up to seconds later, and one
// commit can take the file through several growing sizes. So this value
// can change at any moment. Neither fsync nor unix.Sync hastens that,
// but a unix.Sync call often takes long enough that the blocks are
// counted when it returns. A test that needs the value to change waits
// for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
func diskSizeOf(t *testing.T, path string) int64 {
t.Helper()
@@ -1448,20 +1454,26 @@ func TestScanStoresDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == 0 || want >= length {
size := diskSizeOf(t, p)
if size == 0 || size >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", want, length)
"and fewer than its length %d", size, length)
}
for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t)
// ZFS may count the blocks during the scan; see diskSizeOf.
before := diskSizeOf(t, p)
syncTree(t, db, operand)
after := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, want)
if got < before || got > after {
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
operand, got, before, after)
}
}
}
@@ -1479,7 +1491,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
syncTree(t, db, dir)
before := recordByPath(t, dbRecords(t, db), p).diskSize
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
@@ -1492,22 +1504,53 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Fatal(err)
}
unix.Sync() // see diskSizeOf
// Wait for ZFS to count the write; see diskSizeOf.
deadline := time.Now().Add(30 * time.Second)
for diskSizeOf(t, p) == recorded {
if time.Now().After(deadline) {
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
"after the write into its hole", recorded)
}
want := diskSizeOf(t, p)
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
unix.Sync()
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
// ZFS may still be counting, so the size may change during a rescan.
// Only a rescan with the same size on both sides of it can be compared
// exactly. A rescan that keeps the old size still fails, because the
// wait made before differ from it.
for {
before := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("disk_size = %d, want %d", got, want)
st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if before == after {
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
}
}
@@ -1533,15 +1576,21 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf
// ZFS may count the blocks during the rescan; see diskSizeOf.
before := diskSizeOf(t, p)
// A file read again would count as updated.
st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d", got, want)
if got < before || got > after {
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
}
}