Compare commits

..
Author SHA1 Message Date
sneak c75421fd11 Carry the README changes from next into the rewrite
check / check (push) Canceled after 0s
`next` changed the old README after this branch was cut, in
#146 (the `totals` table, commit
98193b2) and #147 (`disk_size`,
commit 4ed2e37). This commit states the same facts in the rewritten
README, in the places #140 (comment)
lists, and changes nothing else. Every statement was checked against the
code on `next`.

Model: fable-5-1
2026-11-02 21:47:23 +00:00
sneak d8b58bf281 Rewrite README.md sentence by sentence for a newcomer
Rewrites `README.md` from scratch for
#135. Every statement was checked
against the code on `next` and against what the built binary prints on
small trees under `/var/tmp`. The headings that the code comments cite
are kept.

Where the old text and the repository disagreed, the new text states
what the code does. #140 lists
every such point. Among them: `@latest` installs the newest tagged
release, ten million files in both "Rationale" and "Design", no Windows
excludes, a relative operand is joined to the working directory as `pwd`
prints it, and an operand reached through a symlinked directory below
another operand is dropped (#144).

Dropped from "Constraints": the permission to add `github.com/spf13/viper`
if configuration-file support is ever needed.

Model: fable-5-1
2026-11-02 21:29:48 +00:00
3 changed files with 474 additions and 692 deletions

No files matched your search

+13 -16
View File
@@ -408,8 +408,8 @@ operands:
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A block is the unit in which a filesystem allocates disk space. A file write. A file scanned within those seconds gets its real `disk_size` at the
scanned within those seconds gets its real `disk_size` at the next scan. next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -777,11 +777,10 @@ first dupe files size disk_size
A file's `disk_size` is the space its blocks take on disk. It can be more than A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. A sparse file has gaps that take no file's size on a compressed or sparse file. So two copies of one tree can differ
space on disk. So two copies of one tree can differ in `disk_size`. When a file in `disk_size`. When a file in one copy is a hard link to one in the other, both
in one copy is a hard link to one in the other, both copies count its blocks in copies count its blocks in full. Removing it from one copy then frees nothing,
full. Removing it from one copy then frees nothing, as the limitation under as the limitation under "`trees` mode" says.
"`trees` mode" says.
The paths are escaped as described under "Report output format". The root The paths are escaped as described under "Report output format". The root
directory's path is `/`. directory's path is `/`.
@@ -942,8 +941,7 @@ Only one `scan` at a time can hold the lock.
#### Schema #### Schema
The schema is the database's layout: its tables, its index and its triggers. A The schema is the database's layout: its tables, its index and its triggers.
trigger is a statement SQLite runs by itself when a table changes.
`PRAGMA user_version` holds the schema version, currently 1. An empty database `PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error. is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal `report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
@@ -984,10 +982,9 @@ END;
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not - `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text. necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system - `disk_size` is the space the file's blocks take on disk, `st_blocks` × 512
call that stats a file without following a symlink. Its `st_blocks` field from lstat. It is less than `size` on a compressed or sparse file (see "Trees
counts that space in units of 512 bytes. `disk_size` is less than `size` on a output format").
compressed or sparse file (see "Trees output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and - `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any `mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -1005,9 +1002,9 @@ END;
`scan` and the `read` phase of `report` and `trees` read the number of records `scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers record. That takes minutes on a database of tens of millions. The triggers
make each insert, each delete and each change of a record's `size` slower, so make each write to `files` slower, so a first scan of many small files takes
a first scan of many small files takes about two fifths longer. On large about two fifths longer. On large files, reading them takes most of the time,
files, reading them takes most of the time, so the difference is small. so the difference is small.
### Progress ### Progress
+436 -602
View File
File diff suppressed because it is too large. Load diff
+25 -74
View File
@@ -43,16 +43,10 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
} }
// diskSizeOf returns the bytes the file at path takes on disk, from its // diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's // own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
// blocks only when it commits the write, up to seconds later, and one // blocks only seconds after the write, and fsync does not hasten that.
// commit can take the file through several growing sizes. So this value // A test therefore calls unix.Sync, which waits for it, before the scan
// can change at any moment. Neither fsync nor unix.Sync hastens that, // whose disk_size it compares with this.
// but a unix.Sync call often takes long enough that the blocks are
// counted when it returns. A test that needs the value to change waits
// for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
func diskSizeOf(t *testing.T, path string) int64 { func diskSizeOf(t *testing.T, path string) int64 {
t.Helper() t.Helper()
@@ -1454,26 +1448,20 @@ func TestScanStoresDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf unix.Sync() // see diskSizeOf
size := diskSizeOf(t, p) want := diskSizeOf(t, p)
if size == 0 || size >= length { if want == 0 || want >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+ t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", size, length) "and fewer than its length %d", want, length)
} }
for _, operand := range []string{filepath.Dir(p), p} { for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t) db := openTestDB(t)
// ZFS may count the blocks during the scan; see diskSizeOf.
before := diskSizeOf(t, p)
syncTree(t, db, operand) syncTree(t, db, operand)
after := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if got < before || got > after { if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d to %d", t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, before, after) operand, got, want)
} }
} }
} }
@@ -1491,7 +1479,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
syncTree(t, db, dir) syncTree(t, db, dir)
recorded := recordByPath(t, dbRecords(t, db), p).diskSize before := recordByPath(t, dbRecords(t, db), p).diskSize
pokeAt(t, p, 512*1024, pattern(7, 64*1024)) pokeAt(t, p, 512*1024, pattern(7, 64*1024))
@@ -1504,53 +1492,22 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// Wait for ZFS to count the write; see diskSizeOf. unix.Sync() // see diskSizeOf
deadline := time.Now().Add(30 * time.Second)
for diskSizeOf(t, p) == recorded {
if time.Now().After(deadline) {
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
"after the write into its hole", recorded)
}
unix.Sync() want := diskSizeOf(t, p)
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
} }
// ZFS may still be counting, so the size may change during a rescan. st := syncTree(t, db, dir)
// Only a rescan with the same size on both sides of it can be compared if st != (scanStats{walked: 1, updated: 1}) {
// exactly. A rescan that keeps the old size still fails, because the t.Fatalf("rescan stats = %+v, want 1 updated", st)
// wait made before differ from it. }
for {
before := diskSizeOf(t, p)
st := syncTree(t, db, dir) got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
after := diskSizeOf(t, p) t.Errorf("disk_size = %d, want %d", got, want)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if before == after {
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
} }
} }
@@ -1576,21 +1533,15 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf unix.Sync() // see diskSizeOf
// ZFS may count the blocks during the rescan; see diskSizeOf.
before := diskSizeOf(t, p)
// A file read again would count as updated. // A file read again would count as updated.
st := syncTree(t, db, dir) st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, unchanged: 1}) { if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st) t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
} }
got := recordByPath(t, dbRecords(t, db), p).diskSize got := recordByPath(t, dbRecords(t, db), p).diskSize
if got < before || got > after { if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d to %d", got, before, after) t.Errorf("disk_size = %d, want %d", got, want)
} }
} }