Compare commits

..
Author SHA1 Message Date
sneak c75421fd11 Carry the README changes from next into the rewrite
check / check (push) Canceled after 0s
`next` changed the old README after this branch was cut, in
#146 (the `totals` table, commit
98193b2) and #147 (`disk_size`,
commit 4ed2e37). This commit states the same facts in the rewritten
README, in the places #140 (comment)
lists, and changes nothing else. Every statement was checked against the
code on `next`.

Model: fable-5-1
2026-11-02 21:47:23 +00:00
sneak d8b58bf281 Rewrite README.md sentence by sentence for a newcomer
Rewrites `README.md` from scratch for
#135. Every statement was checked
against the code on `next` and against what the built binary prints on
small trees under `/var/tmp`. The headings that the code comments cite
are kept.

Where the old text and the repository disagreed, the new text states
what the code does. #140 lists
every such point. Among them: `@latest` installs the newest tagged
release, ten million files in both "Rationale" and "Design", no Windows
excludes, a relative operand is joined to the working directory as `pwd`
prints it, and an operand reached through a symlinked directory below
another operand is dropped (#144).

Dropped from "Constraints": the permission to add `github.com/spf13/viper`
if configuration-file support is ever needed.

Model: fable-5-1
2026-11-02 21:29:48 +00:00
3 changed files with 474 additions and 692 deletions

No files matched your search

+13 -16
View File
@@ -408,8 +408,8 @@ operands:
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A block is the unit in which a filesystem allocates disk space. A file
scanned within those seconds gets its real `disk_size` at the next scan.
write. A file scanned within those seconds gets its real `disk_size` at the
next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -777,11 +777,10 @@ first dupe files size disk_size
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. A sparse file has gaps that take no
space on disk. So two copies of one tree can differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the limitation under
"`trees` mode" says.
file's size on a compressed or sparse file. So two copies of one tree can differ
in `disk_size`. When a file in one copy is a hard link to one in the other, both
copies count its blocks in full. Removing it from one copy then frees nothing,
as the limitation under "`trees` mode" says.
The paths are escaped as described under "Report output format". The root
directory's path is `/`.
@@ -942,8 +941,7 @@ Only one `scan` at a time can hold the lock.
#### Schema
The schema is the database's layout: its tables, its index and its triggers. A
trigger is a statement SQLite runs by itself when a table changes.
The schema is the database's layout: its tables, its index and its triggers.
`PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
@@ -984,10 +982,9 @@ END;
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
call that stats a file without following a symlink. Its `st_blocks` field
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
compressed or sparse file (see "Trees output format").
- `disk_size` is the space the file's blocks take on disk, `st_blocks` × 512
from lstat. It is less than `size` on a compressed or sparse file (see "Trees
output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -1005,9 +1002,9 @@ END;
`scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers
make each insert, each delete and each change of a record's `size` slower, so
a first scan of many small files takes about two fifths longer. On large
files, reading them takes most of the time, so the difference is small.
make each write to `files` slower, so a first scan of many small files takes
about two fifths longer. On large files, reading them takes most of the time,
so the difference is small.
### Progress
+436 -602
View File
File diff suppressed because it is too large. Load diff
+25 -74
View File
@@ -43,16 +43,10 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
}
// diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
// blocks only when it commits the write, up to seconds later, and one
// commit can take the file through several growing sizes. So this value
// can change at any moment. Neither fsync nor unix.Sync hastens that,
// but a unix.Sync call often takes long enough that the blocks are
// counted when it returns. A test that needs the value to change waits
// for the change. A test that compares it with a scan's disk_size reads
// it before and after the scan. The rewrite test rescans until the two
// reads match, then compares exactly. The other two tests fail only on
// a disk_size outside the two reads.
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
// blocks only seconds after the write, and fsync does not hasten that.
// A test therefore calls unix.Sync, which waits for it, before the scan
// whose disk_size it compares with this.
func diskSizeOf(t *testing.T, path string) int64 {
t.Helper()
@@ -1454,26 +1448,20 @@ func TestScanStoresDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf
size := diskSizeOf(t, p)
if size == 0 || size >= length {
want := diskSizeOf(t, p)
if want == 0 || want >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", size, length)
"and fewer than its length %d", want, length)
}
for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t)
// ZFS may count the blocks during the scan; see diskSizeOf.
before := diskSizeOf(t, p)
syncTree(t, db, operand)
after := diskSizeOf(t, p)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got < before || got > after {
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
operand, got, before, after)
if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, want)
}
}
}
@@ -1491,7 +1479,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
syncTree(t, db, dir)
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
before := recordByPath(t, dbRecords(t, db), p).diskSize
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
@@ -1504,53 +1492,22 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Fatal(err)
}
// Wait for ZFS to count the write; see diskSizeOf.
deadline := time.Now().Add(30 * time.Second)
for diskSizeOf(t, p) == recorded {
if time.Now().After(deadline) {
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
"after the write into its hole", recorded)
}
unix.Sync() // see diskSizeOf
unix.Sync()
want := diskSizeOf(t, p)
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
}
// ZFS may still be counting, so the size may change during a rescan.
// Only a rescan with the same size on both sides of it can be compared
// exactly. A rescan that keeps the old size still fails, because the
// wait made before differ from it.
for {
before := diskSizeOf(t, p)
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
if before == after {
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != after {
t.Errorf("disk_size = %d, want %d", got, after)
}
return
}
if time.Now().After(deadline) {
t.Fatalf("the size on disk was still changing 30 seconds after "+
"the write into its hole: %d, then %d", before, after)
}
// A later mtime makes the next rescan read the file again.
later = later.Add(time.Second)
err = os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
@@ -1576,21 +1533,15 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
unix.Sync() // see diskSizeOf
// ZFS may count the blocks during the rescan; see diskSizeOf.
before := diskSizeOf(t, p)
// A file read again would count as updated.
st := syncTree(t, db, dir)
after := diskSizeOf(t, p)
if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got < before || got > after {
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}