Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a00b97e299 | ||
|
|
30dae35c51 | ||
|
|
4ecf8dc484 | ||
|
|
2322f8bb02 |
No files matched your search
@@ -407,9 +407,9 @@ operands:
|
||||
is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
|
||||
write. A file scanned within those seconds gets its real `disk_size` at the
|
||||
next scan.
|
||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
||||
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||
@@ -777,10 +777,11 @@ first dupe files size disk_size
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It can be more than
|
||||
the file's size, because space is allocated in whole blocks. It is less than the
|
||||
file's size on a compressed or sparse file. So two copies of one tree can differ
|
||||
in `disk_size`. When a file in one copy is a hard link to one in the other, both
|
||||
copies count its blocks in full. Removing it from one copy then frees nothing,
|
||||
as the limitation under "`trees` mode" says.
|
||||
file's size on a compressed or sparse file. A sparse file has gaps that take no
|
||||
space on disk. So two copies of one tree can differ in `disk_size`. When a file
|
||||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||
full. Removing it from one copy then frees nothing, as the limitation under
|
||||
"`trees` mode" says.
|
||||
|
||||
The paths are escaped as described under "Report output format". The root
|
||||
directory's path is `/`.
|
||||
@@ -941,16 +942,17 @@ Only one `scan` at a time can hold the lock.
|
||||
|
||||
#### Schema
|
||||
|
||||
The schema is the database's layout, its table and its index.
|
||||
The schema is the database's layout: its tables, its index and its triggers. A
|
||||
trigger is a statement SQLite runs by itself when a table changes.
|
||||
`PRAGMA user_version` holds the schema version, currently 1. An empty database
|
||||
is at version 0, and `scan` sets it up. Any other version is a fatal error.
|
||||
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
|
||||
error too. `scan` creates the table and the index and sets the version in one
|
||||
transaction (see "`info` mode"). So the three land together or not at all. A
|
||||
first scan stopped while doing so leaves an empty database, which the next scan
|
||||
sets up. A database at version 0 that already has a `files` table was not made
|
||||
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
|
||||
the file and rescan.
|
||||
error too. `scan` creates the tables, the index and the triggers and sets the
|
||||
version in one transaction (see "`info` mode"). So they all land together or not
|
||||
at all. A first scan stopped while doing so leaves an empty database, which the
|
||||
next scan sets up. A database at version 0 that already has a `files` table was
|
||||
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
|
||||
remove the file and rescan.
|
||||
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
@@ -982,9 +984,10 @@ END;
|
||||
|
||||
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
|
||||
necessarily UTF-8 text.
|
||||
- `disk_size` is the space the file's blocks take on disk, `st_blocks` × 512
|
||||
from lstat. It is less than `size` on a compressed or sparse file (see "Trees
|
||||
output format").
|
||||
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
|
||||
call that stats a file without following a symlink. Its `st_blocks` field
|
||||
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
|
||||
compressed or sparse file (see "Trees output format").
|
||||
- `mtime` holds the whole seconds of the file's mtime, rounded down, and
|
||||
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any
|
||||
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
|
||||
@@ -1002,9 +1005,9 @@ END;
|
||||
`scan` and the `read` phase of `report` and `trees` read the number of records
|
||||
from it as their total. Counting the records in `files` would read every
|
||||
record. That takes minutes on a database of tens of millions. The triggers
|
||||
make each write to `files` slower, so a first scan of many small files takes
|
||||
about two fifths longer. On large files, reading them takes most of the time,
|
||||
so the difference is small.
|
||||
make each insert, each delete and each change of a record's `size` slower, so
|
||||
a first scan of many small files takes about two fifths longer. On large
|
||||
files, reading them takes most of the time, so the difference is small.
|
||||
|
||||
### Progress
|
||||
|
||||
|
||||
@@ -554,7 +554,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
|
||||
// The space a file takes on disk can change while its size and
|
||||
// mtime stay the same. ZFS, for one, counts a new file's blocks
|
||||
// only seconds after the write.
|
||||
// up to seconds after the write.
|
||||
if old.diskSize != ev.rec.diskSize {
|
||||
s.diskSizes = append(s.diskSizes, ev.rec)
|
||||
}
|
||||
|
||||
+74
-25
@@ -43,10 +43,16 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
||||
}
|
||||
|
||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
|
||||
// blocks only seconds after the write, and fsync does not hasten that.
|
||||
// A test therefore calls unix.Sync, which waits for it, before the scan
|
||||
// whose disk_size it compares with this.
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
|
||||
// blocks only when it commits the write, up to seconds later, and one
|
||||
// commit can take the file through several growing sizes. So this value
|
||||
// can change at any moment. Neither fsync nor unix.Sync hastens that,
|
||||
// but a unix.Sync call often takes long enough that the blocks are
|
||||
// counted when it returns. A test that needs the value to change waits
|
||||
// for the change. A test that compares it with a scan's disk_size reads
|
||||
// it before and after the scan. The rewrite test rescans until the two
|
||||
// reads match, then compares exactly. The other two tests fail only on
|
||||
// a disk_size outside the two reads.
|
||||
func diskSizeOf(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
@@ -1448,20 +1454,26 @@ func TestScanStoresDiskSize(t *testing.T) {
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == 0 || want >= length {
|
||||
size := diskSizeOf(t, p)
|
||||
if size == 0 || size >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want more than none "+
|
||||
"and fewer than its length %d", want, length)
|
||||
"and fewer than its length %d", size, length)
|
||||
}
|
||||
|
||||
for _, operand := range []string{filepath.Dir(p), p} {
|
||||
db := openTestDB(t)
|
||||
|
||||
// ZFS may count the blocks during the scan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
syncTree(t, db, operand)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d",
|
||||
operand, got, want)
|
||||
if got < before || got > after {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
|
||||
operand, got, before, after)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1479,7 +1491,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
before := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
|
||||
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
|
||||
|
||||
@@ -1492,22 +1504,53 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
// Wait for ZFS to count the write; see diskSizeOf.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for diskSizeOf(t, p) == recorded {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
|
||||
"after the write into its hole", recorded)
|
||||
}
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == before {
|
||||
t.Fatalf("the file still takes %d bytes on disk after the write "+
|
||||
"into its hole", want)
|
||||
unix.Sync()
|
||||
}
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
// ZFS may still be counting, so the size may change during a rescan.
|
||||
// Only a rescan with the same size on both sides of it can be compared
|
||||
// exactly. A rescan that keeps the old size still fails, because the
|
||||
// wait made before differ from it.
|
||||
for {
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
if before == after {
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != after {
|
||||
t.Errorf("disk_size = %d, want %d", got, after)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the size on disk was still changing 30 seconds after "+
|
||||
"the write into its hole: %d, then %d", before, after)
|
||||
}
|
||||
|
||||
// A later mtime makes the next rescan read the file again.
|
||||
later = later.Add(time.Second)
|
||||
|
||||
err = os.Chtimes(p, later, later)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1533,15 +1576,21 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
// ZFS may count the blocks during the rescan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
// A file read again would count as updated.
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, unchanged: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
|
||||
}
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if want := diskSizeOf(t, p); got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
if got < before || got > after {
|
||||
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user