1 Commits
Author SHA1 Message Date
sneak 4ae7274bc0 Record each file's size on disk and print it in trees (closes #147)
check / check (push) Waiting to run
The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. When a rescan finds a file with its size
and mtime unchanged but a different disk_size, it updates disk_size
alone, without reading the file. trees adds disk_size up per directory
and prints it as a column after size. On a compressed or sparse file it
is less than size, so two copies of one tree can differ; each row shows
the dupe copy's.

Judgement calls: a row's disk_size is the dupe copy's, not the first's.
A file whose record gets only a new disk_size counts as unchanged.

Model: opus-5-5
2026-10-27 18:41:24 +00:00
6 changed files with 256 additions and 91 deletions
+27 -19
View File
@@ -226,12 +226,12 @@ Goals, in order:
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
stay in the database). The reporting commands do not hold every file's
hashes either: `report` reads the records in signature order and holds the
duplicate paths of one size at a time, so its memory grows with the
duplicates of the size that has the most of them, and `trees` reads the
under resilver). Holding one small record (path, size, size on disk, mtime)
per file in memory during a scan is acceptable; holding every file's hashes
is not (they stay in the database). The reporting commands do not hold every
file's hashes either: `report` reads the records in signature order and
holds the duplicate paths of one size at a time, so its memory grows with
the duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the
@@ -368,7 +368,8 @@ All four subcommands operate on a single SQLite database file:
- While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, and the deletions.
- In the update phase: the rest of the hashed records, the new `disk_size`
of unchanged files whose size on disk changed, and the deletions.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read
@@ -515,6 +516,11 @@ operands:
- A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap.
- When such a file's size on disk differs from the recorded `disk_size`, the
scan updates `disk_size` alone. It still does not read the file, and counts it
as unchanged. ZFS, for example, counts a new file's blocks only seconds after
the write. A file scanned within those seconds gets its real `disk_size` at
the next scan.
- A file whose mtime is newer than recorded, or whose size differs, is hashed
again. Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
@@ -542,8 +548,8 @@ inside the walk and hash phases, and batched database writes begin during the
hash phase:
1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size and its mtime, for the walk to compare
each file with.
in memory, by path, with its size, its size on disk and its mtime, for the
walk to compare each file with.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back
@@ -551,9 +557,10 @@ hash phase:
(while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk compares each file with its record as it goes, and
an unchanged file needs nothing more. Every new or changed file is carried
to the hash phase as a (path, size, size on disk, mtime) record.
or busy pools. The walk compares each file with its record as it goes. An
unchanged file is not read again. When its size on disk has changed, the
update phase writes its new `disk_size`. Every new or changed file is
carried to the hash phase as a (path, size, size on disk, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened.
@@ -565,7 +572,8 @@ hash phase:
committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
4. **update** — commit the final partial batch of hashed records, and the new
`disk_size` of each unchanged file whose size on disk changed. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
@@ -823,12 +831,12 @@ first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984
```
A file's `disk_size` is the space its blocks take on disk. It is less than the
file's size on a compressed or sparse file, and a little more on most other
files. Two copies of one tree can therefore differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the known limitation
under "`trees` mode" says.
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file, so two copies of one tree can differ
in `disk_size`. When a file in one copy is a hard link to one in the other, both
copies count its blocks in full. Removing it from one copy then frees nothing,
as the known limitation under "`trees` mode" says.
The `first` and `dupe` paths are escaped as described under "Report output
format". The root directory's path is `/`.
+3 -2
View File
@@ -32,8 +32,9 @@
- each record holds `disk_size`, the bytes the file takes on disk: its allocated
blocks, `st_blocks` × 512, from the lstat the walk already does. `trees`
prints a `disk_size` column after `size`, the sum over the `dupe` copy's
files; on a compressed or sparse file it differs from `size` (2026-10-27,
https://git.eeqj.de/sneak/sfdupes/issues/147)
files; on a compressed or sparse file it differs from `size`. A rescan updates
the `disk_size` of an unchanged file whose size on disk changed, without
reading the file (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/147)
- the database keeps the number of records and the sum of their sizes in the one
row of a `totals` table, which three triggers on `files` keep current. `info`
+67 -18
View File
@@ -729,15 +729,15 @@ func readDatabaseCounts(ctx context.Context,
return c, nil
}
// loadFileMeta streams every record's path, size and mtime to fn. Scan
// change detection needs no hash values, and skipping the hash columns
// keeps the scan's in-memory index small on multi-million-file
// databases.
// loadFileMeta streams every record's path, size, disk_size and mtime
// to fn. Scan change detection needs no hash values, and skipping the
// hash columns keeps the scan's in-memory index small on
// multi-million-file databases.
func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, size int64, mtime time.Time),
fn func(path string, m fileMeta),
) error {
rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec FROM files")
"SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
if err != nil {
return fmt.Errorf("read records: %w", err)
}
@@ -746,16 +746,19 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() {
var (
path []byte
size, sec, nsec int64
path []byte
m fileMeta
sec, nsec int64
)
err = rows.Scan(&path, &size, &sec, &nsec)
err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
if err != nil {
return fmt.Errorf("read record: %w", err)
}
fn(string(path), size, time.Unix(sec, nsec))
m.mtime = time.Unix(sec, nsec)
fn(string(path), m)
}
err = rows.Err()
@@ -773,21 +776,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// WAL small and let concurrent reports observe progress.
const updateBatchSize = 10000
// applyChanges writes one scan's database changes — upserts for new and
// changed files, deletes for vanished ones — in batched transactions.
// Progress is rendered on prog (one increment per change).
// applyChanges writes upserts for new and changed files and deletes for
// vanished ones, in batched transactions. Progress is rendered on prog
// (one increment per change).
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
deletes []string, prog *progress,
) error {
for batch := range slices.Chunk(upserts, updateBatchSize) {
err := applyBatch(ctx, db, batch, nil, prog)
err := applyBatch(ctx, db, batch, nil, nil, prog)
if err != nil {
return err
}
}
for batch := range slices.Chunk(deletes, updateBatchSize) {
err := applyBatch(ctx, db, nil, batch, prog)
err := applyBatch(ctx, db, nil, nil, batch, prog)
if err != nil {
return err
}
@@ -796,10 +799,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
return nil
}
// applyBatch commits one batch of upserts and deletes in a single
// transaction.
// applyDiskSizes writes the disk_size of each of recs into its record,
// in batched transactions, and leaves the rest of the record alone.
// Progress is rendered on prog (one increment per record).
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
prog *progress,
) error {
for batch := range slices.Chunk(recs, updateBatchSize) {
err := applyBatch(ctx, db, nil, batch, nil, prog)
if err != nil {
return err
}
}
return nil
}
// applyBatch commits one batch of upserts, disk_size updates and
// deletes in a single transaction.
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
deletes []string, prog *progress,
diskSizes []fileRec, deletes []string, prog *progress,
) error {
tx, err := db.BeginTx(ctx, nil)
if err != nil {
@@ -813,6 +832,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
return err
}
err = execDiskSizes(ctx, tx, diskSizes, prog)
if err != nil {
return err
}
err = execDeletes(ctx, tx, deletes, prog)
if err != nil {
return err
@@ -851,6 +875,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
return nil
}
// execDiskSizes sets the disk_size of the record of each of recs. It
// does not touch size, so the totals triggers do not fire.
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
prog *progress,
) error {
st, err := tx.PrepareContext(ctx,
"UPDATE files SET disk_size = ? WHERE path = ?")
if err != nil {
return fmt.Errorf("prepare disk_size update: %w", err)
}
defer func() { _ = st.Close() }()
for _, r := range recs {
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
if err != nil {
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
}
prog.increment()
}
return nil
}
// execDeletes removes the records for paths no longer present.
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
prog *progress,
+23 -4
View File
@@ -818,6 +818,23 @@ func scanFixture(t *testing.T) []string {
return dupes
}
// storedDiskSize returns the disk_size of the record for path in the
// database at dbPath. A test of a trees run compares with this rather
// than with a new lstat, which on ZFS can differ from the scan's (see
// diskSizeOf).
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
t.Helper()
db, err := openReportDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
return recordByPath(t, dbRecords(t, db), path).diskSize
}
// scanOK runs scan over operands, fails the test unless it exits 0 with
// nothing on stdout, and returns everything it printed to stderr.
func scanOK(t *testing.T, operands ...string) string {
@@ -1143,7 +1160,7 @@ func TestRunTreesSucceeds(t *testing.T) {
// trees of each other.
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want)
}
@@ -1179,8 +1196,8 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
dupes[0] + "\t" + dupes[1] + "\t300\n",
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
"\t1\t300\t" + strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) +
"\n",
"\t1\t300\t" +
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
}
for name, want := range cases {
@@ -1215,9 +1232,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Errorf("report stdout = %q, want %q", got, want)
}
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
strconv.FormatInt(diskSizeOf(t, dupes[1]), 10) + "\n"
strconv.FormatInt(diskSize, 10) + "\n"
if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want)
}
+42 -27
View File
@@ -66,11 +66,12 @@ type fileRec struct {
}
// fileMeta is the in-memory index entry for one existing database
// record: just enough for change detection. Hashes stay on disk; at
// record: what the walk compares the file with. Hashes stay on disk; at
// tens of millions of records they would dominate the scan's memory.
type fileMeta struct {
size int64
mtime time.Time
size int64
diskSize int64
mtime time.Time
}
// scanUnit is what every scan phase counts, as its progress display
@@ -341,11 +342,12 @@ type scanStats struct {
// whatever remains after the walk and hash phases is deleted by the
// update phase.
type scanState struct {
db *sql.DB
existing map[string]fileMeta
toHash []fileRec // new and changed files: must be read
batch []scanRec
st scanStats
db *sql.DB
existing map[string]fileMeta
toHash []fileRec // new and changed files: must be read
diskSizes []fileRec // unchanged files whose disk_size changed
batch []scanRec
st scanStats
}
// syncScan synchronizes the database with the filesystem under roots;
@@ -474,24 +476,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db,
func(path string, size int64, mtime time.Time) {
prog.increment()
return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
prog.increment()
if underAnyRoot(path, roots) {
s.existing[path] = fileMeta{size: size, mtime: mtime}
}
})
if underAnyRoot(path, roots) {
s.existing[path] = m
}
})
}
// walkPhase drains the walk and queues every new or changed file for
// the hash phase. An unchanged file needs nothing more, and its record
// counts as verified. A record at or below a path the walk left out,
// because -x stopped at it, it is a default exclude, or it is not on a
// persistent filesystem, counts as outside the roots: it is not
// deleted. That does not hold below a root at or inside that path,
// since that root's own walk verifies the records below it; nor is a
// line printed for such a root.
// the hash phase. An unchanged file is not read, and its record counts
// as verified. When its disk_size differs from the record's, it is
// queued for the update phase, which writes the new disk_size. A record
// at or below a path the walk left out, because -x stopped at it, it is
// a default exclude, or it is not on a persistent filesystem, counts as
// outside the roots: it is not deleted. That does not hold below a root
// at or inside that path, since that root's own walk verifies the
// records below it; nor is a line printed for such a root.
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The records the last scan left under the roots estimate how many
// files the walk will find. A first scan has none, and no estimate.
@@ -549,6 +551,13 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
delete(s.existing, ev.rec.path)
s.st.unchanged++
// The space a file takes on disk can change while its size and
// mtime stay the same. ZFS, for one, counts a new file's blocks
// only seconds after the write.
if old.diskSize != ev.rec.diskSize {
s.diskSizes = append(s.diskSizes, ev.rec)
}
}
prog.finish()
@@ -719,7 +728,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
return nil
}
err := applyBatch(ctx, s.db, s.batch, nil, nil)
err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
if err != nil {
return err
}
@@ -730,9 +739,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
}
// updatePhase writes the scan's tail under one progress display: the
// final partial batch of hashed records, and deletions for every record
// the scan did not verify (vanished files, plus paths that failed to
// stat or hash).
// final partial batch of hashed records, the new disk_size of every
// unchanged file whose disk_size changed, and deletions for every
// record the scan did not verify (vanished files, plus paths that
// failed to stat or hash).
func (s *scanState) updatePhase(ctx context.Context) error {
deletes := make([]string, 0, len(s.existing))
for path := range s.existing {
@@ -744,7 +754,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.st.removed = len(deletes)
total := len(s.batch) + len(deletes)
total := len(s.batch) + len(s.diskSizes) + len(deletes)
prog := newProgress(progressParams{
label: scanPhase("update"), unit: scanUnit, total: int64(total),
stalledOn: "db",
@@ -759,6 +769,11 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.batch = nil
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
if err != nil {
return err
}
return applyChanges(ctx, s.db, nil, deletes, prog)
}
+94 -21
View File
@@ -43,7 +43,10 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
}
// diskSizeOf returns the bytes the file at path takes on disk, from its
// own lstat: st_blocks counts 512-byte units.
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
// blocks only seconds after the write, and fsync does not hasten that.
// A test therefore calls unix.Sync, which waits for it, before the scan
// whose disk_size it compares with this.
func diskSizeOf(t *testing.T, path string) int64 {
t.Helper()
@@ -1432,45 +1435,115 @@ func TestScanStoresRawHashes(t *testing.T) {
}
// TestScanStoresDiskSize scans a sparse file, which takes less space on
// disk than its length, and checks the disk_size stored for it against
// the file's own lstat.
// disk than its length, once through its directory and once as the
// operand itself. Each scan stores the disk_size of the file's lstat.
func TestScanStoresDiskSize(t *testing.T) {
t.Parallel()
const length = 1024 * 1024
p := sparseFile(t, t.TempDir(), "sparse", length)
pokeAt(t, p, 0, []byte("x"))
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == 0 || want >= length {
t.Fatalf("the file takes %d bytes on disk, want more than none "+
"and fewer than its length %d", want, length)
}
for _, operand := range []string{filepath.Dir(p), p} {
db := openTestDB(t)
syncTree(t, db, operand)
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("scan of %s: disk_size = %d, want %d",
operand, got, want)
}
}
}
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
// sparse file and moves its mtime on. The rescan hashes the file again
// and rewrites its record with the new disk_size.
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "sparse", []byte("x"))
// Truncate makes the file longer without giving it blocks for the
// new bytes.
const length = 1024 * 1024
err := os.Truncate(p, length)
if err != nil {
t.Fatal(err)
}
p := sparseFile(t, dir, "sparse", 1024*1024)
pokeAt(t, p, 0, []byte("x"))
syncTree(t, db, dir)
want := diskSizeOf(t, p)
if want >= length {
t.Fatalf("the file takes %d bytes on disk, want fewer than its "+
"length %d", want, length)
}
before := recordByPath(t, dbRecords(t, db), p).diskSize
var got int64
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
err = db.QueryRowContext(t.Context(),
"SELECT disk_size FROM files WHERE path = ?", []byte(p)).Scan(&got)
// The write may leave the mtime the scan recorded; a later one makes
// the rescan read the file again.
later := time.Now().Add(time.Hour)
err := os.Chtimes(p, later, later)
if err != nil {
t.Fatal(err)
}
unix.Sync() // see diskSizeOf
want := diskSizeOf(t, p)
if want == before {
t.Fatalf("the file still takes %d bytes on disk after the write "+
"into its hole", want)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
// disk_size the file does not have, as when ZFS had not yet counted the
// file's blocks at the last scan. The rescan writes the file's own
// disk_size into the record without reading the file again.
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
p := writeFile(t, dir, "a.bin", pattern(1, 500))
syncTree(t, db, dir)
// lstat counts 512-byte units, so no file takes 1 byte on disk.
_, err := db.ExecContext(t.Context(),
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
if err != nil {
t.Fatal(err)
}
unix.Sync() // see diskSizeOf
// A file read again would count as updated.
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
}
got := recordByPath(t, dbRecords(t, db), p).diskSize
if want := diskSizeOf(t, p); got != want {
t.Errorf("disk_size = %d, want %d", got, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file. The operand is