Stop the disk size tests failing at random on ZFS (closes #150)

One ZFS commit can take a file through several sizes, and unix.Sync can
return before the blocks are counted. A test that expected the size on
disk to be settled therefore failed at random. The test that needs the
size to change now calls unix.Sync in a loop until it does, for at most
30 seconds. It then rescans until the size reads the same before and
after a rescan, and compares disk_size exactly, within the same 30
seconds. The other two tests fail only on a disk_size outside their two
reads. They call unix.Sync once first, so a wrong 512 usually fails them.

Model: opus-5-5

README.md

@@ -18,11 +18,11 @@ a strong candidate, not proof.
 
 `sfdupes` is meant for multi-terabyte servers, where reading every byte of every
 file takes too long. It has four subcommands. `scan` walks the filesystem and
-records every file in an SQLite database. SQLite keeps a whole database in one
+records the files it finds in an SQLite database. SQLite keeps a whole database
-file. The database is kept between runs. A rescan reads only new and changed
+in one file. The database is kept between runs. A rescan reads only new and
-files. That makes `scan` cheap enough to run every day from `cron`, the system's
+changed files. That makes `scan` cheap enough to run every day from `cron`, the
-job scheduler. `report`, `trees` and `info` read the database alone and can run
+system's job scheduler. `report`, `trees` and `info` read the database alone and
-at any time.
+can run at any time.
 
 ## Getting Started
 
@@ -38,17 +38,18 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
 `scan` walks the trees named by its operands, the paths given after `scan` on
 the command line. The walk is the listing of every directory under them. `scan`
 keeps one database record per regular file. A regular file is an ordinary file,
-as opposed to a directory, a symlink or a device. `report` prints the file-level
+as opposed to a directory, a symlink or a device node. A symlink is a file that
-duplicates. `trees` prints the duplicate directory trees. `info` prints the
+holds the path of another file and stands in for it. A device node is a file
-database's path, size and record count.
+that stands for a piece of hardware, such as a disk or a terminal. `report`
+prints the file-level duplicates. `trees` prints the duplicate directory trees.
+`info` prints the database's path, size and record count.
 
 The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
 another file. The first `scan` creates it.
 
-`report`, `trees` and `info` write their data to stdout. Everything else, such
+`report`, `trees` and `info` write their data to stdout. Progress, warnings and
-as progress and warnings, goes to stderr. stdout and stderr are a program's two
+errors go to stderr. stdout and stderr are a program's two output streams. The
-output streams. The shell shows both on the terminal, and `>` redirects stdout
+shell shows both on the terminal, and `>` redirects stdout alone.
-alone.
 
 ### Install
 
@@ -68,11 +69,12 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
 
 `sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
 `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
-a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
+a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
-string for an untagged commit. That string ends in the commit's hash. A binary
+one. Otherwise it is a pseudo-version, Go's version string for an untagged
-built from a clone carries the version `git describe --tags --always --dirty`
+commit. That string ends in the commit's hash. A binary built from a clone
-prints in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev`
+carries the output of `git describe --tags --always --dirty` in that clone, such
-when no version is known. To see everything Go recorded in an installed binary:
+as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
+see everything Go recorded in an installed binary:
 
 ```sh
 go version -m "$(command -v sfdupes)"
@@ -106,9 +108,10 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
 
 ### Daily scan from `cron`
 
-Run `scan` as root, so that it can read every file. A path it cannot read is
+Run `scan` as root, so that it can read every file. `scan` warns about a
-skipped with a warning and loses its database record. As the file
+directory it cannot read, and about a new or changed file it cannot read. It
-`/etc/cron.d/sfdupes`:
+skips the path and deletes the records at or below it (see "Rules for the
+walk"). As the file `/etc/cron.d/sfdupes`:
 
 ```
 30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
@@ -122,8 +125,11 @@ skipped with a warning and loses its database record. As the file
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each path the walk leaves out as a system or temporary
+    - a line for each system or temporary directory the walk leaves out (see
-      directory (see "Rules for the walk");
+      "Rules for the walk");
+    - a line for each path left out because its filesystem is one the kernel,
+      the core of the operating system, makes up or keeps in memory, such as
+      `tmpfs`;
     - the summary line;
     - a warning when a report still has the database open as the scan ends (see
       "Concurrent reads").
@@ -133,8 +139,8 @@ skipped with a warning and loses its database record. As the file
 - A scan that fails exits 1, with the reason among the last lines of the log.
   `tail` prints those lines, and `cron` mails them to root if the host can send
   mail.
-- A scan still running when the next one starts carries on. The new one fails at
+- A running scan carries on when the next one starts. The new one fails at once,
-  once, and the lines `cron` mails include
+  and the lines that `cron` mails include
   `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
 - `report`, `trees` and `info` need only read access to the database. The umask
   is the setting that removes permissions from the files a process creates.
@@ -152,8 +158,8 @@ to keep is your choice.
 
 A row is a candidate, not proof:
 
-- The reports show the files as they were at the last scan. A file may have
+- The reports show each file as it was when a scan last recorded it. A file may
-  changed or gone since.
+  have changed or gone since.
 - A file of 100 MiB or more is compared only on samples of its content (see
   "Duplicate detection").
 - Two paths that are hard links, two names for one file, are listed as
@@ -172,8 +178,8 @@ where the files differ. The second line prints `hard links` when both paths are
 the same file.
 
 A path holding a special character, such as a tab, is escaped in the reports.
-The character is written as a backslash and a letter, `\t` for a tab (see
+The character is written as a backslash and one more character, `\t` for a tab
-"Report output format"). Undo the escapes before using such a path.
+(see "Report output format"). Undo the escapes before using such a path.
 `printf '%b'` does that, because every backslash in an escaped path starts an
 escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
 newline at the end of a path, print an `x` after the path and remove it
@@ -200,9 +206,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
 makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
 of reads, at the price of a comparison that is not exact.
 
-The hashes live in a database that is kept between runs. A rescan reads only the
+The hashes live in a database that is kept between runs. A rescan reads only new
-files that are new or whose size or modification time (mtime) changed. All
+files, and files whose size changed or whose modification time (mtime) is newer
-analysis then works from the database alone.
+than recorded. All analysis then works from the database alone.
 
 The goal is whole duplicated trees, not only single files. A download extracted
 twice, or a project copied once too often, is such a tree. It can be removed as
@@ -223,10 +229,9 @@ In order:
    its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
    in full. The target is a filesystem of about ten million files and 150 TB, on
    disks that may be slow or busy. During a scan, one small record per file is
-   held in memory. It holds the path and the two values the walk compares, the
+   held in memory. The hashes stay in the database. `report` holds the duplicate
-   size and the mtime. The hashes stay in the database. `report` holds the
+   paths of one file size at a time. `trees` holds one entry per directory, plus
-   duplicate paths of one file size at a time. `trees` holds one entry per
+   the files of the directories it is reading.
-   directory, plus the files of the directories it is reading.
 3. **Rescan cheaply, analyze from the database.** The scan keeps its database
    between runs, and an unchanged file is never read again. `report`, `trees`
    and `info` work from the database alone and never touch the scanned
@@ -260,11 +265,14 @@ The reference below is also the program's specification.
 
 ## Non-goals
 
-- No byte-for-byte compare, and no deleting or linking of duplicates. The
+- No byte-for-byte compare.
-  reports are advisory. Acting on them is your job.
+- No deleting of duplicates.
+- No replacing a copy with a hard link to another copy. The reports are
+  advisory. Acting on them is your job.
 - The SQLite database is the only stored data. There are no export or import
   formats.
-- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
+- `sfdupes` does not run in the background or watch the filesystem for changes.
+  Scheduling rescans is `cron`'s job.
 
 ## Reference
 
@@ -279,9 +287,9 @@ The reference below is also the program's specification.
   hashes them. The default is the number of CPUs. `N` must be at least 1. A
   smaller value is a usage error, reported in one line on stderr with exit 2
   before anything is scanned.
-- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
+- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
-  operand's filesystem, as the same flag does for `du` and `rsync`. By default
+  directories on other filesystems, as the same flag does for `du` and `rsync`.
-  the walk also enters directories on other filesystems.
+  By default the walk also enters directories on other filesystems.
 - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
   Those are the system and temporary directories the walk otherwise leaves out,
   such as `/var/tmp` on Linux (see "Rules for the walk").
@@ -342,31 +350,50 @@ default. An operand may be a directory or a regular file.
   the exit code is 2.
 - An operand that does not exist is a fatal error. `scan` then stops before it
   walks anything, and exits 1.
-- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
+- Each operand is made absolute before the walk. A relative operand is joined to
-  as text, and symlinks are not followed. Each record is stored under that
+  the working directory as `pwd` prints it, with its symlinks kept. A `.` or
-  absolute path, so results do not depend on the working directory.
+  `..` in it is resolved as text, and symlinks are not followed. Each record is
+  stored under that absolute path.
 - All operands belong to one scan, and one walk covers them all at once (see
   "Concurrency").
-- An operand that repeats another, or lies under another, is dropped before the
+- An operand that repeats another is dropped before the walk.
-  walk, without a line. Every file is then reached once. An operand under
+- An operand under another operand is dropped the same way, unless one of the
-  another is kept when the outer operand's walk would not reach it. That is the
+  cases below applies. No path is then walked twice.
-  case when the inner operand, or a directory between the two, is one of these:
+- An operand under another is kept, and walked on its own, when the inner
+  operand or a directory between the two is one of these:
     - a default exclude (see "Rules for the walk");
     - with `-x`, a directory on another filesystem than the outer operand.
 
-    It is also the case when a directory between the two is on a filesystem that
+    It is also kept when a directory between the two is on a filesystem that is
-    is not persistent. Such a filesystem is one the kernel makes up or keeps in
+    not persistent. Such a filesystem is one the kernel makes up or keeps in
     memory (see "Rules for the walk"). The outer operand's walk does not enter
     such a directory, so the inner operand is walked on its own.
 
-- An operand that is neither a regular file nor a directory is not scanned, and
+- No other case keeps an operand under another. An operand reached through a
-  neither is a directory named `.zfs`. `scan` prints a warning naming the path
+  symlinked directory below the outer operand is dropped too. The outer
-  and what it is. It counts the operand as skipped and drops it. An operand
+  operand's walk does not follow the symlink, so no file under the inner operand
-  beneath a dropped one is scanned all the same. The scan keeps the records
+  is scanned. The records that an earlier scan left under it are deleted. The
-  beneath a dropped operand, unless another operand covers them.
+  `-x` case above also covers an operand reached that way.
+- An operand that is neither a regular file nor a directory is not scanned.
+  `scan` prints a warning naming the path and what it is. It counts the operand
+  as skipped and drops it. An operand beneath a symlink operand is scanned all
+  the same. The scan keeps the records beneath a dropped operand, unless another
+  operand covers them. After a `cd` through a symlink, `scan .` names the
+  symlink and is skipped as a symlink operand. `scan sub` in that directory
+  records its files under the symlink's path. `report` can then list each as a
+  duplicate of the same file recorded under its real path.
+- An operand at or inside a directory named `.zfs` is never scanned, whether
+  given alone or under another operand. The check looks at the path as given and
+  at the path with its symlinks resolved. `scan` prints a warning naming the
+  path, counts the operand as skipped and drops it. The records beneath it are
+  kept, unless it lies under another operand. Then they are deleted, because
+  that operand's walk never enters `.zfs` (see "Rules for the walk").
 - An operand on a filesystem that is not persistent is dropped too. `scan`
   prints a line naming it and its filesystem type, and does not count it as
   skipped.
+- `scan` prints these warnings, and the line for an operand that is not on a
+  persistent filesystem, even when the operand repeats another or lies under
+  another.
 - A scan whose every operand is dropped walks nothing and exits 0.
 
 #### What a scan does to the database
@@ -378,15 +405,21 @@ operands:
 - A file already in the database is not read again when its size and mtime match
   its record. The mtime matches when it is not newer than the recorded one. This
   is what makes a daily rescan cheap.
-- A file whose size differs, or whose mtime is newer, is hashed again. The mtime
+- When such a file's size on disk differs from its recorded `disk_size`, `scan`
-  is compared to the nanosecond, as finely as the filesystem records it. A
+  updates `disk_size` alone. It still does not read the file, and counts it as
+  unchanged. ZFS, for example, counts a new file's blocks only seconds after the
+  write. A block is the unit in which a filesystem allocates disk space. A file
+  scanned within those seconds gets its real `disk_size` at the next scan.
+- `scan` hashes a file again when its size differs or its mtime is newer. The
+  mtime is compared to the nanosecond, as finely as the filesystem records it. A
   same-size rewrite counts as a change whenever the filesystem gives it a later
   mtime, even within the same second.
 - The scan deletes each record under a scanned operand whose file it could not
   verify. That covers deleted files, and paths the scan could not stat or read
   this run. To stat a file is to read its details, such as its size, without
-  opening it. So the database holds only what the latest scan over each path
+  opening it. So after a complete scan, the database holds only what that scan
-  could verify. A later successful scan adds such files again.
+  could verify under its operands. A later successful scan adds such files
+  again.
 - Records outside the scanned operands are left alone, so separate trees can be
   scanned on different schedules into one database.
 - The scan keeps every record at or below a path that the walk leaves out for
@@ -397,9 +430,8 @@ operands:
 
     Such records count as outside the scanned operands. So with `-x`, a
     filesystem mounted under an operand can be scanned on its own schedule into
-    the same database. An operand may lie at or inside a default exclude, or,
+    the same database. An operand may itself lie at or inside such a path, as
-    with `-x`, at or inside a directory on another filesystem. Such an operand
+    "Operands" describes. Its walk verifies the records below it, so they are
-    is walked on its own. Its walk verifies the records below it, so they are
     deleted when their files are gone.
 
 #### Phases
@@ -408,14 +440,17 @@ operands:
 phases.
 
 1. `load` reads the existing records. Each record under the operands is kept in
-   memory by path, with its size and mtime, for the walk to compare against.
+   memory by path, with its size, its size on disk and its mtime, for the walk
+   to compare against.
 2. `walk` lists the trees under all operands. Several workers share one queue of
-   directories to read. A worker is one thread of the program, and "Concurrency"
+   directories to read. A worker is one thread of the program. A thread is one
-   says how many there are. Each worker reads one directory at a time. It hands
+   of several parts of a program that run at the same time and share its memory.
-   the subdirectories it finds back to the queue, and it stats each regular file
+   "Concurrency" says how many workers there are. Each worker reads one
-   as it finds it. The walk compares each file with its record. An unchanged
+   directory at a time. It hands the subdirectories it finds back to the queue,
-   file needs nothing more. Every new or changed file is passed to the hash
+   and it stats each regular file as it finds it. The walk compares each file
-   phase.
+   with its record. An unchanged file is not read again. When its size on disk
+   has changed, the update phase writes its new `disk_size`. Every new or
+   changed file is passed to the hash phase.
 3. `hash` reads every new or changed file and computes its three hashes. Files
    are read in inode order. An inode is the filesystem's own record of a file.
    Files with nearby inode numbers usually lie near each other on disk. Reading
@@ -424,8 +459,9 @@ phases.
    tree made of hard links therefore costs one read per file. Completed records
    are written to the database 10,000 at a time while hashing runs. So a scan
    interrupted after hours keeps everything hashed so far.
-4. `update` writes the hashed records not yet written, then deletes the records
+4. `update` writes the hashed records not yet written, and the new `disk_size`
-   the scan did not verify.
+   of each unchanged file whose size on disk changed. Then it deletes the
+   records the scan did not verify.
 
 Reading a tree one directory at a time waits on the disk at every step and takes
 hours at millions of files. Reading many directories at once is what makes the
@@ -434,10 +470,11 @@ walk practical on large or busy storage pools.
 #### Rules for the walk
 
 - Only regular files are recorded. Every other kind of file, such as a symlink
-  or a device node, is passed over. Symlinks are never followed.
+  or a device node, is passed over. The walk follows no symlink it finds.
 - The walk never enters a directory named `.zfs`. Such a directory holds the
   filesystem's ZFS snapshots, which are its saved earlier states. Walking it
-  would list every file once per snapshot.
+  would list every file once per snapshot. `scan` never starts a walk at or
+  inside one either (see "Operands").
 - The walk never records the scan's own database files:
     - the database file;
     - its `.lock` file (see "The scan lock");
@@ -451,14 +488,14 @@ walk practical on large or busy storage pools.
     operand at the path given, even when that path runs through a symlinked
     directory. Such an operand can lead to the database files, and the walk
     passes over them there as well. The walk prints no line for the database
-    files and does not count them as skipped. A record an earlier scan left for
+    files and does not count them as skipped. A record that an earlier scan left
-    one of them is deleted.
+    for one of them is deleted.
 
 - By default the walk enters directories on other filesystems than its
   operand's. With `-x`, it never enters a directory on a different filesystem
-  than its operand. `scan` prints no line for that directory and does not count
+  than its operand. `scan` prints no line for a directory that only `-x` leaves
-  it as skipped. The records below it are kept, as described under "What a scan
+  out, and does not count it as skipped. The records below it are kept, as
-  does to the database".
+  described under "What a scan does to the database".
 - The walk never records anything on a filesystem that is not persistent. A
   filesystem is not persistent when the kernel makes it up or keeps it in
   memory. These are the types, as `sfdupes scan --help` lists them on Linux and
@@ -494,11 +531,10 @@ walk practical on large or busy storage pools.
     The path is not counted as skipped. The records below it are kept, as
     described under "What a scan does to the database".
 
-- Below an operand, the walk does not enter a default exclude. The default
+- Below an operand, the walk does not enter a default exclude. On Linux, the
-  excludes hold the kernel's view of the running system and temporary files, so
+  default excludes hold temporary files and the kernel's view of the running
-  that `sfdupes scan /` walks only the files worth comparing. They depend on the
+  system. They depend on the system, and `sfdupes scan --help` lists those of
-  system, and `sfdupes scan --help` lists those of the system it runs on. On
+  the system it runs on. On Linux they are:
-  Linux they are:
 
     ```
     /proc, /sys, /dev, /run, /tmp, /var/tmp
@@ -527,21 +563,27 @@ walk practical on large or busy storage pools.
     scan: skipped /var/tmp (default exclude)
     ```
 
-    The directory is not counted as skipped. A default exclude on a filesystem
+    The directory is not counted as skipped. `scan` prints the filesystem line
-    that is not persistent, such as `/proc`, gets the line for its filesystem
+    instead for a default exclude that is not on a persistent filesystem, such
-    instead. One that is itself an operand gets no line, because it is walked as
+    as `/proc`. One that is itself an operand gets no line, because it is walked
-    that operand. The scan keeps the records below a default exclude that the
+    as that operand. The scan keeps the records below a default exclude that the
     walk did not enter, as described under "What a scan does to the database".
 
-- A path that cannot be read, for lack of permission or any other error, gets a
+- `scan` skips each of these, with a one-line warning on stderr:
-  one-line warning on stderr and is skipped. The scan continues, and the summary
+    - a directory it cannot read;
-  line says how many paths were skipped. A skipped path that has a record from
+    - a path it cannot stat;
-  an earlier scan loses that record, and so does every path under an unreadable
+    - a new or changed file it cannot read.
-  directory. The database mirrors what the latest scan could verify.
+
+    An unchanged file is not opened, so it keeps its record even when it has
+    become unreadable. The scan continues, and the summary line says how many
+    paths were skipped. A skipped path that has a record from an earlier scan
+    loses that record, and so does every path under an unreadable directory.
+
 - A file or directory that disappears while the scan runs is not an error. The
   scan prints no line for it and does not count it as skipped. A path gone when
   the walk reaches it gets no record, and so does a file gone when the hash
-  phase opens it. The records an earlier scan left at or below it are deleted.
+  phase opens it. The records that an earlier scan left at or below it are
+  deleted.
 
 #### Concurrency
 
@@ -559,7 +601,8 @@ from a thread of its own and never blocks the workers.
 
 - one line naming the database (see "Database");
 - for each dropped operand, in operand order, the warning or the line described
-  under "Operands". A repeated operand, or one under another, gets neither;
+  under "Operands". `scan` prints neither for an operand it drops only because
+  it repeats another or lies under another;
 - the progress display of each phase (see "Progress"). `scan` prints each
   warning for a skipped path as it happens. It prints a line the same way for
   each path left out as a default exclude or as not on a persistent filesystem;
@@ -586,13 +629,14 @@ signal `kill` sends by default:
 - The records hashed but not yet written are written, so the next scan does not
   hash them again.
 - Nothing else is written or deleted. Records are deleted only after a complete
-  walk. So the records under paths an interrupted walk never reached are kept.
+  walk. So the records under paths that an interrupted walk never reached are
-- The database is closed and the lock released, as on any other exit.
+  kept.
+- The database is closed and the lock released.
 - The line `scan: interrupted after N files` goes to stderr. `N` is the number
   of files the walk reached. The exit code is 1.
 
-The next scan reads none of the files already recorded and brings the database
+The next scan reads none of the unchanged files already recorded, and brings the
-up to date as usual.
+database up to date as usual.
 
 After the first signal `scan` stops catching signals. A second one ends it at
 once, as any uncaught signal does. The records not yet written are lost, and the
@@ -602,12 +646,12 @@ and `scan` leaves it ignored.
 
 ### `report` mode
 
-`report` reads every record from the database. It takes no positional arguments,
+`report` reads every record from the database. It takes no positional arguments.
-meaning nothing may follow `report` on the command line but flags. It never
+A positional argument is anything on the command line that is not a flag, such
-touches the scanned filesystem. It neither stats nor opens any path in the
+as the paths `scan` takes. `report` never touches the scanned filesystem. It
-records. It reads the database and writes to stdout and stderr, and nothing
+neither stats nor opens any path in the records. It reads the database and
-else. Its output is the same whether or not the scanned filesystem is still
+writes to stdout and stderr, and nothing else. Its output is the same whether or
-mounted.
+not the scanned filesystem is still mounted.
 
 Processing:
 
@@ -620,14 +664,14 @@ Processing:
   the same for the same database contents.
 
 `report` has two phases, each with its own progress display on stderr (see
-"Progress"). `open` opens the database. `read` counts the records, then reads
+"Progress"). `open` opens the database. `read` gets the number of records from
-them in signature order, largest size first. The database keeps an index named
+the `totals` table (see "Schema"), then reads them in signature order, largest
-`files_signature`, a sorted list of the records by signature (see "Schema").
+size first. The database keeps an index named `files_signature`, a sorted list
-Reading it backwards gives that order, so SQLite sorts nothing and returns the
+of the records by signature (see "Schema"). Reading it backwards gives that
-first record at once. Records with one signature come together, and so do the
+order, so SQLite sorts nothing and returns the first record at once. Records
-records of one size. So `report` holds only the duplicate groups of the size it
+with one signature come together, and so do the records of one size. So `report`
-is reading. Once it reaches a smaller size, it writes them, ordered by `first`
+holds only the duplicate groups of the size it is reading. Once it reaches a
-path.
+smaller size, it writes them, ordered by `first` path.
 
 #### Report output format
 
@@ -651,8 +695,8 @@ Four bytes are written as escapes:
 
 Every other byte is written unchanged, including bytes that are not valid UTF-8,
 the usual text encoding. Undoing the four escapes gives back the stored path.
-Grouping and ordering use the stored path. The warnings `scan` prints are
+Grouping and ordering use the stored path. The warnings about skipped paths that
-escaped the same way, so each warning stays one line.
+`scan` prints are escaped the same way, so each stays one line.
 
 The summary line on stderr gives:
 
@@ -690,11 +734,11 @@ Processing:
 - `trees` builds the directory tree and computes every directory's digest. It
   then groups the directories by digest. Every group with two or more
   directories is a duplicate-tree group.
-- Only the outermost duplicate trees are reported. A group is left out when its
+- `trees` leaves out a group that a group of its members' parents already
-  members' parents are distinct directories that all have the same digest. The
+  implies. A group is left out when its members' parents are distinct
-  parents' group, or an ancestor's, already implies it. A group holding sibling
+  directories that all have the same digest. The parents' group, or an
-  directories, or members whose parents have different digests, is always
+  ancestor's, already implies it. A group is always reported when it holds
-  reported.
+  sibling directories, or when its members' parents have different digests.
 - Within a group, paths are sorted in byte order. The first is `first`, and
   every other path is a `dupe`.
 - Groups are ordered by total tree size, largest first, then by `first` path.
@@ -706,26 +750,39 @@ Two limitations are accepted:
   They share storage, so removing one frees nothing. The scan uses the inode to
   avoid reading a file twice, and does not store it.
 - Only the regular files in the database define a tree. Empty directories are
-  invisible. A file skipped during the scan in one copy but not in the other
+  invisible. When the scan skips a file in one copy but not in the other, the
-  makes two otherwise identical trees differ.
+  two trees differ.
 
 `trees` has four phases, each with its own progress display on stderr (see
-"Progress"). `open` opens the database. `read` counts the records, then reads
+"Progress"). `open` opens the database. `read` gets the number of records from
-them in path order. It builds the directory tree and computes the digests as it
+the `totals` table, then reads them in path order. It builds the directory tree
-goes. `group` groups the directories by digest and picks the groups to report.
+and computes the digests as it goes. `group` groups the directories by digest
-`write` writes them to stdout.
+and picks the groups to report. `write` writes them to stdout.
 
 #### Trees output format
 
 The report is TSV on stdout, with a header line and then one row per duplicate
-tree. A group of N trees gives N-1 rows. `files` is the number of regular files
+tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come
-in one copy of the tree, and `size` their total size in bytes:
+three numbers:
+
+- `files`, the number of regular files in one copy of the tree;
+- `size`, their total size in bytes;
+- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
+  added up.
 
 ```
-first	dupe	files	size
+first	dupe	files	size	disk_size
-/srv/a/project	/srv/backup/project	3417	104857600
+/srv/a/project	/srv/backup/project	3417	104857600	61865984
 ```
 
+A file's `disk_size` is the space its blocks take on disk. It can be more than
+the file's size, because space is allocated in whole blocks. It is less than the
+file's size on a compressed or sparse file. A sparse file has gaps that take no
+space on disk. So two copies of one tree can differ in `disk_size`. When a file
+in one copy is a hard link to one in the other, both copies count its blocks in
+full. Removing it from one copy then frees nothing, as the limitation under
+"`trees` mode" says.
+
 The paths are escaped as described under "Report output format". The root
 directory's path is `/`.
 
@@ -772,10 +829,11 @@ size of recorded files	150000000000000 (136.4 TiB)
   of all recorded files, in bytes, then rounded to a unit such as TiB.
 
 `info` has two phases, each with its own progress display on stderr (see
-"Progress"). `open` opens the database. `count` reads the schema version and
+"Progress"). `open` opens the database. `count` reads the schema version and the
-counts the records in one transaction, a read of the database at one moment. So
+row of `totals` (see "Schema") in one transaction. A transaction is a group of
-the two agree with each other even while a scan is writing. `info` writes its
+reads or writes that SQLite carries out as one step, against the database as it
-lines after the `count` display has ended.
+stands at one moment. So both describe the database at one moment, even while a
+scan is writing. `info` writes its lines after the `count` display has ended.
 
 ### Database
 
@@ -783,7 +841,7 @@ All four subcommands work on one SQLite database file.
 
 #### Location
 
-The database is the file `SFDUPES_DATABASE` names, when that is set and not
+The database is the file named by `SFDUPES_DATABASE`, when that is set and not
 empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
 
 Each subcommand prints one line on stderr naming the database it uses, before
@@ -817,20 +875,21 @@ the error for a missing database comes after it.
 
 #### The scan lock
 
-Only one `scan` runs against a database at a time.
+Only one `scan` at a time can hold the lock.
 
 - For its whole run, `scan` holds a lock on a lock file beside the database. The
-  lock is taken with the `flock(2)` system call, and only one process can hold
+  lock is taken with the `flock(2)` system call, a request a program makes to
-  it. The lock file's name is the database's path, with symlinks resolved, plus
+  the kernel, and only one process can hold it. The lock file's name is the
-  `.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
+  database's path, with symlinks resolved, plus `.lock`. By default that is
-  one database therefore names the same lock file.
+  `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
+  lock file.
 - `scan` takes the lock before it walks the filesystem or opens the database.
   Just before that, it creates the database file, empty, if it is missing. So
   the lock file is named the same way before the first scan as after. A symlink
   at the database path is followed, and its target is created.
-- A second `scan` against the same database does not wait. It fails at once with
+- A second `scan` that finds the lock held does not wait. It fails at once with
-  a one-line error naming the lock file, and exits 1. It has walked nothing and
+  an error naming the lock file, and exits 1. It has walked nothing and has not
-  opened nothing. The running scan carries on.
+  opened the database. The running scan carries on.
 - The lock file is created on first use and never deleted. Only its owner may
   open it. A leftover file blocks nothing. The lock ends with the process that
   holds it, however that process ends. Deleting the file while a scan runs would
@@ -843,8 +902,9 @@ Only one `scan` runs against a database at a time.
   SQLite appends changes to a `-wal` file beside the database, and keeps a
   lookup table for that file in a `-shm` file. Readers do not block the writer,
   so running a report during a scan is safe.
-- Every connection has a busy timeout of 10 seconds. That is how long it waits
+- Every subcommand opens the database with a busy timeout of 10 seconds. That is
-  for a database another connection has locked before it gives up.
+  how long it waits for a database another process has locked before it gives
+  up.
 - The filesystem is authoritative, and the database follows it. Hashed records
   are written 10,000 at a time while the scan runs, so a report may see a scan's
   changes partly applied. A scan that dies partway leaves a valid database
@@ -869,23 +929,27 @@ Only one `scan` runs against a database at a time.
 - Then, before the walk as well, it writes:
     - the `-wal` and `-shm` files, which stay beside the database while the scan
       runs;
-    - in a new database, the table, the index and the schema version described
+    - in a new database, the tables, the index, the triggers and the schema
-      under "Schema".
+      version described under "Schema".
 - During the `load` and `walk` phases, no records are written. A first scan's
   database holds no records until the hash phase, however long its walk runs.
 - During the `hash` phase, records are written 10,000 at a time.
-- During the `update` phase, the rest of the hashed records are written, and the
+- During the `update` phase, the rest of the hashed records are written, and so
-  deletions are made.
+  is the new `disk_size` of each unchanged file whose size on disk changed. Then
+  the deletions are made.
 - As `scan` closes the database, the `-wal` and `-shm` files go away, unless a
   report still has the database open.
 
 #### Schema
 
-The schema is the database's layout, its table and its index.
+The schema is the database's layout: its tables, its index and its triggers. A
-`PRAGMA user_version` holds the schema version, currently 1. A database with any
+trigger is a statement SQLite runs by itself when a table changes.
-other version is a fatal error. `scan` creates the table and the index and sets
+`PRAGMA user_version` holds the schema version, currently 1. An empty database
-the version in one transaction, a group of writes that lands whole or not at
+is at version 0, and `scan` sets it up. Any other version is a fatal error.
-all. A first scan stopped while doing so leaves an empty database, which the
+`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
+error too. `scan` creates the tables, the index and the triggers and sets the
+version in one transaction (see "`info` mode"). So they all land together or not
+at all. A first scan stopped while doing so leaves an empty database, which the
 next scan sets up. A database at version 0 that already has a `files` table was
 not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
 remove the file and rescan.
@@ -894,6 +958,7 @@ remove the file and rescan.
 CREATE TABLE files (
     path       BLOB PRIMARY KEY,  -- absolute path, raw bytes
     size       INTEGER NOT NULL,  -- bytes
+    disk_size  INTEGER NOT NULL,  -- bytes allocated on disk (st_blocks × 512)
     mtime      INTEGER NOT NULL,  -- whole Unix seconds of the mtime
     mtime_nsec INTEGER NOT NULL,  -- nanoseconds within that second
     head       BLOB NOT NULL,     -- raw 32-byte SHA-256 of the first 64 KiB
@@ -901,10 +966,28 @@ CREATE TABLE files (
     content    BLOB NOT NULL      -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
 ) WITHOUT ROWID;
 CREATE INDEX files_signature ON files (size, head, tail, content);
+CREATE TABLE totals (
+    records INTEGER NOT NULL,  -- number of records in files
+    size    INTEGER NOT NULL   -- sizes of all records in files added up
+);
+INSERT INTO totals (records, size) VALUES (0, 0);
+CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
+    UPDATE totals SET records = records + 1, size = size + NEW.size;
+END;
+CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
+    UPDATE totals SET records = records - 1, size = size - OLD.size;
+END;
+CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
+    UPDATE totals SET size = size + NEW.size - OLD.size;
+END;
 ```
 
 - `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
   necessarily UTF-8 text.
+- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
+  call that stats a file without following a symlink. Its `st_blocks` field
+  counts that space in units of 512 bytes. `disk_size` is less than `size` on a
+  compressed or sparse file (see "Trees output format").
 - `mtime` holds the whole seconds of the file's mtime, rounded down, and
   `mtime_nsec` the nanoseconds past that second. Split this way, they hold any
   mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -915,18 +998,29 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
   not 32 bytes.
 - `files_signature` lets `report` read the records in signature order without
   sorting the table.
+- `totals` holds one row: the number of records in `files` and the sum of their
+  sizes. Three triggers on `files` keep that row current. Each runs in the
+  transaction of the change to `files` that fires it, so the row and the records
+  always agree. `info` reads both numbers from this row. The `load` phase of
+  `scan` and the `read` phase of `report` and `trees` read the number of records
+  from it as their total. Counting the records in `files` would read every
+  record. That takes minutes on a database of tens of millions. The triggers
+  make each insert, each delete and each change of a record's `size` slower, so
+  a first scan of many small files takes about two fifths longer. On large
+  files, reading them takes most of the time, so the difference is small.
 
 ### Progress
 
-Every subcommand always shows that it is working. Each phase gets a display of
+Each phase of every subcommand gets a display of its own on stderr from the
-its own on stderr from the moment it starts. The display names the phase and
+moment it starts. The display names the phase and shows its progress. The parts
-shows its progress. The parts of its line are listed below.
+of its line are listed below.
 
 The phases, and what each display counts:
 
 - `scan` has four phases:
-    - `[1/4] load` counts the records first, then reads them against that count.
+    - `[1/4] load` gets the number of records from the `totals` table first,
-      While it counts, the display says `counting the records`.
+      then reads them against that number. Until it has the number, the display
+      says `counting the records`.
     - `[2/4] walk` has no exact total. Its estimate is the number of records
       under the operands, shown with a `~`. A scan with no records under its
       operands, such as a first scan, has no estimate. The time remaining is
@@ -938,18 +1032,18 @@ The phases, and what each display counts:
 - `report` has two phases:
     - `[1/2] open` has no total. The display says `opening the database`, also
       while it waits on a busy one.
-    - `[2/2] read` counts the records first, then reads them against that count.
+    - `[2/2] read` gets the number of records from the `totals` table first,
-      It writes rows to stdout as it goes.
+      then reads them against that number. Until it has the number, the display
+      says `counting the records`. It writes rows to stdout as it goes.
 - `trees` has four phases:
     - `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
       writes nothing.
-    - `[3/4] group` has the directories `read` found as its total.
+    - `[3/4] group` has the directories that `read` found as its total.
     - `[4/4] write` has the duplicate-tree groups as its total.
 - `info` has two phases:
     - `[1/2] open` is as for `report`.
-    - `[2/2] count` has no total either, because one request to the database
+    - `[2/2] count` has no total either, because it reads one row. The display
-      counts everything. The display says `counting the records` until that
+      says `counting the records` until it has the row.
-      request returns.
 
 Each display is one line holding:
 
@@ -981,9 +1075,9 @@ phase waits, in place of the rate. That happens while one large file is read, or
 while a disk or the database is slow. Each phase names the places it may wait
 in, and a phase that may wait in two places names both, as `disk/db`:
 
-- `walk` says `disk`, for files.
+- `walk` says `disk`, because it reads directories and stats files.
-- `hash` says `disk/db`: it reads files and writes their records to the database
+- `hash` says `disk/db`, because it reads files and writes their records to the
-  between them.
+  database between them.
 - `load`, `update` and the `read` phase of `trees` say `db`.
 - The `read` phase of `report` says `db/out`, because it writes the report to
   stdout while it reads.
@@ -1009,19 +1103,18 @@ On a terminal:
     ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s            12% |█         |
     ```
 
-- No line reaches the terminal's last column, however narrow the terminal is. A
+- No line reaches the terminal's last column. A line too wide for the terminal
-  line too wide for the terminal first leaves out its rate, the words of a
+  first leaves out its rate, the words of a stalled phase, or what the phase is
-  stalled phase, or what the phase is waiting for. It leaves out its bar next.
+  waiting for. It leaves out its bar next. It is cut short at the edge only when
-  It is cut short at the edge only when the rest does not fit on its own. The
+  the rest does not fit on its own. The rest is:
-  rest is:
     - the phase and its place;
     - the progress, when the phase shows one;
     - the elapsed time;
     - the time remaining.
 
-    That happens hours into a scan of millions of files, on a terminal narrower
+    On a terminal 80 columns wide, no line is cut short, even hours into a scan
-    than about 75 columns. A terminal made narrower while a phase runs is fitted
+    of millions of files. A terminal made narrower during a phase is fitted at
-    at the next draw.
+    the next draw.
 
 - A warning printed during a phase lands on a line of its own, never inside the
   display. So does every line written to stdout when stdout and stderr are the
@@ -1037,8 +1130,8 @@ Each phase prints its line:
 - every 5 seconds while it runs, whether or not an item has completed;
 - once more as it ends.
 
-A phase that stops short of its total, as an interrupted one does, keeps showing
+When a phase stops short of its total, as an interrupted one does, it keeps
-the count it stopped at.
+showing the count it stopped at.
 
 ### Error handling and exit codes
 
@@ -1047,12 +1140,14 @@ the count it stopped at.
   error is one of these:
     - an operand that does not exist, or whose details `scan` cannot read;
     - a list of mounted filesystems that `scan` cannot read;
-    - another `scan` already running against the same database;
+    - another `scan` holding the lock (see "The scan lock");
-    - a database that cannot be created or opened, or whose reads or writes
+    - a database, its directory or its lock file that cannot be created or
-      fail, its directory and its lock file included;
+      opened;
+    - a read or write of the database that fails;
     - a relative database path when the working directory cannot be read;
-    - a database with another schema version, or one at version 0 that already
+    - a database at a schema version other than 0 or 1;
-      has a `files` table;
+    - a database at version 0 read by `report`, `trees` or `info`;
+    - a database at version 0 that already has a `files` table (see "Schema");
     - a record with a hash that is not 32 bytes, read by `report` or `trees`;
     - a missing database for `report`, `trees` or `info`;
     - a failed write to stdout.
@@ -1065,9 +1160,9 @@ the count it stopped at.
     - `--workers` below 1;
     - `report`, `trees` or `info` with a positional argument.
 
-A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
+A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
-failed write to stdout, such as a full disk, is reported the same way. Two cases
+stdout, such as a full disk, is reported the same way. Two cases never reach
-never reach `sfdupes` as a failed write:
+`sfdupes` as a failed write:
 
 - The reader of a stdout pipe may exit early, as `head` does in
   `sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
@@ -1084,9 +1179,8 @@ This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
 workflow, and the `Makefile` targets call them, except `build` and `clean` (see
-"Build"). Every script runs under plain `sh`, the standard Unix shell, and uses
+"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds
-no `bash` feature. It finds the repository root itself, so it can be run from
+the repository root itself, so it can be run from any directory.
-any directory.
 
 - `script/bootstrap` installs what building and developing need. It can be run
   again at any time, and installed tools are kept. `git`, `make` and `go` are
@@ -1102,8 +1196,8 @@ any directory.
     - An installed `node` is used when its major version is 22. A `yarn` already
       on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
       activated through `corepack`, the `yarn` installer that ships with `node`.
-      When there is no `corepack` either, `npm`, the package manager that ships
+      When there is no `corepack` either, the script installs `yarn` with `npm`,
-      with `node`, installs it.
+      the package manager that ships with `node`.
     - Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
       version manager, and activates `yarn` 1.22.22 under that `node` through
       `corepack`. When `nvm` is missing, the script first downloads it and
@@ -1133,9 +1227,9 @@ any directory.
 - `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
   for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
   writes no image. The stage starts from the `golangci/golangci-lint` image,
-  pinned by its digest, and runs three build steps:
+  pinned by its digest, and runs these checks as build steps:
-    - the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
+    - the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
-      formatter, would change;
+      change a Go source;
     - `golangci-lint config verify`, which checks the linter's settings file
       `.golangci.yml`;
     - `golangci-lint run`, which lints the Go sources.
@@ -1147,8 +1241,8 @@ any directory.
   with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as
   `.prettierrc` sets and as the script also passes on the command line. It wraps
   at 80 columns, its default width. Both run on the host. `prettier` runs
-  through `yarn` at the version `yarn.lock` pins. When the `node` on `PATH` is
+  through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH`
-  not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
+  is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
   installed.
 - `script/fmt-check` is the read-only form of `script/fmt`. It names any
   unformatted file and exits non-zero instead of writing. `gofmt` and `prettier`
@@ -1169,8 +1263,8 @@ any directory.
   every check passed. The tests and the linter run twice, once in `script/check`
   and again as stages of the image build.
 - `script/precommit` runs from the `git` pre-commit hook, a script that `git`
-  runs before each commit. It fails the commit when `go mod tidy` would change
+  runs before each commit. It runs `go mod tidy`, and fails the commit when that
-  anything. Then it runs `script/check`.
+  changes `go.mod` or `go.sum`. Then it runs `script/check`.
 - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
   `script/precommit`.
 
@@ -1180,9 +1274,9 @@ install system packages and download the Go modules. Every step that checks or
 compiles the code comes after it. Every `RUN` step after it runs again on each
 build. Without that, Docker would serve the test and lint steps from its cache
 on an unchanged tree, and the build would pass without running them. The steps
-above `ARG CACHEBUST` stay cached until something they depend on changes, such
+above `ARG CACHEBUST` stay cached until a change to something they depend on,
-as `go.mod`. Go's build cache is kept between builds, so only what changed is
+such as `go.mod`. Go's build cache is kept between builds, so only what changed
-compiled again. The tests run with `-count=1`, so no test reports a stored
+is compiled again. The tests run with `-count=1`, so no test reports a stored
 result. `golangci-lint run` keeps the linter's own cache as well, so an
 unchanged package is not linted again.
 
@@ -1223,10 +1317,10 @@ commands of their own:
 
     No other dependency outside the standard library is added.
 
-- The code stays pure Go, with no C code. Cross-compilation is not a concern.
+- The code stays pure Go, with no C code. Building on one system for another is
-  `modernc.org/sqlite` is pure Go too, so the binary builds without cgo. The
+  not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
-  `Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
+  cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
-  because the race detector needs cgo (see `script/test`).
+  exception, because the race detector needs cgo (see `script/test`).
 - `report` and `trees` give the same output for the same database contents,
   whatever order the records were inserted in.
 
@@ -1242,6 +1336,7 @@ All of the following, run in this directory, must pass:
    data.
 
     ```sh
+    make build
     d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
     export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
     mkdir -p "$d/a" "$d/b"
@@ -1281,9 +1376,10 @@ All of the following, run in this directory, must pass:
     - The groups come in order of size, largest first.
 
     `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
-    The row shows 2 files and 3100 bytes. The two `sub` directories are left
+    The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the
-    out, because the `t1` and `t2` group implies them. `t3` appears nowhere,
+    filesystem. The two `sub` directories are left out, because the `t1` and
-    since one of its file names differs.
+    `t2` group implies them. `t3` appears nowhere, since one of its file names
+    differs.
 
     After the rescan, the second `report` shows `one.bin` out of its group,
     because its content changed. `copy.bin` and `copy2.bin` remain as one pair.
@@ -1294,11 +1390,11 @@ All of the following, run in this directory, must pass:
 
 ## TODO
 
-Tracked in [TODO.md](TODO.md).
+Tracked in [`TODO.md`](TODO.md).
 
 ## License
 
-MIT. See [LICENSE](LICENSE).
+MIT. See [`LICENSE`](LICENSE).
 
 ## Author
 

db.go

@@ -47,14 +47,15 @@ const dbFilePerm = 0o644
 const lockFilePerm = 0o600
 
 // createTableSQL is the schema applied to a fresh database. Paths are
-// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
+// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
-// holds whole Unix seconds and mtime_nsec the nanoseconds within that
+// disk_size holds the bytes the file takes on disk. mtime holds whole
-// second. head, tail and content each hold the raw 32 bytes of a
+// Unix seconds and mtime_nsec the nanoseconds within that second. head,
-// SHA-256.
+// tail and content each hold the raw 32 bytes of a SHA-256.
 const createTableSQL = `
 CREATE TABLE files (
     path       BLOB PRIMARY KEY,
     size       INTEGER NOT NULL,
+    disk_size  INTEGER NOT NULL,
     mtime      INTEGER NOT NULL,
     mtime_nsec INTEGER NOT NULL,
     head       BLOB NOT NULL,
@@ -69,14 +70,59 @@ const createIndexSQL = `
 CREATE INDEX files_signature ON files (size, head, tail, content)
 `
 
+// createTotalsSQL creates the totals table, which holds one row: the
+// number of records and the sizes of all records added up. Counting
+// them over files would read every record. The three triggers below
+// keep the row current, in the transaction of each change to files.
+const createTotalsSQL = `
+CREATE TABLE totals (
+    records INTEGER NOT NULL,
+    size    INTEGER NOT NULL
+)
+`
+
+// insertTotalsSQL adds the one row of totals, for an empty files table.
+const insertTotalsSQL = `
+INSERT INTO totals (records, size) VALUES (0, 0)
+`
+
+// totalsAfterInsertSQL counts each inserted record into totals.
+const totalsAfterInsertSQL = `
+CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
+    UPDATE totals SET records = records + 1, size = size + NEW.size;
+END
+`
+
+// totalsAfterDeleteSQL takes each deleted record out of totals.
+const totalsAfterDeleteSQL = `
+CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
+    UPDATE totals SET records = records - 1, size = size - OLD.size;
+END
+`
+
+// totalsAfterUpdateSQL replaces a record's old size in totals with its
+// new one. scan's upsert of an existing record is such an update.
+const totalsAfterUpdateSQL = `
+CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
+    UPDATE totals SET size = size + NEW.size - OLD.size;
+END
+`
+
+// recordCountSQL reads the number of records from totals.
+const recordCountSQL = `
+SELECT records FROM totals
+`
+
 // upsertSQL inserts one file record, replacing any existing record for
 // the same path.
 const upsertSQL = `
-INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
+INSERT INTO files (
-VALUES (?, ?, ?, ?, ?, ?, ?)
+    path, size, disk_size, mtime, mtime_nsec, head, tail, content
+)
+VALUES (?, ?, ?, ?, ?, ?, ?, ?)
 ON CONFLICT (path) DO UPDATE SET
-    size = excluded.size, mtime = excluded.mtime,
+    size = excluded.size, disk_size = excluded.disk_size,
-    mtime_nsec = excluded.mtime_nsec,
+    mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
     head = excluded.head, tail = excluded.tail,
     content = excluded.content
 `
@@ -150,9 +196,14 @@ func createDatabaseDirectory(path string) error {
 // scanParams are the connection parameters for scan: read-write, with
 // WAL journaling and a busy timeout, so a report can run while a cron
 // scan is in progress. closeScanDatabase leaves WAL mode again.
+//
+// Each write to files fires a trigger, so SQLite keeps a statement
+// journal for it. temp_store(MEMORY) keeps that journal in memory,
+// which costs far less than a temporary file.
 const scanParams = "_pragma=busy_timeout(10000)" +
 	"&_pragma=journal_mode(WAL)" +
-	"&_pragma=synchronous(NORMAL)"
+	"&_pragma=synchronous(NORMAL)" +
+	"&_pragma=temp_store(MEMORY)"
 
 // reportParams are the connection parameters for report, trees and
 // info: read-only, with the same busy timeout. They set no journal
@@ -426,14 +477,14 @@ func createSchema(ctx context.Context, db *sql.DB) error {
 
 	defer func() { _ = tx.Rollback() }()
 
-	_, err = tx.ExecContext(ctx, createTableSQL)
+	for _, stmt := range []string{
+		createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL,
+		totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL,
+	} {
+		_, err = tx.ExecContext(ctx, stmt)
 		if err != nil {
 			return fmt.Errorf("create schema: %w", err)
 		}
-
-	_, err = tx.ExecContext(ctx, createIndexSQL)
-	if err != nil {
-		return fmt.Errorf("create schema: %w", err)
 	}
 
 	_, err = tx.ExecContext(ctx,
@@ -480,7 +531,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 
 	var n int64
 
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
+	err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
 	if err != nil {
 		return fmt.Errorf("count records: %w", err)
 	}
@@ -488,8 +539,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 	total(n)
 
 	rows, err := tx.QueryContext(ctx,
-		"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
+		"SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
-			"FROM files ORDER BY path")
+			"content FROM files ORDER BY path")
 	if err != nil {
 		return fmt.Errorf("read records: %w", err)
 	}
@@ -503,7 +554,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 			r                         scanRec
 		)
 
-		err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content)
+		err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
+			&head, &tail, &content)
 		if err != nil {
 			return fmt.Errorf("read record: %w", err)
 		}
@@ -538,10 +590,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
 `
 
 // loadSignatureRows gives total the number of records, then streams
-// every record's signature and path to fn, without its mtime, in the
+// every record's signature and path to fn, without its mtime or its
-// order of signatureRowsSQL. The count and the records are read in one
+// disk_size, in the order of signatureRowsSQL. The count and the
-// transaction, so they agree while a scan is committing. An error from
+// records are read in one transaction, so they agree while a scan is
-// fn stops the reading and is returned as it is.
+// committing. An error from fn stops the reading and is returned as it
+// is.
 func loadSignatureRows(ctx context.Context, db *sql.DB,
 	total func(n int64), fn func(r scanRec) error,
 ) error {
@@ -556,7 +609,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
 
 	var n int64
 
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
+	err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
 	if err != nil {
 		return fmt.Errorf("count records: %w", err)
 	}
@@ -622,7 +675,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error {
 func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
 	var n int64
 
-	err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
+	err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n)
 	if err != nil {
 		return 0, fmt.Errorf("count records: %w", err)
 	}
@@ -630,12 +683,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
 	return n, nil
 }
 
-// databaseCountsSQL counts the records and adds up their sizes, in one
+// databaseCountsSQL reads the number of records and the sizes of all
-// pass over the table. A SUM over no rows is NULL, so on an empty table
+// records added up from totals.
-// the SUM falls back to 0.
 const databaseCountsSQL = `
-SELECT COUNT(*), COALESCE(SUM(size), 0)
+SELECT records, size FROM totals
-FROM files
 `
 
 // databaseCounts is what info reads from the database: its schema
@@ -678,15 +729,15 @@ func readDatabaseCounts(ctx context.Context,
 	return c, nil
 }
 
-// loadFileMeta streams every record's path, size and mtime to fn. Scan
+// loadFileMeta streams every record's path, size, disk_size and mtime
-// change detection needs no hash values, and skipping the hash columns
+// to fn. Scan change detection needs no hash values, and skipping the
-// keeps the scan's in-memory index small on multi-million-file
+// hash columns keeps the scan's in-memory index small on
-// databases.
+// multi-million-file databases.
 func loadFileMeta(ctx context.Context, db *sql.DB,
-	fn func(path string, size int64, mtime time.Time),
+	fn func(path string, m fileMeta),
 ) error {
 	rows, err := db.QueryContext(ctx,
-		"SELECT path, size, mtime, mtime_nsec FROM files")
+		"SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
 	if err != nil {
 		return fmt.Errorf("read records: %w", err)
 	}
@@ -696,15 +747,18 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
 	for rows.Next() {
 		var (
 			path      []byte
-			size, sec, nsec int64
+			m         fileMeta
+			sec, nsec int64
 		)
 
-		err = rows.Scan(&path, &size, &sec, &nsec)
+		err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
 		if err != nil {
 			return fmt.Errorf("read record: %w", err)
 		}
 
-		fn(string(path), size, time.Unix(sec, nsec))
+		m.mtime = time.Unix(sec, nsec)
+
+		fn(string(path), m)
 	}
 
 	err = rows.Err()
@@ -722,21 +776,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
 // WAL small and let concurrent reports observe progress.
 const updateBatchSize = 10000
 
-// applyChanges writes one scan's database changes — upserts for new and
+// applyChanges writes upserts for new and changed files and deletes for
-// changed files, deletes for vanished ones — in batched transactions.
+// vanished ones, in batched transactions. Progress is rendered on prog
-// Progress is rendered on prog (one increment per change).
+// (one increment per change).
 func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
 	deletes []string, prog *progress,
 ) error {
 	for batch := range slices.Chunk(upserts, updateBatchSize) {
-		err := applyBatch(ctx, db, batch, nil, prog)
+		err := applyBatch(ctx, db, batch, nil, nil, prog)
 		if err != nil {
 			return err
 		}
 	}
 
 	for batch := range slices.Chunk(deletes, updateBatchSize) {
-		err := applyBatch(ctx, db, nil, batch, prog)
+		err := applyBatch(ctx, db, nil, nil, batch, prog)
 		if err != nil {
 			return err
 		}
@@ -745,10 +799,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
 	return nil
 }
 
-// applyBatch commits one batch of upserts and deletes in a single
+// applyDiskSizes writes the disk_size of each of recs into its record,
-// transaction.
+// in batched transactions, and leaves the rest of the record alone.
+// Progress is rendered on prog (one increment per record).
+func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
+	prog *progress,
+) error {
+	for batch := range slices.Chunk(recs, updateBatchSize) {
+		err := applyBatch(ctx, db, nil, batch, nil, prog)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// applyBatch commits one batch of upserts, disk_size updates and
+// deletes in a single transaction.
 func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
-	deletes []string, prog *progress,
+	diskSizes []fileRec, deletes []string, prog *progress,
 ) error {
 	tx, err := db.BeginTx(ctx, nil)
 	if err != nil {
@@ -762,6 +832,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
 		return err
 	}
 
+	err = execDiskSizes(ctx, tx, diskSizes, prog)
+	if err != nil {
+		return err
+	}
+
 	err = execDeletes(ctx, tx, deletes, prog)
 	if err != nil {
 		return err
@@ -787,7 +862,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
 	defer func() { _ = st.Close() }()
 
 	for _, r := range upserts {
-		_, err = st.ExecContext(ctx, []byte(r.path), r.size,
+		_, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
 			r.mtime.Unix(), r.mtime.Nanosecond(),
 			r.head[:], r.tail[:], r.content[:])
 		if err != nil {
@@ -800,6 +875,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
 	return nil
 }
 
+// execDiskSizes sets the disk_size of the record of each of recs. It
+// does not touch size, so the totals triggers do not fire.
+func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
+	prog *progress,
+) error {
+	st, err := tx.PrepareContext(ctx,
+		"UPDATE files SET disk_size = ? WHERE path = ?")
+	if err != nil {
+		return fmt.Errorf("prepare disk_size update: %w", err)
+	}
+
+	defer func() { _ = st.Close() }()
+
+	for _, r := range recs {
+		_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
+		if err != nil {
+			return fmt.Errorf("update disk_size of %s: %w", r.path, err)
+		}
+
+		prog.increment()
+	}
+
+	return nil
+}
+
 // execDeletes removes the records for paths no longer present.
 func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
 	prog *progress,

db_test.go

@@ -535,17 +535,12 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) {
 	}
 }
 
-// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
+// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY
-// from the files_signature index alone, with nothing to sort: the
+// PLAN for query.
-// first record comes at once, so report's read phase counts each
+func queryPlan(t *testing.T, db *sql.DB, query string) []string {
-// record as it comes rather than waiting on SQLite for all of them.
+	t.Helper()
-func TestSignatureRowsNeedNoSort(t *testing.T) {
-	t.Parallel()
 
-	db := openTestDB(t)
+	rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query)
-
-	rows, err := db.QueryContext(t.Context(),
-		"EXPLAIN QUERY PLAN "+signatureRowsSQL)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -573,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
 		t.Fatal(err)
 	}
 
+	return plan
+}
+
+// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
+// from the files_signature index alone, with nothing to sort: the
+// first record comes at once, so report's read phase counts each
+// record as it comes rather than waiting on SQLite for all of them.
+func TestSignatureRowsNeedNoSort(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	plan := queryPlan(t, db, signatureRowsSQL)
+
 	want := []string{"SCAN files USING COVERING INDEX files_signature"}
 	if !slices.Equal(plan, want) {
 		t.Errorf("query plan = %q, want %q", plan, want)
 	}
 }
 
+// TestCountsReadOnlyTotals checks that SQLite answers the queries for
+// the number of records and their total size from the one row of
+// totals. A plan that reads files or its index reads every record.
+func TestCountsReadOnlyTotals(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	for _, query := range []string{recordCountSQL, databaseCountsSQL} {
+		plan := queryPlan(t, db, query)
+
+		want := []string{"SCAN totals"}
+		if !slices.Equal(plan, want) {
+			t.Errorf("query plan of %q = %q, want %q", query, plan, want)
+		}
+	}
+}
+
+// TestCountsComeFromTotals sets the row of totals to numbers that files
+// does not hold, then checks that scan's load phase, the read phases of
+// trees and report, and info each report the numbers from totals.
+func TestCountsComeFromTotals(t *testing.T) {
+	t.Parallel()
+
+	// files is empty, so a count over files gives 0 records of 0 bytes.
+	const records, size = 7, 9000
+
+	db := openTestDB(t)
+
+	_, err := db.ExecContext(t.Context(),
+		"UPDATE totals SET records = ?, size = ?", records, size)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	scanTotal, err := countRecords(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	var treesTotal, reportTotal int64
+
+	err = loadFileRows(t.Context(), db,
+		func(n int64) { treesTotal = n }, func(scanRec) {})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = loadSignatureRows(t.Context(), db,
+		func(n int64) { reportTotal = n },
+		func(scanRec) error { return nil })
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	info, err := readDatabaseCounts(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for name, got := range map[string]int64{
+		"scan's load":   scanTotal,
+		"trees' read":   treesTotal,
+		"report's read": reportTotal,
+		"info":          info.records,
+	} {
+		if got != records {
+			t.Errorf("%s: %d records, want %d from totals",
+				name, got, records)
+		}
+	}
+
+	if info.size != size {
+		t.Errorf("info: size %d, want %d from totals", info.size, size)
+	}
+}
+
+// assertTotalsMatchFiles fails unless the row of totals holds the
+// number of records and the sizes of all records added up, as counted
+// over files.
+func assertTotalsMatchFiles(t *testing.T, db *sql.DB) {
+	t.Helper()
+
+	got, err := readDatabaseCounts(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	var records, size int64
+
+	err = db.QueryRowContext(t.Context(),
+		"SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files").
+		Scan(&records, &size)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if got.records != records || got.size != size {
+		t.Errorf("totals = %d records of %d bytes, files holds %d of %d",
+			got.records, got.size, records, size)
+	}
+}
+
+// TestTotalsMatchFiles scans the smoke tree, rescans it after two files
+// change size, and rescans it after two files are removed. After each
+// scan the row of totals must agree with files.
+func TestTotalsMatchFiles(t *testing.T) {
+	t.Parallel()
+
+	dir := buildSmokeTree(t)
+	db := openTestDB(t)
+
+	syncTree(t, db, dir)
+	assertTotalsMatchFiles(t, db)
+
+	writeFile(t, dir, "a/one.bin", pattern(10, 2500))
+	writeFile(t, dir, "t1/f1", pattern(30, 1000))
+
+	if st := syncTree(t, db, dir); st.updated != 2 {
+		t.Fatalf("rescan stats = %+v, want 2 updated", st)
+	}
+
+	assertTotalsMatchFiles(t, db)
+
+	for _, name := range []string{"a/unique.bin", "t2/sub/f2"} {
+		err := os.Remove(filepath.Join(dir, name))
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	if st := syncTree(t, db, dir); st.removed != 2 {
+		t.Fatalf("rescan stats = %+v, want 2 removed", st)
+	}
+
+	assertTotalsMatchFiles(t, db)
+}
+
 func TestApplyChangesRoundTrip(t *testing.T) {
 	t.Parallel()
 

exclude_test.go

@@ -17,7 +17,9 @@ import (
 func TestDefaultExcludes(t *testing.T) {
 	t.Parallel()
 
-	// README "Rules for the walk": each system's list.
+	// README "Rules for the walk": the Linux and macOS lists, and none
+	// on any other system. The Windows list exists in the code only: the
+	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
 		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
 		goosDarwin: {

info.go

@@ -26,11 +26,11 @@ func infoPhase(name string) string {
 // it. Any database problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and count (count the records,
+// own on stderr: open (open the database) and count (read the number of
-// with one query). Each display ends before the next phase starts, and
+// records and their total size from the totals table). Each display
-// before an error is returned to be printed. The details are written
+// ends before the next phase starts, and before an error is returned
-// once the last display has ended, so no display is drawn while they
+// to be printed. The details are written once the last display has
-// are written.
+// ended, so no display is drawn while they are written.
 func runInfo(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error {
 
 	defer func() { _ = db.Close() }()
 
-	// One query counts everything, so the phase never has a total.
+	// One row holds both counts, so the phase never has a total.
 	prog = newProgress(progressParams{
 		label: infoPhase("count"), total: -1,
 		waitingFor: countingRecords,

main_test.go

@@ -180,9 +180,9 @@ func capture(t *testing.T, std **os.File) func() string {
 }
 
 // brokenDatabase writes a database that opens cleanly and passes the
-// schema-version check but has no files table, so the first query
+// schema-version check but has no tables, so the first query fails
-// fails with the database already open: a fatal error on a path that
+// with the database already open: a fatal error on a path that owns an
-// owns an open database. It closes the database the way scan does.
+// open database. It closes the database the way scan does.
 func brokenDatabase(t *testing.T) string {
 	t.Helper()
 
@@ -265,8 +265,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
 			assertFatalOutput(t, stderr.String(), stdout())
 
 			// Proof that the failure happened after the open: only a
-			// query against the opened database can report this.
+			// query against the opened database can report this. Each
-			if !strings.Contains(stderr.String(), "no such table: files") {
+			// subcommand's first query reads totals.
+			if !strings.Contains(stderr.String(), "no such table: totals") {
 				t.Errorf("stderr = %q, want the failure to come from a "+
 					"query on the open database", stderr.String())
 			}
@@ -773,8 +774,8 @@ func TestBuildVersion(t *testing.T) {
 func TestRunVersionWriteFailureIsFatal(t *testing.T) {
 	t.Parallel()
 
-	// README §Error handling: a stdout write failure exits 1, reported
+	// README §Error handling: a failed write to stdout is a fatal error,
-	// in one line on stderr.
+	// exit 1, reported on stderr starting with "sfdupes:".
 	var stderr bytes.Buffer
 
 	code := run([]string{"--version"}, failingWriter{}, &stderr)
@@ -817,6 +818,23 @@ func scanFixture(t *testing.T) []string {
 	return dupes
 }
 
+// storedDiskSize returns the disk_size of the record for path in the
+// database at dbPath. A test of a trees run compares with this rather
+// than with a new lstat, which on ZFS can differ from the scan's (see
+// diskSizeOf).
+func storedDiskSize(t *testing.T, dbPath, path string) int64 {
+	t.Helper()
+
+	db, err := openReportDatabase(t.Context(), dbPath)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	defer func() { _ = db.Close() }()
+
+	return recordByPath(t, dbRecords(t, db), path).diskSize
+}
+
 // scanOK runs scan over operands, fails the test unless it exits 0 with
 // nothing on stdout, and returns everything it printed to stderr.
 func scanOK(t *testing.T, operands ...string) string {
@@ -928,24 +946,21 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
 }
 
 func TestRunScanSkipsZFSOperand(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
 	zfs := filepath.Join(t.TempDir(), ".zfs")
-	snapshot := filepath.Join(zfs, "snapshot", "hourly")
+	f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
-	f := writeFile(t, snapshot, "f", pattern(1, 10))
 
-	// An operand beneath a .zfs directory is walked, because it is not
+	// No scan records a file inside a .zfs directory, so the record
-	// itself named .zfs.
+	// beneath the operand is written straight into the database.
-	scanOK(t, snapshot)
+	path := seedDatabase(t, []scanRec{{size: 10, path: f}})
+	t.Setenv(databaseEnv, path)
 
 	assertOperandSkipped(t, path, zfs, ".zfs directory", f)
 }
 
 // assertOperandSkipped scans operand alone and checks that it is skipped
 // as kind: a warning naming it, one skip in the summary, exit 0, and the
-// record for kept, which an earlier scan stored beneath operand, still
+// record for kept, stored beneath operand before this scan, still in the
-// in the database at dbPath.
+// database at dbPath.
 func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
 	kept string,
 ) {
@@ -974,6 +989,121 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
 	recordByPath(t, dbRecords(t, db), kept)
 }
 
+// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
+// directory, first alone and then under another operand. Each time it
+// is warned about and counted as skipped, and nothing in it is walked.
+// The other operand is scanned as usual.
+func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := t.TempDir()
+	hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
+	writeFile(t, hourly, "f", pattern(1, 10))
+	f := writeFile(t, dir, "f", pattern(2, 10))
+
+	warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
+
+	for _, tc := range []struct {
+		operands []string
+		summary  string
+	}{
+		{
+			[]string{hourly},
+			"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
+				"0 unchanged), 1 skipped\n",
+		},
+		{
+			[]string{dir, hourly},
+			"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
+				"0 unchanged), 1 skipped\n",
+		},
+	} {
+		stderr := scanOK(t, tc.operands...)
+		if !strings.Contains(stderr, warning) ||
+			!strings.Contains(stderr, tc.summary) {
+			t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
+				stderr, warning, tc.summary)
+		}
+	}
+
+	db, err := openDB(path, reportParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
+	}
+}
+
+// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
+// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
+// by the path given, counted as skipped, and not walked.
+func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
+	t.Setenv(databaseEnv, testDBPath(t))
+
+	dir := t.TempDir()
+	writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
+
+	snaps := filepath.Join(dir, "snaps")
+
+	err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	daily := filepath.Join(snaps, "daily")
+	stderr := scanOK(t, daily)
+
+	warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
+	summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
+		"0 unchanged), 1 skipped\n"
+
+	if !strings.Contains(stderr, warning) ||
+		!strings.Contains(stderr, summary) {
+		t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
+	}
+}
+
+// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
+// a symlink to a directory outside any .zfs directory. The operand is
+// scanned as usual, and its file is recorded by the path given.
+func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := t.TempDir()
+	writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
+
+	data := filepath.Join(dir, "data")
+
+	err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	daily := filepath.Join(data, "daily")
+	stderr := scanOK(t, daily)
+
+	summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
+		"0 unchanged), 0 skipped\n"
+	if !strings.Contains(stderr, summary) {
+		t.Errorf("stderr = %q, want %q", stderr, summary)
+	}
+
+	db, err := openDB(path, reportParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
+}
+
 func TestRunReportSucceeds(t *testing.T) {
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)
@@ -1028,8 +1158,9 @@ func TestRunTreesSucceeds(t *testing.T) {
 
 	// The two directories holding the duplicate pair are duplicate
 	// trees of each other.
-	want := "first\tdupe\tfiles\tsize\n" +
+	want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
-		filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
+		filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
+		strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
 	if got := stdout.String(); got != want {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
@@ -1063,9 +1194,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
 	cases := map[string]string{
 		cmdReport: "first\tdupe\tsize\n" +
 			dupes[0] + "\t" + dupes[1] + "\t300\n",
-		cmdTrees: "first\tdupe\tfiles\tsize\n" +
+		cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
 			filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
-			"\t1\t300\n",
+			"\t1\t300\t" +
+			strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
 	}
 
 	for name, want := range cases {
@@ -1100,8 +1232,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
 		t.Errorf("report stdout = %q, want %q", got, want)
 	}
 
-	want = "first\tdupe\tfiles\tsize\n" +
+	diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
-		filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
+
+	want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
+		filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
+		strconv.FormatInt(diskSize, 10) + "\n"
 	if got := runStdout(t, cmdTrees); got != want {
 		t.Errorf("trees stdout = %q, want %q", got, want)
 	}
@@ -1167,9 +1302,8 @@ func holdScanLock(t *testing.T, path string) {
 }
 
 func TestRunSecondScanFails(t *testing.T) {
-	// README §Database: while one scan holds the lock, a second scan
+	// README §Database: a second scan that finds the lock held fails
-	// fails at once, naming the lock file, without opening the
+	// at once, naming the lock file, without opening the database.
-	// database.
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)
 
@@ -1228,7 +1362,7 @@ func assertSecondScanFails(t *testing.T, path string) {
 }
 
 func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
-	// README §Database: every path to one database names the same lock
+	// README §Database: a path through a symlink names the same lock
 	// file, so while a scan holds it through one path, a scan through
 	// another fails, naming the lock file beside the database's real
 	// path.
@@ -1333,8 +1467,8 @@ func TestRunReportsDuringScan(t *testing.T) {
 }
 
 func TestRunStdoutClosedIsFatal(t *testing.T) {
-	// README §Error handling: a stdout write failure exits 1, reported
+	// README §Error handling: a failed write to stdout is a fatal error,
-	// in one line on stderr.
+	// exit 1, reported on stderr starting with "sfdupes:".
 	for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
 		t.Run(name, func(t *testing.T) {
 			t.Setenv(databaseEnv, testDBPath(t))

progress.go

@@ -60,7 +60,7 @@ const fallbackWidth = 80
 const spinnerMargin = 5
 
 // countingRecords is what the display of a phase that reads the records
-// says while it counts them, before it has its total.
+// says until it has the number of records, its total.
 const countingRecords = "counting the records"
 
 // openPhase is the name of the phase in which report, trees and info

report.go

@@ -21,9 +21,11 @@ const minGroupSize = 2
 
 // scanRec is one file record from the database. The signature (size,
 // head, tail, content) is the duplicate key; mtime is informational
-// only and used by scan for change detection.
+// only and used by scan for change detection. diskSize is the bytes the
+// file takes on disk, which trees adds up.
 type scanRec struct {
 	size     int64
+	diskSize int64
 	mtime    time.Time
 	head     [sha256.Size]byte
 	tail     [sha256.Size]byte
@@ -45,11 +47,11 @@ func reportPhase(name string) string {
 // problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and read (count the records,
+// own on stderr: open (open the database) and read (read the number of
-// then read them in signature order, writing each size's duplicate
+// records from the totals table, then read the records in signature
-// groups to stdout once that size is read; see readReport). Each
+// order, writing each size's duplicate groups to stdout once that size
-// display ends before the next phase starts, and before an error is
+// is read; see readReport). Each display ends before the next phase
-// returned to be printed.
+// starts, and before an error is returned to be printed.
 func runReport(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {

report_test.go

@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
 	h, tl, c := testHash("h"), testHash("t"), testHash("c")
 
 	return []scanRec{
-		{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"},
+		{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
-		{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
+		{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
 	}
 }
 

scan.go

@@ -52,22 +52,25 @@ const workQueueDepth = 1024
 // has already printed its line, so run prints nothing more.
 var errInterrupted = errors.New("scan interrupted")
 
-// fileRec carries one statted file between the scan phases. dev and
+// fileRec carries one statted file between the scan phases. diskSize is
-// ino identify the underlying inode so hard-linked paths can share
+// the bytes the file takes on disk. dev and ino identify the underlying
-// one read; both are zero when the platform exposes no inode.
+// inode so hard-linked paths can share one read; both are zero when the
+// platform exposes no inode.
 type fileRec struct {
 	path     string
 	size     int64
+	diskSize int64
 	mtime    time.Time
 	dev      uint64
 	ino      uint64
 }
 
 // fileMeta is the in-memory index entry for one existing database
-// record: just enough for change detection. Hashes stay on disk; at
+// record: what the walk compares the file with. Hashes stay on disk; at
 // tens of millions of records they would dominate the scan's memory.
 type fileMeta struct {
 	size     int64
+	diskSize int64
 	mtime    time.Time
 }
 
@@ -342,6 +345,7 @@ type scanState struct {
 	db        *sql.DB
 	existing  map[string]fileMeta
 	toHash    []fileRec // new and changed files: must be read
+	diskSizes []fileRec // unchanged files whose disk_size changed
 	batch     []scanRec
 	st        scanStats
 }
@@ -383,7 +387,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 ) error {
 	// Types and filesystems are checked before pruning so that an
 	// operand under a dropped one is still scanned, not dropped as lying
-	// under it.
+	// under it, and so that an operand inside a .zfs directory gets its
+	// warning even when it lies under another operand.
 	roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
 		deviceOfPath)
 
@@ -411,15 +416,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 }
 
 // walkableRoots returns the operands the walk can start from: regular
-// files, and directories not named .zfs. Every other operand is warned
+// files, and directories not named .zfs, none of them inside a
-// about, counted as skipped, and dropped. A dropped operand is no
+// directory named .zfs. Every other operand is warned about, counted as
-// longer a root, so the records stored beneath it are treated as if it
+// skipped, and dropped. A dropped operand is no longer a root, so the
-// had not been given: those under no other root count as outside the
+// records stored beneath it are treated as if it had not been given:
-// roots and are not deleted as unverified. An operand on a filesystem
+// those under no other root count as outside the roots and are not
-// in rules that is not persistent is dropped the same way, with a line
+// deleted as unverified. An operand on a filesystem in rules that is not
-// saying so that is not a warning, and is not counted as skipped. An
+// persistent is dropped the same way, with a line saying so that is not
-// operand that fails lstat here is kept, and the walk warns about it
+// a warning, and is not counted as skipped. An operand that fails lstat
-// unless it no longer exists.
+// here is kept, and the walk warns about it unless it no longer exists.
 func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 	kept := make([]string, 0, len(roots))
 
@@ -454,8 +459,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 // included, so its display counts against the number of records.
 func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 	// Indexing tens of millions of records takes real time; without a
-	// display the scan looks hung before the walk begins. Counting them
+	// display the scan looks hung before the walk begins. The number of
-	// first gives the display its total.
+	// records, read first, gives the display its total.
 	prog := newProgress(progressParams{
 		label: scanPhase("load"), unit: scanUnit, total: -1,
 		waitingFor: countingRecords, stalledOn: "db",
@@ -471,24 +476,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 
 	s.existing = make(map[string]fileMeta)
 
-	return loadFileMeta(ctx, s.db,
+	return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
-		func(path string, size int64, mtime time.Time) {
 		prog.increment()
 
 		if underAnyRoot(path, roots) {
-				s.existing[path] = fileMeta{size: size, mtime: mtime}
+			s.existing[path] = m
 		}
 	})
 }
 
 // walkPhase drains the walk and queues every new or changed file for
-// the hash phase. An unchanged file needs nothing more, and its record
+// the hash phase. An unchanged file is not read, and its record counts
-// counts as verified. A record at or below a path the walk left out,
+// as verified. When its disk_size differs from the record's, it is
-// because -x stopped at it, it is a default exclude, or it is not on a
+// queued for the update phase, which writes the new disk_size. A record
-// persistent filesystem, counts as outside the roots: it is not
+// at or below a path the walk left out, because -x stopped at it, it is
-// deleted. That does not hold below a root at or inside that path,
+// a default exclude, or it is not on a persistent filesystem, counts as
-// since that root's own walk verifies the records below it; nor is a
+// outside the roots: it is not deleted. That does not hold below a root
-// line printed for such a root.
+// at or inside that path, since that root's own walk verifies the
+// records below it; nor is a line printed for such a root.
 func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 	// The records the last scan left under the roots estimate how many
 	// files the walk will find. A first scan has none, and no estimate.
@@ -546,6 +551,13 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 		delete(s.existing, ev.rec.path)
 
 		s.st.unchanged++
+
+		// The space a file takes on disk can change while its size and
+		// mtime stay the same. ZFS, for one, counts a new file's blocks
+		// only seconds after the write.
+		if old.diskSize != ev.rec.diskSize {
+			s.diskSizes = append(s.diskSizes, ev.rec)
+		}
 	}
 
 	prog.finish()
@@ -695,6 +707,7 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
 
 		s.batch = append(s.batch, scanRec{
 			size:     rec.size,
+			diskSize: rec.diskSize,
 			mtime:    rec.mtime,
 			head:     r.head,
 			tail:     r.tail,
@@ -715,7 +728,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
 		return nil
 	}
 
-	err := applyBatch(ctx, s.db, s.batch, nil, nil)
+	err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
 	if err != nil {
 		return err
 	}
@@ -726,9 +739,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
 }
 
 // updatePhase writes the scan's tail under one progress display: the
-// final partial batch of hashed records, and deletions for every record
+// final partial batch of hashed records, the new disk_size of every
-// the scan did not verify (vanished files, plus paths that failed to
+// unchanged file whose disk_size changed, and deletions for every
-// stat or hash).
+// record the scan did not verify (vanished files, plus paths that
+// failed to stat or hash).
 func (s *scanState) updatePhase(ctx context.Context) error {
 	deletes := make([]string, 0, len(s.existing))
 	for path := range s.existing {
@@ -740,7 +754,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 
 	s.st.removed = len(deletes)
 
-	total := len(s.batch) + len(deletes)
+	total := len(s.batch) + len(s.diskSizes) + len(deletes)
 	prog := newProgress(progressParams{
 		label: scanPhase("update"), unit: scanUnit, total: int64(total),
 		stalledOn: "db",
@@ -755,6 +769,11 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 
 	s.batch = nil
 
+	err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
+	if err != nil {
+		return err
+	}
+
 	return applyChanges(ctx, s.db, nil, deletes, prog)
 }
 
@@ -874,9 +893,24 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
 
 // operandWarning returns the one-line warning for an operand the walk
 // does not start from, naming the path and what it is, or "" for one it
-// does: a regular file, or a directory not named .zfs. Symlinks are
+// does: a regular file, or a directory not named .zfs, that is not
-// never followed, including as operands.
+// inside a directory named .zfs, neither as given nor with its symlinks
+// resolved. Symlinks are never followed, including as operands.
 func operandWarning(root string, fi fs.FileInfo) string {
+	// The walk never enters a .zfs directory, so it starts from nothing
+	// inside one either, not even through a symlink. An operand whose
+	// symlinks cannot be resolved is checked as given.
+	resolved, err := filepath.EvalSymlinks(root)
+	if err != nil {
+		resolved = root
+	}
+
+	if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
+		slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
+		return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
+			root)
+	}
+
 	var kind string
 
 	switch mode := fi.Mode(); {
@@ -947,6 +981,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
 	rec := fileRec{
 		path:     root,
 		size:     fi.Size(),
+		diskSize: diskSizeOfInfo(fi),
 		mtime:    fi.ModTime(),
 		dev:      dev,
 		ino:      ino,
@@ -1131,6 +1166,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 	rec := fileRec{
 		path:     p,
 		size:     info.Size(),
+		diskSize: diskSizeOfInfo(info),
 		mtime:    info.ModTime(),
 		dev:      dev,
 		ino:      ino,
@@ -1225,6 +1261,23 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
 	return statDev(st), st.Ino
 }
 
+// statBlockSize is the unit of st_blocks in an lstat result on Linux and
+// macOS, whatever the filesystem's own block size.
+const statBlockSize = 512
+
+// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
+// blocks, st_blocks × 512. On a compressed or sparse file it is less
+// than the file's size. It is 0 when the platform exposes no block
+// count.
+func diskSizeOfInfo(fi fs.FileInfo) int64 {
+	st, ok := fi.Sys().(*syscall.Stat_t)
+	if !ok {
+		return 0
+	}
+
+	return st.Blocks * statBlockSize
+}
+
 // hashResult carries the hashes computed for one inode run (or the
 // error that prevented computing them) from the pool's workers to the
 // hash phase. Its run leaves out the paths found removed.

scan_test.go

@@ -42,6 +42,30 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
 	return p
 }
 
+// diskSizeOf returns the bytes the file at path takes on disk, from its
+// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
+// blocks only when it commits the write, up to seconds later, and one
+// commit can take the file through several growing sizes. So this value
+// can change at any moment. Neither fsync nor unix.Sync hastens that,
+// but a unix.Sync call often takes long enough that the blocks are
+// counted when it returns. A test that needs the value to change waits
+// for the change. A test that compares it with a scan's disk_size reads
+// it before and after the scan. The rewrite test rescans until the two
+// reads match, then compares exactly. The other two tests fail only on
+// a disk_size outside the two reads.
+func diskSizeOf(t *testing.T, path string) int64 {
+	t.Helper()
+
+	var st syscall.Stat_t
+
+	err := syscall.Lstat(path, &st)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	return st.Blocks * 512
+}
+
 // pattern returns n bytes of deterministic content seeded by tag.
 func pattern(tag byte, n int) []byte {
 	data := make([]byte, n)
@@ -1064,8 +1088,9 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
 	}
 }
 
-// buildSmokeTree recreates the README smoke-test filesystem layout
+// buildSmokeTree recreates the tree of the test under README
-// with deterministic content and returns the tree root.
+// "Definition of done" with deterministic content and returns the tree
+// root.
 func buildSmokeTree(t *testing.T) string {
 	t.Helper()
 
@@ -1228,9 +1253,9 @@ func TestScanPipeline(t *testing.T) {
 // TestScanProgressNamesEachPhase checks the plain progress lines of a
 // first scan, and of a rescan of one operand holding two of the
 // database's records. Each phase's first line names it with its place
-// among the four: load counts the records before reading them, and the
+// among the four: load gets the number of records before reading them,
-// walk estimates its total from the records under its operand, or says
+// and the walk estimates its total from the records under its operand,
-// its time remaining is unknown when there are none.
+// or says its time remaining is unknown when there are none.
 //
 //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
 func TestScanProgressNamesEachPhase(t *testing.T) {
@@ -1416,6 +1441,159 @@ func TestScanStoresRawHashes(t *testing.T) {
 	}
 }
 
+// TestScanStoresDiskSize scans a sparse file, which takes less space on
+// disk than its length, once through its directory and once as the
+// operand itself. Each scan stores the disk_size of the file's lstat.
+func TestScanStoresDiskSize(t *testing.T) {
+	t.Parallel()
+
+	const length = 1024 * 1024
+
+	p := sparseFile(t, t.TempDir(), "sparse", length)
+	pokeAt(t, p, 0, []byte("x"))
+
+	unix.Sync() // see diskSizeOf
+
+	size := diskSizeOf(t, p)
+	if size == 0 || size >= length {
+		t.Fatalf("the file takes %d bytes on disk, want more than none "+
+			"and fewer than its length %d", size, length)
+	}
+
+	for _, operand := range []string{filepath.Dir(p), p} {
+		db := openTestDB(t)
+
+		// ZFS may count the blocks during the scan; see diskSizeOf.
+		before := diskSizeOf(t, p)
+
+		syncTree(t, db, operand)
+
+		after := diskSizeOf(t, p)
+
+		got := recordByPath(t, dbRecords(t, db), p).diskSize
+		if got < before || got > after {
+			t.Errorf("scan of %s: disk_size = %d, want %d to %d",
+				operand, got, before, after)
+		}
+	}
+}
+
+// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
+// sparse file and moves its mtime on. The rescan hashes the file again
+// and rewrites its record with the new disk_size.
+func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	db := openTestDB(t)
+	p := sparseFile(t, dir, "sparse", 1024*1024)
+	pokeAt(t, p, 0, []byte("x"))
+
+	syncTree(t, db, dir)
+
+	recorded := recordByPath(t, dbRecords(t, db), p).diskSize
+
+	pokeAt(t, p, 512*1024, pattern(7, 64*1024))
+
+	// The write may leave the mtime the scan recorded; a later one makes
+	// the rescan read the file again.
+	later := time.Now().Add(time.Hour)
+
+	err := os.Chtimes(p, later, later)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Wait for ZFS to count the write; see diskSizeOf.
+	deadline := time.Now().Add(30 * time.Second)
+	for diskSizeOf(t, p) == recorded {
+		if time.Now().After(deadline) {
+			t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
+				"after the write into its hole", recorded)
+		}
+
+		unix.Sync()
+	}
+
+	// ZFS may still be counting, so the size may change during a rescan.
+	// Only a rescan with the same size on both sides of it can be compared
+	// exactly. A rescan that keeps the old size still fails, because the
+	// wait made before differ from it.
+	for {
+		before := diskSizeOf(t, p)
+
+		st := syncTree(t, db, dir)
+
+		after := diskSizeOf(t, p)
+
+		if st != (scanStats{walked: 1, updated: 1}) {
+			t.Fatalf("rescan stats = %+v, want 1 updated", st)
+		}
+
+		if before == after {
+			got := recordByPath(t, dbRecords(t, db), p).diskSize
+			if got != after {
+				t.Errorf("disk_size = %d, want %d", got, after)
+			}
+
+			return
+		}
+
+		if time.Now().After(deadline) {
+			t.Fatalf("the size on disk was still changing 30 seconds after "+
+				"the write into its hole: %d, then %d", before, after)
+		}
+
+		// A later mtime makes the next rescan read the file again.
+		later = later.Add(time.Second)
+
+		err = os.Chtimes(p, later, later)
+		if err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
+// disk_size the file does not have, as when ZFS had not yet counted the
+// file's blocks at the last scan. The rescan writes the file's own
+// disk_size into the record without reading the file again.
+func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	db := openTestDB(t)
+	p := writeFile(t, dir, "a.bin", pattern(1, 500))
+
+	syncTree(t, db, dir)
+
+	// lstat counts 512-byte units, so no file takes 1 byte on disk.
+	_, err := db.ExecContext(t.Context(),
+		"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	unix.Sync() // see diskSizeOf
+
+	// ZFS may count the blocks during the rescan; see diskSizeOf.
+	before := diskSizeOf(t, p)
+
+	// A file read again would count as updated.
+	st := syncTree(t, db, dir)
+
+	after := diskSizeOf(t, p)
+
+	if st != (scanStats{walked: 1, unchanged: 1}) {
+		t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
+	}
+
+	got := recordByPath(t, dbRecords(t, db), p).diskSize
+	if got < before || got > after {
+		t.Errorf("disk_size = %d, want %d to %d", got, before, after)
+	}
+}
+
 // TestSyncScanSameSecondRewrite rewrites a file in place at the same
 // size with an mtime later in the same second as the recorded one: the
 // next scan must notice the change and re-hash the file. The operand is

trees.go

@@ -22,6 +22,10 @@ type treeNode struct {
 	digest    [sha256.Size]byte
 	fileCount int64
 	totalSize int64
+	// diskSize is the bytes the directory's files take on disk. Two
+	// copies of one tree have the same totalSize, but their diskSize
+	// can differ.
+	diskSize int64
 }
 
 // treesPhase names one phase of trees with its place among all of them,
@@ -40,11 +44,11 @@ func treesPhase(name string) string {
 // database problem, including a missing database, is fatal.
 //
 // The work runs in four phases, each under a progress display of its
-// own on stderr: open (open the database), read (count the records,
+// own on stderr: open (open the database), read (read the number of
-// then read them into the hierarchy), group (group the directories by
+// records from the totals table, then read the records into the
-// digest) and write (write the groups to stdout). Each display ends
+// hierarchy), group (group the directories by digest) and write (write
-// before the next phase starts, and before an error is returned to be
+// the groups to stdout). Each display ends before the next phase
-// printed.
+// starts, and before an error is returned to be printed.
 func runTrees(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -139,15 +143,16 @@ func readTree(ctx context.Context, db *sql.DB,
 
 // writeTreeGroups writes dupes to stdout as TSV, a header line and then
 // a row for every member of each group but its first, counting each
-// group on prog once its rows are written. stdout may be the terminal
+// group on prog once its rows are written. The disk_size column is the
-// prog is drawn on, so the lines go through prog's writer. It returns
+// dupe's own. stdout may be the terminal prog is drawn on, so the lines
-// the number of rows (dupe trees) and the bytes they could reclaim.
+// go through prog's writer. It returns the number of rows (dupe trees)
+// and the bytes they could reclaim.
 func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
 	prog *progress,
 ) (int, int64, error) {
 	out := prog.writer(stdout)
 
-	_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
+	_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
 	if err != nil {
 		return 0, 0, fmt.Errorf("write stdout: %w", err)
 	}
@@ -159,9 +164,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
 	for _, g := range dupes {
 		first := g[0]
 		for _, n := range g[1:] {
-			_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
+			_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
 				escapePath(first.path), escapePath(n.path),
-				first.fileCount, first.totalSize)
+				first.fileCount, first.totalSize, n.diskSize)
 			if err != nil {
 				return 0, 0, fmt.Errorf("write stdout: %w", err)
 			}
@@ -232,6 +237,7 @@ func (b *treeBuilder) add(r scanRec) {
 	dir.entries = append(dir.entries, fileEntry(name, r))
 	dir.fileCount++
 	dir.totalSize += r.size
+	dir.diskSize += r.diskSize
 }
 
 // openDir starts the directory called name inside the innermost open
@@ -270,6 +276,7 @@ func (b *treeBuilder) closeTo(n int) {
 			"d\x00"+name+"\x00"+string(dir.digest[:]))
 		dir.parent.fileCount += dir.fileCount
 		dir.parent.totalSize += dir.totalSize
+		dir.parent.diskSize += dir.diskSize
 		b.dirs = append(b.dirs, dir)
 	}
 }

trees_test.go

@@ -15,20 +15,26 @@ import (
 const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
 	"remain unknown"
 
-// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
+// smokeTreeRecs mirrors the tree of the test under README "Definition
-// /d/t2 are identical, /d/t3 differs from them only by one filename.
+// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
+// by one filename. Each file takes one 4 KiB block on disk.
 func smokeTreeRecs() []scanRec {
 	f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
 	f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
 
 	return []scanRec{
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
+		{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
+			content: f1Content, path: "/d/t1/f1"},
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
+		{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
+			content: f2Content, path: "/d/t1/sub/f2"},
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
+		{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
+			content: f1Content, path: "/d/t2/f1"},
-			path: "/d/t3/sub/f2renamed"},
+		{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
+			content: f2Content, path: "/d/t2/sub/f2"},
+		{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
+			content: f1Content, path: "/d/t3/f1"},
+		{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
+			content: f2Content, path: "/d/t3/sub/f2renamed"},
 	}
 }
 
@@ -198,13 +204,33 @@ func TestRunTreesEscapesPaths(t *testing.T) {
 			code, exitOK, stderr.String())
 	}
 
-	want := "first\tdupe\tfiles\tsize\n" +
+	want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
-		`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
+		`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
 	if got := stdout.String(); got != want {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
 }
 
+// TestRunTreesDiskSize runs trees on two copies of a tree whose files
+// take different space on disk. The row's disk_size is the sum over the
+// dupe copy's files, those in its subdirectory included.
+func TestRunTreesDiskSize(t *testing.T) {
+	f, g := testHash("f"), testHash("g")
+	recs := []scanRec{
+		{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
+		{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
+		{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
+		{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
+	}
+
+	t.Setenv(databaseEnv, seedDatabase(t, recs))
+
+	want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
+	if got := runStdout(t, cmdTrees); got != want {
+		t.Errorf("stdout = %q, want %q", got, want)
+	}
+}
+
 // TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
 // records with stderr not a terminal, and checks every line it prints
 // there. After the line naming the database, each phase prints a line
@@ -226,7 +252,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
 			code, exitOK, stderr())
 	}
 
-	report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
+	report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
+		"/d/t1\t/d/t2\t2\t3100\t8192\n"
 	if got := stdout.String(); got != report {
 		t.Errorf("stdout = %q, want %q", got, report)
 	}
@@ -291,7 +318,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
 		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
 	}
 
-	want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
+	want := []string{
+		"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
+	}
 	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
 		t.Errorf("terminal shows lines %q, want %q first", shown, want)
 	}