Rewrite README.md sentence by sentence for a newcomer

The old README was one dense specification: long sentences, lists
chained with commas, exceptions ahead of the rules, and release
history in the user text. This rewrite leads with what the program is
and how to use it, then gives a reference for the settings, each
subcommand, the database, the output formats, the progress display
and the exit codes. Non-goals come before the reference; the
constraints, the entrypoints and the build come last. Every statement
was checked against the code on next and against what the built
binary prints on small trees. The section names the code comments
cite are kept. Five comments in main_test.go cite Settings, and one
each in scan_test.go, trees_test.go and exclude_test.go cites what
the README now says. TODO.md records the step.

Model: fable-5-1

README.md

@@ -18,11 +18,11 @@ a strong candidate, not proof.
 
 `sfdupes` is meant for multi-terabyte servers, where reading every byte of every
 file takes too long. It has four subcommands. `scan` walks the filesystem and
-records every file in an SQLite database. SQLite keeps a whole database in one
-file. The database is kept between runs. A rescan reads only new and changed
-files. That makes `scan` cheap enough to run every day from `cron`, the system's
-job scheduler. `report`, `trees` and `info` read the database alone and can run
-at any time.
+records the files it finds in an SQLite database. SQLite keeps a whole database
+in one file. The database is kept between runs. A rescan reads only new and
+changed files. That makes `scan` cheap enough to run every day from `cron`, the
+system's job scheduler. `report`, `trees` and `info` read the database alone and
+can run at any time.
 
 ## Getting Started
 
@@ -45,10 +45,9 @@ database's path, size and record count.
 The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
 another file. The first `scan` creates it.
 
-`report`, `trees` and `info` write their data to stdout. Everything else, such
-as progress and warnings, goes to stderr. stdout and stderr are a program's two
-output streams. The shell shows both on the terminal, and `>` redirects stdout
-alone.
+`report`, `trees` and `info` write their data to stdout. Progress, warnings and
+errors go to stderr. stdout and stderr are a program's two output streams. The
+shell shows both on the terminal, and `>` redirects stdout alone.
 
 ### Install
 
@@ -122,8 +121,10 @@ skipped with a warning and loses its database record. As the file
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each path the walk leaves out as a system or temporary
-      directory (see "Rules for the walk");
+    - a line for each system or temporary directory the walk leaves out (see
+      "Rules for the walk");
+    - a line for each path left out because its filesystem is one the kernel
+      makes up or keeps in memory, such as `tmpfs`;
     - the summary line;
     - a warning when a report still has the database open as the scan ends (see
       "Concurrent reads").
@@ -172,8 +173,8 @@ where the files differ. The second line prints `hard links` when both paths are
 the same file.
 
 A path holding a special character, such as a tab, is escaped in the reports.
-The character is written as a backslash and a letter, `\t` for a tab (see
-"Report output format"). Undo the escapes before using such a path.
+The character is written as a backslash and one more character, `\t` for a tab
+(see "Report output format"). Undo the escapes before using such a path.
 `printf '%b'` does that, because every backslash in an escaped path starts an
 escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
 newline at the end of a path, print an `x` after the path and remove it
@@ -347,23 +348,36 @@ default. An operand may be a directory or a regular file.
   absolute path, so results do not depend on the working directory.
 - All operands belong to one scan, and one walk covers them all at once (see
   "Concurrency").
-- An operand that repeats another, or lies under another, is dropped before the
-  walk, without a line. Every file is then reached once. An operand under
-  another is kept when the outer operand's walk would not reach it. That is the
-  case when the inner operand, or a directory between the two, is one of these:
+- An operand that repeats another is dropped before the walk, without a line.
+- An operand under another operand is dropped the same way, unless one of the
+  cases below applies. No file is then reached twice.
+- An operand under another is kept, and walked on its own, when the inner
+  operand or a directory between the two is one of these:
     - a default exclude (see "Rules for the walk");
     - with `-x`, a directory on another filesystem than the outer operand.
 
-    It is also the case when a directory between the two is on a filesystem that
-    is not persistent. Such a filesystem is one the kernel makes up or keeps in
+    It is also kept when a directory between the two is on a filesystem that is
+    not persistent. Such a filesystem is one the kernel makes up or keeps in
     memory (see "Rules for the walk"). The outer operand's walk does not enter
-    such a directory, so the inner operand is walked on its own.
+    such a directory, so the inner operand is walked on its own. For an inner
+    operand that is a regular file, the check starts at its directory.
 
-- An operand that is neither a regular file nor a directory is not scanned, and
-  neither is a directory named `.zfs`. `scan` prints a warning naming the path
-  and what it is. It counts the operand as skipped and drops it. An operand
-  beneath a dropped one is scanned all the same. The scan keeps the records
-  beneath a dropped operand, unless another operand covers them.
+- No other case keeps an operand under another. An operand reached through a
+  symlinked directory below the outer operand is dropped too, without a line.
+  The outer operand's walk does not follow the symlink, so no file under the
+  inner operand is scanned. The records an earlier scan left under it are
+  deleted.
+- An operand that is neither a regular file nor a directory is not scanned.
+  `scan` prints a warning naming the path and what it is. It counts the operand
+  as skipped and drops it. An operand beneath a dropped one is scanned all the
+  same. The scan keeps the records beneath a dropped operand, unless another
+  operand covers them.
+- An operand at or inside a directory named `.zfs` is never scanned, whether
+  given alone or under another operand. The check looks at the path as given and
+  at the path with its symlinks resolved. `scan` prints a warning naming the
+  path, counts the operand as skipped and drops it. The records beneath it are
+  kept, unless it lies under another operand. Then they are deleted, because
+  that operand's walk never enters `.zfs` (see "Rules for the walk").
 - An operand on a filesystem that is not persistent is dropped too. `scan`
   prints a line naming it and its filesystem type, and does not count it as
   skipped.
@@ -437,7 +451,8 @@ walk practical on large or busy storage pools.
   or a device node, is passed over. Symlinks are never followed.
 - The walk never enters a directory named `.zfs`. Such a directory holds the
   filesystem's ZFS snapshots, which are its saved earlier states. Walking it
-  would list every file once per snapshot.
+  would list every file once per snapshot. `scan` never starts a walk at or
+  inside one either (see "Operands").
 - The walk never records the scan's own database files:
     - the database file;
     - its `.lock` file (see "The scan lock");
@@ -559,7 +574,8 @@ from a thread of its own and never blocks the workers.
 
 - one line naming the database (see "Database");
 - for each dropped operand, in operand order, the warning or the line described
-  under "Operands". A repeated operand, or one under another, gets neither;
+  under "Operands". An operand dropped as a repeat of another, or as lying under
+  another, gets neither;
 - the progress display of each phase (see "Progress"). `scan` prints each
   warning for a skipped path as it happens. It prints a line the same way for
   each path left out as a default exclude or as not on a persistent filesystem;
@@ -602,12 +618,12 @@ and `scan` leaves it ignored.
 
 ### `report` mode
 
-`report` reads every record from the database. It takes no positional arguments,
-meaning nothing may follow `report` on the command line but flags. It never
-touches the scanned filesystem. It neither stats nor opens any path in the
-records. It reads the database and writes to stdout and stderr, and nothing
-else. Its output is the same whether or not the scanned filesystem is still
-mounted.
+`report` reads every record from the database. It takes no positional arguments.
+A positional argument is anything on the command line that is not a flag, such
+as the paths `scan` takes. `report` never touches the scanned filesystem. It
+neither stats nor opens any path in the records. It reads the database and
+writes to stdout and stderr, and nothing else. Its output is the same whether or
+not the scanned filesystem is still mounted.
 
 Processing:
 
@@ -773,9 +789,10 @@ size of recorded files	150000000000000 (136.4 TiB)
 
 `info` has two phases, each with its own progress display on stderr (see
 "Progress"). `open` opens the database. `count` reads the schema version and
-counts the records in one transaction, a read of the database at one moment. So
-the two agree with each other even while a scan is writing. `info` writes its
-lines after the `count` display has ended.
+counts the records in one transaction. A transaction is a group of reads or
+writes that SQLite carries out as one step, against the database as it stands at
+one moment. So both describe the database at one moment, even while a scan is
+writing. `info` writes its lines after the `count` display has ended.
 
 ### Database
 
@@ -884,11 +901,11 @@ Only one `scan` runs against a database at a time.
 The schema is the database's layout, its table and its index.
 `PRAGMA user_version` holds the schema version, currently 1. A database with any
 other version is a fatal error. `scan` creates the table and the index and sets
-the version in one transaction, a group of writes that lands whole or not at
-all. A first scan stopped while doing so leaves an empty database, which the
-next scan sets up. A database at version 0 that already has a `files` table was
-not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
-remove the file and rescan.
+the version in one transaction (see "`info` mode"). So the three land together
+or not at all. A first scan stopped while doing so leaves an empty database,
+which the next scan sets up. A database at version 0 that already has a `files`
+table was not made by `sfdupes`. Every subcommand refuses it, with an error
+telling you to remove the file and rescan.
 
 ```sql
 CREATE TABLE files (
@@ -982,8 +999,8 @@ while a disk or the database is slow. Each phase names the places it may wait
 in, and a phase that may wait in two places names both, as `disk/db`:
 
 - `walk` says `disk`, for files.
-- `hash` says `disk/db`: it reads files and writes their records to the database
-  between them.
+- `hash` says `disk/db`, because it reads files and writes their records to the
+  database between them.
 - `load`, `update` and the `read` phase of `trees` say `db`.
 - The `read` phase of `report` says `db/out`, because it writes the report to
   stdout while it reads.
@@ -1084,9 +1101,8 @@ This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
 workflow, and the `Makefile` targets call them, except `build` and `clean` (see
-"Build"). Every script runs under plain `sh`, the standard Unix shell, and uses
-no `bash` feature. It finds the repository root itself, so it can be run from
-any directory.
+"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds
+the repository root itself, so it can be run from any directory.
 
 - `script/bootstrap` installs what building and developing need. It can be run
   again at any time, and installed tools are kept. `git`, `make` and `go` are
@@ -1223,10 +1239,10 @@ commands of their own:
 
     No other dependency outside the standard library is added.
 
-- The code stays pure Go, with no C code. Cross-compilation is not a concern.
-  `modernc.org/sqlite` is pure Go too, so the binary builds without cgo. The
-  `Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
-  because the race detector needs cgo (see `script/test`).
+- The code stays pure Go, with no C code. Building on one system for another is
+  not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
+  cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
+  exception, because the race detector needs cgo (see `script/test`).
 - `report` and `trees` give the same output for the same database contents,
   whatever order the records were inserted in.
 
@@ -1294,11 +1310,11 @@ All of the following, run in this directory, must pass:
 
 ## TODO
 
-Tracked in [TODO.md](TODO.md).
+Tracked in [`TODO.md`](TODO.md).
 
 ## License
 
-MIT. See [LICENSE](LICENSE).
+MIT. See [`LICENSE`](LICENSE).
 
 ## Author
 

TODO.md

@@ -33,8 +33,14 @@
   it. It leads with what the program is and how to use it. A reference section
   covers every setting and every subcommand. Development and testing come last.
   Every statement was checked against the code on `next`. Five comments in
-  `main_test.go` cite "Settings", where their rules now live (2026-10-26,
-  https://git.eeqj.de/sneak/sfdupes/issues/135)
+  `main_test.go` cite "Settings", where their rules now live. One comment each
+  in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
+  now says (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
+
+- an operand at or inside a directory named `.zfs` is never scanned, whether
+  given alone or under another operand. `scan` warns about it, counts it as
+  skipped and does not walk it (2026-10-26,
+  https://git.eeqj.de/sneak/sfdupes/issues/142)
 
 - a file or directory removed while the scan runs is not an error, in the walk
   or in the hash phase. The scan prints no line for it and does not count it as

exclude_test.go

@@ -17,7 +17,9 @@ import (
 func TestDefaultExcludes(t *testing.T) {
 	t.Parallel()
 
-	// README "Rules for the walk": each system's list.
+	// README "Rules for the walk": the Linux and macOS lists, and none
+	// on any other system. The Windows list exists in the code only: the
+	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
 		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
 		goosDarwin: {

main_test.go

@@ -928,24 +928,21 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
 }
 
 func TestRunScanSkipsZFSOperand(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
 	zfs := filepath.Join(t.TempDir(), ".zfs")
-	snapshot := filepath.Join(zfs, "snapshot", "hourly")
-	f := writeFile(t, snapshot, "f", pattern(1, 10))
+	f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
 
-	// An operand beneath a .zfs directory is walked, because it is not
-	// itself named .zfs.
-	scanOK(t, snapshot)
+	// No scan records a file inside a .zfs directory, so the record
+	// beneath the operand is written straight into the database.
+	path := seedDatabase(t, []scanRec{{size: 10, path: f}})
+	t.Setenv(databaseEnv, path)
 
 	assertOperandSkipped(t, path, zfs, ".zfs directory", f)
 }
 
 // assertOperandSkipped scans operand alone and checks that it is skipped
 // as kind: a warning naming it, one skip in the summary, exit 0, and the
-// record for kept, which an earlier scan stored beneath operand, still
-// in the database at dbPath.
+// record for kept, stored beneath operand before this scan, still in the
+// database at dbPath.
 func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
 	kept string,
 ) {
@@ -974,6 +971,121 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
 	recordByPath(t, dbRecords(t, db), kept)
 }
 
+// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
+// directory, first alone and then under another operand. Each time it
+// is warned about and counted as skipped, and nothing in it is walked.
+// The other operand is scanned as usual.
+func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := t.TempDir()
+	hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
+	writeFile(t, hourly, "f", pattern(1, 10))
+	f := writeFile(t, dir, "f", pattern(2, 10))
+
+	warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
+
+	for _, tc := range []struct {
+		operands []string
+		summary  string
+	}{
+		{
+			[]string{hourly},
+			"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
+				"0 unchanged), 1 skipped\n",
+		},
+		{
+			[]string{dir, hourly},
+			"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
+				"0 unchanged), 1 skipped\n",
+		},
+	} {
+		stderr := scanOK(t, tc.operands...)
+		if !strings.Contains(stderr, warning) ||
+			!strings.Contains(stderr, tc.summary) {
+			t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
+				stderr, warning, tc.summary)
+		}
+	}
+
+	db, err := openDB(path, reportParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
+	}
+}
+
+// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
+// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
+// by the path given, counted as skipped, and not walked.
+func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
+	t.Setenv(databaseEnv, testDBPath(t))
+
+	dir := t.TempDir()
+	writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
+
+	snaps := filepath.Join(dir, "snaps")
+
+	err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	daily := filepath.Join(snaps, "daily")
+	stderr := scanOK(t, daily)
+
+	warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
+	summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
+		"0 unchanged), 1 skipped\n"
+
+	if !strings.Contains(stderr, warning) ||
+		!strings.Contains(stderr, summary) {
+		t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
+	}
+}
+
+// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
+// a symlink to a directory outside any .zfs directory. The operand is
+// scanned as usual, and its file is recorded by the path given.
+func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := t.TempDir()
+	writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
+
+	data := filepath.Join(dir, "data")
+
+	err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	daily := filepath.Join(data, "daily")
+	stderr := scanOK(t, daily)
+
+	summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
+		"0 unchanged), 0 skipped\n"
+	if !strings.Contains(stderr, summary) {
+		t.Errorf("stderr = %q, want %q", stderr, summary)
+	}
+
+	db, err := openDB(path, reportParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
+}
+
 func TestRunReportSucceeds(t *testing.T) {
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)

scan.go

@@ -383,7 +383,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 ) error {
 	// Types and filesystems are checked before pruning so that an
 	// operand under a dropped one is still scanned, not dropped as lying
-	// under it.
+	// under it, and so that an operand inside a .zfs directory gets its
+	// warning even when it lies under another operand.
 	roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
 		deviceOfPath)
 
@@ -411,15 +412,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 }
 
 // walkableRoots returns the operands the walk can start from: regular
-// files, and directories not named .zfs. Every other operand is warned
-// about, counted as skipped, and dropped. A dropped operand is no
-// longer a root, so the records stored beneath it are treated as if it
-// had not been given: those under no other root count as outside the
-// roots and are not deleted as unverified. An operand on a filesystem
-// in rules that is not persistent is dropped the same way, with a line
-// saying so that is not a warning, and is not counted as skipped. An
-// operand that fails lstat here is kept, and the walk warns about it
-// unless it no longer exists.
+// files, and directories not named .zfs, none of them inside a
+// directory named .zfs. Every other operand is warned about, counted as
+// skipped, and dropped. A dropped operand is no longer a root, so the
+// records stored beneath it are treated as if it had not been given:
+// those under no other root count as outside the roots and are not
+// deleted as unverified. An operand on a filesystem in rules that is not
+// persistent is dropped the same way, with a line saying so that is not
+// a warning, and is not counted as skipped. An operand that fails lstat
+// here is kept, and the walk warns about it unless it no longer exists.
 func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 	kept := make([]string, 0, len(roots))
 
@@ -874,9 +875,24 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
 
 // operandWarning returns the one-line warning for an operand the walk
 // does not start from, naming the path and what it is, or "" for one it
-// does: a regular file, or a directory not named .zfs. Symlinks are
-// never followed, including as operands.
+// does: a regular file, or a directory not named .zfs, that is not
+// inside a directory named .zfs, neither as given nor with its symlinks
+// resolved. Symlinks are never followed, including as operands.
 func operandWarning(root string, fi fs.FileInfo) string {
+	// The walk never enters a .zfs directory, so it starts from nothing
+	// inside one either, not even through a symlink. An operand whose
+	// symlinks cannot be resolved is checked as given.
+	resolved, err := filepath.EvalSymlinks(root)
+	if err != nil {
+		resolved = root
+	}
+
+	if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
+		slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
+		return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
+			root)
+	}
+
 	var kind string
 
 	switch mode := fi.Mode(); {

scan_test.go

@@ -1064,8 +1064,9 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
 	}
 }
 
-// buildSmokeTree recreates the README smoke-test filesystem layout
-// with deterministic content and returns the tree root.
+// buildSmokeTree recreates the tree of the test under README
+// "Definition of done" with deterministic content and returns the tree
+// root.
 func buildSmokeTree(t *testing.T) string {
 	t.Helper()
 

trees_test.go

@@ -15,8 +15,9 @@ import (
 const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
 	"remain unknown"
 
-// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
-// /d/t2 are identical, /d/t3 differs from them only by one filename.
+// smokeTreeRecs mirrors the tree of the test under README "Definition
+// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
+// by one filename.
 func smokeTreeRecs() []scanRec {
 	f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
 	f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")