Rewrite README.md sentence by sentence for a newcomer

The old README was a specification written as one dense document:
sentences past 30 words, lists chained with commas, exceptions ahead
of the rules, and release history in the user text. This rewrite
leads with what the program is, what it is for and how to use it,
then gives a reference for the settings, each subcommand, the
database, the output formats, the progress display and the exit
codes. Non-goals come before the reference; the constraints, the
entrypoints and the build come last. Every statement was checked
against the code on next and against what the built binary prints on
a small tree. The section names the code comments cite are kept, and
three comments in main_test.go now cite Settings. TODO.md records the
step.

Model: fable-5-1

README.md

@@ -20,7 +20,7 @@ proof.
 file takes too long. It has four subcommands. `scan` walks the filesystem and
 keeps its results between runs in an SQLite database, a database held in one
 file. A rescan reads only new and changed files, so `scan` can run every day
-from cron, the system's job scheduler. `report`, `trees` and `info` read the
+from `cron`, the system's job scheduler. `report`, `trees` and `info` read the
 database alone and can run at any time.
 
 ## Getting Started
@@ -67,9 +67,9 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
 `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
 a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
 string for an untagged commit, which ends in the commit's hash. A binary built
-from a clone carries the git tag or commit it was built from. `VERSION` is `dev`
+from a clone carries the `git` tag or commit it was built from. `VERSION` is
-only when the binary carries no version at all. To see everything Go recorded in
+`dev` when no version is known. To see everything Go recorded in an installed
-an installed binary:
+binary:
 
 ```sh
 go version -m "$(command -v sfdupes)"
@@ -83,14 +83,15 @@ cd sfdupes
 make build
 ```
 
-Copy the binary to `/usr/local/bin` for the cron job below.
+Copy the binary to `/usr/local/bin` for the `cron` job below.
 
 `make docker` builds a Docker image tagged `sfdupes`. The build runs the tests
 and the linter first (see "Build"). Inside the image `sfdupes` runs as root,
-with the database at its default path. Bind-mount `/var/lib/sfdupes` to keep the
+with the database at its default path. Each `-v` flag below makes a bind mount,
-database between runs. Mount the tree to scan at the same path inside the
+which shows a host directory at a path inside the container. Bind-mount
-container as on the host; read-only is enough. The database then records the
+`/var/lib/sfdupes` to keep the database between runs. Mount the tree to scan at
-host's paths, and so do the reports.
+the same path inside the container as on the host; read-only is enough. The
+database then records the host's paths, and so do the reports.
 
 ```sh
 make docker
@@ -99,7 +100,7 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
 docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
 ```
 
-### Daily scan from cron
+### Daily scan from `cron`
 
 Run `scan` as root, so that it can read every file. A path it cannot read is
 skipped with a warning and loses its database record. As the file
@@ -120,18 +121,21 @@ skipped with a warning and loses its database record. As the file
     - a warning for each path it skips;
     - a line for each path the walk leaves out on purpose (see "Rules for the
       walk");
-    - the summary line.
+    - the summary line;
+    - a warning when a report still has the database open as the scan ends (see
+      "Concurrent reads").
 - The log grows with every scan; rotate it like any other log.
-- Skipped paths do not fail a scan. It still exits 0, and cron sends no mail.
+- Skipped paths do not fail a scan. It still exits 0, and `cron` sends no mail.
 - A scan that fails exits 1, with the reason among the last lines of the log.
-  `tail` prints those lines, and cron mails them to root if the host can send
+  `tail` prints those lines, and `cron` mails them to root if the host can send
   mail.
 - A scan still running when the next one starts carries on. The new one fails at
-  once, and the lines cron mails include
+  once, and the lines `cron` mails include
   `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
-- `report`, `trees` and `info` need only read access to the database. With the
+- `report`, `trees` and `info` need only read access to the database. The umask
-  usual umask of `022`, the first scan creates it readable by every user, so an
+  is the setting that removes permissions from the files a process creates.
-  unprivileged user can run them against root's database.
+  Under the usual umask of `022`, the first scan creates the database readable
+  by every user, so an unprivileged user can run them against root's database.
 
 ### Reading the reports
 
@@ -181,7 +185,8 @@ symlink, which `trees` does not see.
 
 Duplicate finders that hash every file in full do not scale to the target
 environment. That is about ten million files and 150 TB. The disks may be slow
-or busy, as a ZFS storage pool is while it rebuilds a disk.
+or busy. A ZFS storage pool, a set of disks managed as one filesystem, is busy
+while it rebuilds a disk.
 
 `sfdupes` reads a file under 100 MiB in full. It reads a larger file only at its
 two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
@@ -217,7 +222,7 @@ In order:
 3. **Scan incrementally, analyze offline.** The scan keeps a persistent
    database, and an unchanged file is never read again. `report`, `trees` and
    `info` work from the database alone and never touch the scanned filesystem.
-   `scan` is built to run from cron; the reports run at any time, against what
+   `scan` is built to run from `cron`; the reports run at any time, against what
    the database holds.
 4. **Keep the streams apart.** Everything on stdout is machine-readable data.
    Progress, warnings, summaries, help and usage text go to stderr.
@@ -248,7 +253,7 @@ The reference below is also the program's specification.
 - No byte-for-byte compare, and no deleting or linking of duplicates. The
   reports are advisory; acting on them is your job.
 - No persistence beyond the SQLite database; no export or import formats.
-- No daemon or filesystem watcher; scheduling rescans is cron's job.
+- No daemon or filesystem watcher; scheduling rescans is `cron`'s job.
 
 ## Reference
 
@@ -266,8 +271,8 @@ The reference below is also the program's specification.
   operand's filesystem, as the same flag does for `du` and `rsync`. By default
   the walk crosses filesystem boundaries.
 - `--no-default-excludes` (`scan`) lets the walk enter the default excludes: the
-  system directories the walk otherwise leaves out, such as `/proc` and `/tmp`
+  system directories the walk otherwise leaves out, such as `/var/tmp` on Linux
-  on Linux (see "Rules for the walk").
+  (see "Rules for the walk").
 - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and exits
   0, writing nothing to stderr.
 - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
@@ -325,16 +330,20 @@ default. An operand may be a directory or a regular file.
 - Each operand is made absolute before the walk. A `.` or `..` in it is resolved
   as text, and symlinks are not followed. Records are keyed by that absolute
   path, so results do not depend on the working directory.
-- All operands belong to one scan, and all are walked at once by the same
+- All operands belong to one scan, and one walk covers them all at once (see
-  workers (see "Concurrency").
+  "Concurrency").
 - An operand that repeats another, or lies under another, is dropped before the
   walk, without a line, so every file is reached once. An operand under another
-  is kept when the outer operand's walk would not reach it. That happens when
+  is kept only when the inner operand, or a directory between the two, is one of
-  the inner operand, or a directory between the two, is one of these:
+  these:
     - a default exclude (see "Rules for the walk");
     - a directory on a filesystem that is not persistent: one the kernel makes
       up or keeps in memory (see "Rules for the walk");
     - with `-x`, a directory on another filesystem than the outer operand.
+
+    The outer operand's walk does not enter such a directory, so the inner
+    operand is walked on its own.
+
 - An operand that is a symlink, socket, FIFO or device node is not scanned, and
   neither is a directory named `.zfs`. `scan` prints a warning naming the path
   and what it is, counts it as skipped, and drops the operand. Another operand
@@ -359,10 +368,10 @@ operands:
   same-size rewrite counts as a change whenever the filesystem gives it a later
   mtime, even within the same second.
 - The scan deletes each record under a scanned operand whose file it could not
-  verify. That covers deleted files, and also paths whose details or contents
+  verify. That covers deleted files, and paths the scan could not stat or read
-  the scan could not read this run. So the database holds only what the latest
+  this run. To stat a file is to read its size, type and times without opening
-  scan over each path could verify. A later successful scan adds such files
+  it. So the database holds only what the latest scan over each path could
-  again.
+  verify. A later successful scan adds such files again.
 - Records outside the scanned operands are left alone, so disjoint trees can be
   scanned on different schedules into one database.
 - The scan keeps every record at or below a path that the walk leaves out on
@@ -384,21 +393,20 @@ phases.
 
 1. `load` reads the existing records. Each record under the operands is kept in
    memory by path, with its size and mtime, for the walk to compare against.
-2. `walk` lists the trees under all operands. Several workers (see
+2. `walk` lists the trees under all operands. Several workers share one queue of
-   "Concurrency") share one queue of directories to read. Each worker reads one
+   directories to read. A worker is one thread of the program, and "Concurrency"
-   directory at a time, hands the subdirectories it finds back to the queue, and
+   says how many there are. Each worker reads one directory at a time, hands the
-   stats each regular file as it finds it. To stat a file is to read its size,
+   subdirectories it finds back to the queue, and stats each regular file as it
-   type and times without opening it. The walk compares each file with its
+   finds it. The walk compares each file with its record. An unchanged file
-   record. An unchanged file needs nothing more. Every new or changed file is
+   needs nothing more. Every new or changed file is passed to the hash phase.
-   passed to the hash phase.
 3. `hash` reads every new or changed file and computes its three hashes. Files
-   are read in inode order. An inode is the filesystem's own record of a file,
+   are read in inode order. An inode is the filesystem's own record of a file.
-   and files with nearby inode numbers usually lie near each other on disk, so
+   Files with nearby inode numbers usually lie near each other on disk, so this
-   this order keeps a spinning disk from seeking back and forth. Paths that are
+   order keeps a spinning disk from seeking back and forth. Paths that are hard
-   hard links to one file are read once and share the result, so a backup tree
+   links to one file are read once and share the result, so a backup tree made
-   made of hard links costs one read per file. Completed records are committed
+   of hard links costs one read per file. Completed records are committed to the
-   to the database 10,000 at a time while hashing runs. A scan interrupted after
+   database 10,000 at a time while hashing runs. A scan interrupted after hours
-   hours therefore keeps everything hashed so far.
+   therefore keeps everything hashed so far.
 4. `update` commits the last partial batch of hashed records, then deletes the
    records the scan did not verify.
 
@@ -414,8 +422,10 @@ walk practical on large or busy storage pools.
   snapshots, and walking it would list every file once per snapshot.
 - The walk never records the scan's own database files. Those are the database,
   its `.lock` file, and the `-wal` and `-shm` files SQLite keeps beside it while
-  the scan runs (see "Concurrent reads"). It knows them by device and inode, so
+  the scan runs (see "Concurrent reads"). It knows them by device ID and inode
-  it passes over them however it reaches them, through a symlinked directory
+  number. The device ID is the number by which the kernel tells filesystems
+  apart, and the inode number tells the files of one filesystem apart. So the
+  walk passes over them however it reaches them, through a symlinked directory
   included. It prints no line for them and does not count them as skipped. A
   record an earlier scan left for one of them is deleted.
 - By default the walk crosses filesystem boundaries. With `-x`, it never enters
@@ -435,8 +445,7 @@ walk practical on large or busy storage pools.
 
     `devfs` is the macOS filesystem of `/dev`; the others are Linux names. As it
     starts, `scan` lists the mounted filesystems and notes the device ID of each
-    one of those types. The device ID is the number by which the kernel tells
+    one of those types. On Linux the list comes from `/proc/self/mountinfo`,
-    filesystems apart. On Linux the list comes from `/proc/self/mountinfo`,
     which gives each filesystem's own device ID. On macOS it comes from the
     system's mount list, and the device ID is read from the mount point. Where
     two filesystems share a mount point on macOS, the one mounted last counts.
@@ -496,14 +505,13 @@ walk practical on large or busy storage pools.
 
 #### Concurrency
 
-The walk phase and the hash phase each use a pool of workers. A worker is one
+The walk phase and the hash phase each use a pool of workers, and each pool has
-thread of the program, and each pool has `--workers` of them. The walk works on
+`--workers` of them. The walk works on several directories at once, and the hash
-several directories at once, and the hash phase on several files at once. On
+phase on several files at once. On spinning disks both phases wait mostly on
-spinning disks both phases wait mostly on seeks, the time the disk head takes to
+seeks, the time the disk head takes to move. A `--workers` value well above the
-move. A `--workers` value well above the CPU count can therefore help on storage
+CPU count can therefore help on storage pools with many disks. The main thread
-pools with many disks. The main thread of the program owns the database writes
+of the program owns the database writes and the progress counts. Each progress
-and the progress counts. Each progress display draws from a thread of its own
+display draws from a thread of its own and never blocks the workers.
-and never blocks the workers.
 
 #### What `scan` prints
 
@@ -512,10 +520,12 @@ and never blocks the workers.
 - one line naming the database (see "Database");
 - for each dropped operand, in operand order, the warning or the line described
   under "Operands". A repeated operand, or one under another, gets neither;
-- the progress display of each phase (see "Progress"), with a warning for each
+- the progress display of each phase (see "Progress"). A warning for each path
-  path it skips and a line for each path the walk leaves out on purpose, as they
+  it skips, and a line for each path the walk leaves out on purpose, come as
-  happen;
+  they happen;
-- the summary line.
+- the summary line;
+- a warning when a report still has the database open as the scan ends (see
+  "Concurrent reads").
 
 The summary line reports the files seen this run, by what happened to each, plus
 the skips:
@@ -528,7 +538,8 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),
 
 #### Stopping a scan
 
-`scan` stops cleanly on `SIGINT` (Ctrl-C) or `SIGTERM`:
+`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the
+signal `kill` sends by default:
 
 - The workers stop taking work. Each finishes at most the directory listing or
   the file it is reading, and the progress display is finished.
@@ -540,20 +551,22 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),
 - The line `scan: interrupted after N files` goes to stderr, `N` being the
   number of files the walk reached, and the exit code is 1.
 
-The next scan skips the records already written and converges as usual.
+The next scan skips the records already written and brings the database up to
+date as usual.
 
 After the first signal `scan` stops catching signals. A second one ends it at
 once, as any uncaught signal does: the records not yet committed are lost, and
 the database stays valid. `scan` catches `SIGTERM` even when it inherits it as
-ignored. A `SIGINT` that `scan` inherits as ignored, as a background job started
+ignored. A background job started from a script inherits `SIGINT` as ignored,
-from a script does, stays ignored.
+and `scan` leaves it ignored.
 
 ### `report` mode
 
-`report` reads every record from the database and takes no positional arguments.
+`report` reads every record from the database. It takes no positional arguments:
-It never touches the scanned filesystem: it does not stat, open or otherwise
+nothing may follow `report` on the command line but flags. It never touches the
-access any path in the records. Its only I/O is the database, stdout and stderr.
+scanned filesystem: it does not stat, open or otherwise access any path in the
-Its output is the same whether or not the scanned filesystem is still mounted.
+records. Its only I/O is the database, stdout and stderr. Its output is the same
+whether or not the scanned filesystem is still mounted.
 
 Processing:
 
@@ -625,8 +638,8 @@ Definitions:
 
 Processing:
 
-- Build the hierarchy, compute every directory's digest, and group the
+- `trees` builds the hierarchy, computes every directory's digest, and groups
-  directories by digest. Every group with two or more directories is a
+  the directories by digest. Every group with two or more directories is a
   duplicate-tree group.
 - Only the outermost duplicate trees are reported. A group is left out when its
   members' parents are distinct directories that all have the same digest: the
@@ -792,17 +805,19 @@ Only one `scan` runs against a database at a time.
   needs the database to itself. A `scan` that starts while a report is still
   reading waits for it up to the busy timeout, then fails. A `scan` that ends
   while a report has the database open prints a warning and leaves the database
-  in WAL mode until the next scan. `report` writes its rows while it reads, so
+  in WAL mode until the next scan. `report` writes its rows while it reads. It
-  it is still reading while a pager such as `less`, or a stalled pipe, holds up
+  is therefore still reading while a pager or a stalled pipe holds up its
-  its output.
+  output. A pager, such as `less`, shows output one screen at a time and waits
+  for a key press.
 
 #### What `scan` writes, and when
 
 - As it starts, before the walk: the directory, an empty database file and the
   `.lock` file, each only if missing.
-- Then, still before the walk: the schema, the table and index described under
+- Then, still before the walk:
-  "Schema", in a new database, and the `-wal` and `-shm` files, which stay
+    - the `-wal` and `-shm` files, which stay beside the database while the scan
-  beside the database while the scan runs.
+      runs;
+    - in a new database, the table and the index described under "Schema".
 - During the `load` and `walk` phases: no records. A first scan's database holds
   no records until the hash phase, however long its walk runs.
 - During the `hash` phase: records, committed 10,000 at a time.
@@ -988,10 +1003,11 @@ A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
 failed write to stdout, such as a full disk, is reported the same way. Two cases
 never reach `sfdupes` as a failed write:
 
-- When the reader of a stdout pipe exits early, as in `sfdupes report | head`,
+- The reader of a stdout pipe may exit early, as `head` does in
-  the next write ends `sfdupes` with `SIGPIPE`, the signal for a write to a
+  `sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
-  closed pipe. It ends quietly and without a summary, as `cat` or `sort` would.
+  the signal for a write to a closed pipe. It ends quietly and without a
-  The shell reports the signal, status 141 in most shells, not exit 1.
+  summary, as `cat` or `sort` would. The shell reports the signal, status 141 in
+  most shells, not exit 1.
 - When stdout is closed outright, as in `sfdupes report >&-`, the Go runtime
   opens `/dev/null` in its place before `sfdupes` starts. The output is
   discarded, and the run succeeds, as with `> /dev/null`.
@@ -1001,23 +1017,29 @@ never reach `sfdupes` as a failed write:
 This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
-workflow, and the `Makefile` targets are thin shims that call them. Every script
+workflow, and each `Makefile` target does nothing but call its script. Every
-is POSIX `sh`, finds the repository root itself, and can be run from any
+script is POSIX `sh`, finds the repository root itself, and can be run from any
 directory.
 
 - `script/bootstrap` installs what building and developing need. It can be run
   again at any time; installed tools are kept. `git`, `make` and `go` are
   checked for presence only. A missing one is installed from the first of `nix`,
-  `apt`, `brew` or `apk` found on the host. An installed `node` is used when its
+  `apt`, `brew` or `apk` found on the host. Then come `node`, the runtime that
-  major version is 22. Otherwise the script downloads `nvm`, checks the
+  `prettier` needs, and `yarn`, the package manager that installs `prettier`:
-  archive's SHA-256, and installs `node` 22.17.0 through it. It then activates
+    - An installed `node` is used when its major version is 22. A `yarn` already
-  `yarn` 1.22.22 under that `node` through `corepack`. With an installed `node`
+      on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
-  22, a `yarn` already on `PATH` is used as it is. Otherwise `yarn` 1.22.22 is
+      activated through `corepack`, the `yarn` installer that ships with `node`,
-  activated through `corepack`, or installed with `npm` when there is no
+      or installed with `npm` when there is no `corepack`.
-  `corepack`. `yarn install --frozen-lockfile` then installs the `prettier` that
+    - Otherwise the script downloads `nvm`, a `node` version manager, and checks
-  `package.json` and `yarn.lock` pin. The linter is never installed; it runs in
+      the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
-  Docker (see `script/lint`). `docker` is not installed either, and testing,
+      activates `yarn` 1.22.22 under that `node` through `corepack`.
-  linting and the image build need it. The script ends with `go mod download`.
+
+    `yarn install --frozen-lockfile` then installs the `prettier` that
+    `package.json` and `yarn.lock` pin. `prettier` is the Markdown formatter.
+    The linter is never installed; it runs in Docker (see `script/lint`).
+    `docker` is not installed either, and testing, linting and the image build
+    need it. The script ends with `go mod download`.
+
 - `script/setup` makes a fresh clone ready for development: `script/bootstrap`,
   then `script/install-precommit`.
 - `script/projectname` prints `sfdupes`. Scripts that need the name call it, so
@@ -1064,7 +1086,7 @@ directory.
   image as `script/docker` does. The Gitea workflow runs it on every push, and a
   successful run means every check passed. The tests and the linter run twice:
   once in `script/check`, and again as stages of the image build.
-- `script/precommit` runs from the git pre-commit hook. `go mod tidy` must
+- `script/precommit` runs from the `git` pre-commit hook. `go mod tidy` must
   change nothing, or the commit fails; then `script/check` runs.
 - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
   `script/precommit`.
@@ -1083,8 +1105,9 @@ unchanged package is not linted again.
 
 ## Build
 
-The `script/` entrypoints above hold the implementations. The `Makefile` targets
+The `script/` entrypoints above hold the implementations. Each `Makefile` target
-are shims onto them, except `build`, which carries the compile recipe:
+does nothing but call its script, except `build` and `clean`, which carry their
+own recipes:
 
 - `make` or `make build` builds the `sfdupes` binary with cgo disabled. Building
   is the default target.

TODO.md

@@ -33,7 +33,8 @@
   it. It leads with what the program is and how to use it. A reference section
   covers the settings, the database, the output formats and the exit codes;
   development and testing come last. Every statement was checked against the
-  code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
+  code on `next`. Three comments in `main_test.go` cite "Settings", where their
+  rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
 
 - a file or directory removed while the scan runs is not an error, in the walk
   or in the hash phase. The scan prints no line for it and does not count it as

main_test.go

@@ -648,7 +648,7 @@ func TestRunUsageErrors(t *testing.T) {
 }
 
 func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
-	// README §scan mode: --workers below 1 is a usage error reported in
+	// README §Settings: --workers below 1 is a usage error reported in
 	// one line on stderr, before the scan opens the database.
 	for _, workers := range []string{"0", "-1"} {
 		t.Run(workers, func(t *testing.T) {
@@ -686,8 +686,8 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
 func TestRunHelp(t *testing.T) {
 	t.Parallel()
 
-	// README §Subcommands: help goes to stderr, exits 0, and leaves
+	// README §Settings: help goes to stderr, exits 0, and leaves stdout
-	// stdout empty.
+	// empty.
 	cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
 
 	for _, args := range cases {
@@ -713,8 +713,8 @@ func TestRunHelp(t *testing.T) {
 func TestRunVersion(t *testing.T) {
 	t.Parallel()
 
-	// README §Subcommands: the version is one line on stdout, with
+	// README §Settings: the version is one line on stdout, with nothing
-	// nothing on stderr, and exits 0.
+	// on stderr, and exits 0.
 	info, _ := debug.ReadBuildInfo()
 
 	for _, arg := range []string{"--version", "-v"} {