Rewrite README.md sentence by sentence for a newcomer

The old README was a specification written as one dense document:
sentences past 30 words, lists chained with commas, exceptions ahead
of the rules, and release history in the user text. This rewrite
leads with what the program is, what it is for and how to use it,
then gives a reference for the settings, each subcommand, the
database, the output formats, the progress display and the exit
codes. Non-goals come before the reference; the constraints, the
entrypoints and the build come last. Every statement was checked
against the code on next and against what the built binary prints on
a small tree. The section names the code comments cite are kept, and
five comments in main_test.go now cite Settings. TODO.md records the
step.

Model: fable-5-1

README.md

@@ -45,9 +45,10 @@ database's path, size and record count.
 The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
 another file. The first `scan` creates it.
 
-Every command writes its data to stdout. Everything else, such as progress and
+`report`, `trees` and `info` write their data to stdout. Everything else, such
-warnings, goes to stderr. stdout and stderr are a program's two output streams.
+as progress and warnings, goes to stderr. stdout and stderr are a program's two
-The shell shows both on the terminal, and `>` redirects stdout alone.
+output streams. The shell shows both on the terminal, and `>` redirects stdout
+alone.
 
 ### Install
 
@@ -69,9 +70,9 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
 `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
 a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
 string for an untagged commit. That string ends in the commit's hash. A binary
-built from a clone carries the `git` tag or commit it was built from. `VERSION`
+built from a clone carries the version `git describe --tags --always --dirty`
-is `dev` when no version is known. To see everything Go recorded in an installed
+prints in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev`
-binary:
+when no version is known. To see everything Go recorded in an installed binary:
 
 ```sh
 go version -m "$(command -v sfdupes)"
@@ -170,8 +171,9 @@ cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
 where the files differ. The second line prints `hard links` when both paths are
 the same file.
 
-A path holding a special character, such as a tab, is escaped in the reports
+A path holding a special character, such as a tab, is escaped in the reports.
-(see "Report output format"). Undo the escapes before using such a path.
+The character is written as a backslash and a letter, `\t` for a tab (see
+"Report output format"). Undo the escapes before using such a path.
 `printf '%b'` does that, because every backslash in an escaped path starts an
 escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
 newline at the end of a path, print an `x` after the path and remove it
@@ -214,8 +216,8 @@ In order:
 
 1. **Find whole duplicate trees, not just files.** The aim is to find places
    where the same set of files and directories exists at two or more paths. A
-   whole subtree can then be removed at once. File-level detection comes first.
+   whole subtree can then be removed at once. Tree-level detection is built on
-   Tree-level detection is built on top of it.
+   file-level detection.
 2. **Bound the reads per file.** A file is read only when it is new or has
    changed. A file under 100 MiB is then read in full, and a larger one only at
    its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
@@ -250,7 +252,9 @@ sfdupes --version
 sfdupes [command] --help
 ```
 
-A missing or unknown subcommand prints the usage text on stderr and exits 2.
+A missing or unknown subcommand is a usage error, a mistake in the command line.
+`sfdupes` then prints the usage text, a summary of the commands and their flags,
+on stderr and exits 2.
 
 The reference below is also the program's specification.
 
@@ -336,7 +340,8 @@ default. An operand may be a directory or a regular file.
 
 - `scan` with no operand is a usage error. The usage text goes to stderr, and
   the exit code is 2.
-- An operand that does not exist is a fatal error, exit 1.
+- An operand that does not exist is a fatal error. `scan` then stops before it
+  walks anything, and exits 1.
 - Each operand is made absolute before the walk. A `.` or `..` in it is resolved
   as text, and symlinks are not followed. Each record is stored under that
   absolute path, so results do not depend on the working directory.
@@ -344,15 +349,15 @@ default. An operand may be a directory or a regular file.
   "Concurrency").
 - An operand that repeats another, or lies under another, is dropped before the
   walk, without a line. Every file is then reached once. An operand under
-  another is kept only when the inner operand, or a directory between the two,
+  another is kept when the outer operand's walk would not reach it. That is the
-  is one of these:
+  case when the inner operand, or a directory between the two, is one of these:
     - a default exclude (see "Rules for the walk");
-    - a directory on a filesystem that is not persistent, one the kernel makes
-      up or keeps in memory (see "Rules for the walk");
     - with `-x`, a directory on another filesystem than the outer operand.
 
-    The outer operand's walk does not enter such a directory, so the inner
+    It is also the case when a directory between the two is on a filesystem that
-    operand is walked on its own.
+    is not persistent. Such a filesystem is one the kernel makes up or keeps in
+    memory (see "Rules for the walk"). The outer operand's walk does not enter
+    such a directory, so the inner operand is walked on its own.
 
 - An operand that is neither a regular file nor a directory is not scanned, and
   neither is a directory named `.zfs`. `scan` prints a warning naming the path
@@ -384,16 +389,17 @@ operands:
   could verify. A later successful scan adds such files again.
 - Records outside the scanned operands are left alone, so separate trees can be
   scanned on different schedules into one database.
-- The scan keeps every record at or below a path that the walk leaves out on
+- The scan keeps every record at or below a path that the walk leaves out for
-  purpose. Such records count as outside the scanned operands. The walk leaves
+  one of these reasons:
-  out:
+    - it is a default exclude;
-    - a default exclude;
+    - it is not on a persistent filesystem;
-    - a path that is not on a persistent filesystem;
+    - with `-x`, it is a directory on another filesystem than its operand.
-    - with `-x`, a directory on another filesystem than its operand.
 
-    So with `-x`, a filesystem mounted under an operand can be scanned on its
+    Such records count as outside the scanned operands. So with `-x`, a
-    own schedule into the same database. An operand at or inside such a path is
+    filesystem mounted under an operand can be scanned on its own schedule into
-    walked on its own. Its walk verifies the records below it, so they are
+    the same database. An operand may lie at or inside a default exclude, or,
+    with `-x`, at or inside a directory on another filesystem. Such an operand
+    is walked on its own. Its walk verifies the records below it, so they are
     deleted when their files are gone.
 
 #### Phases
@@ -469,10 +475,11 @@ walk practical on large or busy storage pools.
     starts, `scan` lists the mounted filesystems and notes the device ID of each
     one of those types. On Linux the list comes from `/proc/self/mountinfo`,
     which gives each filesystem's own device ID. On macOS it comes from the
-    system's mount list, and the device ID is read from the mount point. Where
+    system's mount list. There the device ID is read from the mount point, the
-    two filesystems share a mount point on macOS, the one mounted last counts.
+    directory the filesystem is mounted on. Where two filesystems share a mount
-    On any other system the list is empty. A list that `scan` cannot read is a
+    point on macOS, the one mounted last counts. On any other system the list is
-    fatal error, and nothing is scanned. No flag turns this rule off.
+    empty. A list that `scan` cannot read is a fatal error, and nothing is
+    scanned. No flag turns this rule off.
 
     The walk never enters a directory on one of those devices, and never records
     a file on one. That holds however the walk reaches the path, through a bind
@@ -862,7 +869,8 @@ Only one `scan` runs against a database at a time.
 - Then, before the walk as well, it writes:
     - the `-wal` and `-shm` files, which stay beside the database while the scan
       runs;
-    - in a new database, the table and the index described under "Schema".
+    - in a new database, the table, the index and the schema version described
+      under "Schema".
 - During the `load` and `walk` phases, no records are written. A first scan's
   database holds no records until the hash phase, however long its walk runs.
 - During the `hash` phase, records are written 10,000 at a time.
@@ -903,7 +911,8 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
   count of nanoseconds cannot hold those. They serve change detection only and
   are not part of the signature.
 - `head`, `tail` and `content` each hold the raw 32 bytes of a SHA-256 (see
-  "Duplicate detection"). A record whose hash is not 32 bytes is an error.
+  "Duplicate detection"). `report` and `trees` fail on a record whose hash is
+  not 32 bytes.
 - `files_signature` lets `report` read the records in signature order without
   sorting the table.
 
@@ -946,8 +955,9 @@ Each display is one line holding:
 
 - the phase and its place among the phases;
 - the progress, written `[m/n] x%` for `m` items done of a total `n`, which is
-  `x` percent of it. Before a phase has a total, the line shows the items done
+  `x` percent of it. A phase that has no total shows the items done instead, as
-  instead, or what the phase is waiting for;
+  `1234 files`. A phase waiting for its total shows what it waits for, as
+  `counting the records`, and no rate;
 - the current rate, in the items the phase counts, such as files or records. It
   covers the items completed over the last 5 seconds, or, when stderr is not a
   terminal, since the line before. It falls to 0 while nothing completes. A
@@ -1000,11 +1010,12 @@ On a terminal:
     ```
 
 - No line reaches the terminal's last column, however narrow the terminal is. A
-  line too wide for the terminal first leaves out its rate, or the words shown
+  line too wide for the terminal first leaves out its rate, the words of a
-  in the rate's place. It leaves out its bar next. It is cut short at the edge
+  stalled phase, or what the phase is waiting for. It leaves out its bar next.
-  only when the four remaining parts do not fit on their own. Those are:
+  It is cut short at the edge only when the rest does not fit on its own. The
+  rest is:
     - the phase and its place;
-    - the progress;
+    - the progress, when the phase shows one;
     - the elapsed time;
     - the time remaining.
 
@@ -1037,12 +1048,12 @@ the count it stopped at.
     - an operand that does not exist, or whose details `scan` cannot read;
     - a list of mounted filesystems that `scan` cannot read;
     - another `scan` already running against the same database;
-    - a database that cannot be created, opened, read or written, its directory
+    - a database that cannot be created or opened, or whose reads or writes
-      and its lock file included;
+      fail, its directory and its lock file included;
     - a relative database path when the working directory cannot be read;
     - a database with another schema version, or one at version 0 that already
       has a `files` table;
-    - a record whose `head`, `tail` or `content` is not 32 bytes;
+    - a record with a hash that is not 32 bytes, read by `report` or `trees`;
     - a missing database for `report`, `trees` or `info`;
     - a failed write to stdout.
 - `2`: a usage error, which is one of these:
@@ -1073,8 +1084,9 @@ This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
 workflow, and the `Makefile` targets call them, except `build` and `clean` (see
-"Build"). Every script is POSIX `sh`. It finds the repository root itself, so it
+"Build"). Every script runs under plain `sh`, the standard Unix shell, and uses
-can be run from any directory.
+no `bash` feature. It finds the repository root itself, so it can be run from
+any directory.
 
 - `script/bootstrap` installs what building and developing need. It can be run
   again at any time, and installed tools are kept. `git`, `make` and `go` are
@@ -1086,14 +1098,16 @@ can be run from any directory.
     - `apk`.
 
     Then come `node` and `yarn`. `prettier`, the Markdown formatter, is a `node`
-    program, and `yarn` is the package manager that installs it:
+    program. `yarn` is the package manager that installs it:
     - An installed `node` is used when its major version is 22. A `yarn` already
       on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
       activated through `corepack`, the `yarn` installer that ships with `node`.
-      When there is no `corepack` either, `npm` installs it.
+      When there is no `corepack` either, `npm`, the package manager that ships
-    - Otherwise the script downloads `nvm`, a `node` version manager, and checks
+      with `node`, installs it.
-      the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
+    - Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
-      activates `yarn` 1.22.22 under that `node` through `corepack`.
+      version manager, and activates `yarn` 1.22.22 under that `node` through
+      `corepack`. When `nvm` is missing, the script first downloads it and
+      checks the archive's SHA-256.
 
     `yarn install --frozen-lockfile` then installs the `prettier` that
     `package.json` and `yarn.lock` pin. The linter is never installed. It runs
@@ -1109,18 +1123,22 @@ can be run from any directory.
   unsynchronized access to memory from several threads. The suite runs with a
   90-second timeout, and its coverage of the code is measured. On failure it
   reruns the suite verbosely, so that the log names the failing test. It builds
-  only the `test` stage of the `Dockerfile` and writes no image. The race
+  only the `test` stage of the `Dockerfile` and writes no image. A stage is one
-  detector needs cgo, Go's bridge to C code, and a C compiler. The stage
+  named part of the `Dockerfile`, which starts from an image of its own and can
-  therefore starts from a Debian `golang` image, which has `gcc`. The image is
+  be built alone. The race detector needs cgo, Go's bridge to C code, and a C
-  pinned by its digest, its content hash. The tests run as `nobody`, because
+  compiler. The stage therefore starts from a Debian `golang` image, which has
-  several of them make a file unreadable, and root reads it anyway.
+  the C compiler `gcc`. The image is pinned by its digest, its content hash. The
+  tests run as the user `nobody`, because several of them make a file
+  unreadable, and root reads it anyway.
 - `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
   for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
   writes no image. The stage starts from the `golangci/golangci-lint` image,
   pinned by its digest, and runs three build steps:
-    - the `gofmt` check;
+    - the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
-    - `golangci-lint config verify`;
+      formatter, would change;
-    - `golangci-lint run`.
+    - `golangci-lint config verify`, which checks the linter's settings file
+      `.golangci.yml`;
+    - `golangci-lint run`, which lints the Go sources.
 
     A successful build is a clean lint. The linter never runs on the host, so a
     working `docker` is the one thing linting needs.
@@ -1140,17 +1158,19 @@ can be run from any directory.
     - `script/test` and `script/lint`, which need `docker`;
     - `script/fmt-check`, which needs what `script/bootstrap` installs.
 - `script/docker` builds the Docker image, tagged with the name from
-  `script/projectname`. It passes `git describe --tags --always --dirty`, or
+  `script/projectname`. It passes the output of
-  `unknown` when that is empty, as the `VERSION` build argument. The build stage
+  `git describe --tags --always --dirty`, or `unknown` when that is empty, to
-  depends on the `lint` and `test` stages, so this also runs the tests and the
+  the `Dockerfile` as the build argument `VERSION`. A build argument is a value
-  linter.
+  that a `docker build` hands to the `Dockerfile`. The build stage depends on
+  the `lint` and `test` stages, so this also runs the tests and the linter.
 - `script/cibuild` runs `script/bootstrap` and then `script/check`. Then it
-  builds the image as `script/docker` does. The repository's CI workflow on
+  builds the image as `script/docker` does. The Gitea workflow in
-  Gitea runs it on every push, and a successful run means every check passed.
+  `.gitea/workflows/check.yml` runs it on every push. A successful run means
-  The tests and the linter run twice, once in `script/check` and again as stages
+  every check passed. The tests and the linter run twice, once in `script/check`
-  of the image build.
+  and again as stages of the image build.
-- `script/precommit` runs from the `git` pre-commit hook. It fails the commit
+- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
-  when `go mod tidy` would change anything. Then it runs `script/check`.
+  runs before each commit. It fails the commit when `go mod tidy` would change
+  anything. Then it runs `script/check`.
 - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
   `script/precommit`.
 
@@ -1194,13 +1214,17 @@ commands of their own:
 
 - The language is Go. The module is `sneak.berlin/go/sfdupes`, and the binary is
   `sfdupes`.
-- The dependencies beyond the standard library are:
+- The code imports these modules beyond the standard library:
     - `github.com/spf13/cobra`, for the command line;
     - `github.com/schollz/progressbar/v3`, for the progress display;
     - `golang.org/x/term`, to tell whether stderr is a terminal;
     - `modernc.org/sqlite`, the SQLite driver;
     - `golang.org/x/sys`, for the scan lock and the list of mounted filesystems.
-- `modernc.org/sqlite` is pure Go, so the binary builds without cgo. The
+
+    No other dependency outside the standard library is added.
+
+- The code stays pure Go, with no C code. Cross-compilation is not a concern.
+  `modernc.org/sqlite` is pure Go too, so the binary builds without cgo. The
   `Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
   because the race detector needs cgo (see `script/test`).
 - `report` and `trees` give the same output for the same database contents,
@@ -1212,12 +1236,14 @@ All of the following, run in this directory, must pass:
 
 1. `make check` passes.
 2. `make docker` succeeds.
-3. The test below behaves as described. It builds a throwaway tree in a
+3. The test below behaves as described. It builds a throwaway tree in a new
-   temporary directory. Never test against real data.
+   directory under `/var/tmp`, which is on a persistent filesystem. On many
+   systems `/tmp` is a `tmpfs`, which `scan` leaves out. Never test against real
+   data.
 
     ```sh
-    d=$(mktemp -d)
+    d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
-    export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
+    export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
     mkdir -p "$d/a" "$d/b"
     head -c 2000 /dev/urandom > "$d/a/one.bin"
     cp "$d/a/one.bin" "$d/b/copy.bin"

TODO.md

@@ -32,7 +32,7 @@
 - `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
   it. It leads with what the program is and how to use it. A reference section
   covers every setting and every subcommand. Development and testing come last.
-  Every statement was checked against the code on `next`. Three comments in
+  Every statement was checked against the code on `next`. Five comments in
   `main_test.go` cite "Settings", where their rules now live (2026-10-26,
   https://git.eeqj.de/sneak/sfdupes/issues/135)
 

main_test.go

@@ -393,7 +393,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) {
 }
 
 func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
-	// README §Database: the path names the file exactly. With link a
+	// README §Settings: the path names the file exactly. With link a
 	// symlink to the directory real/sub, link/../db.sqlite names
 	// real/db.sqlite, not db.sqlite beside link. Every subcommand must
 	// use that file, relative or absolute, and its first line must name
@@ -1122,7 +1122,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
 }
 
 func TestRunDatabasePathUsedAsGiven(t *testing.T) {
-	// README §Database: the path names the database file exactly. In
+	// README §Settings: the path names the database file exactly. In
 	// SQLite's connection string an unescaped ? or # would end the file
 	// name and % would start an escape, and a path starting with //
 	// could be read as a host name. %25 is a valid escape, so unescaped