Rewrite README.md sentence by sentence for a newcomer

The old README was a specification written as one dense document:
sentences past 30 words, lists chained with commas, exceptions ahead
of the rules, and release history in the user text. This rewrite
leads with what the program is, what it is for and how to use it,
then gives a reference for the settings, each subcommand, the
database, the output formats, the progress display and the exit
codes. Non-goals come before the reference; the constraints, the
entrypoints and the build come last. Every statement was checked
against the code on next and against what the built binary prints on
a small tree. The section names the code comments cite are kept, and
three comments in main_test.go now cite Settings. TODO.md records the
step.

Model: fable-5-1

README.md

@@ -45,10 +45,9 @@ database's path, size and record count.
 The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
 another file. The first `scan` creates it.
 
-`report`, `trees` and `info` write their data to stdout. Everything else, such
+Every command writes its data to stdout. Everything else, such as progress and
-as progress and warnings, goes to stderr. stdout and stderr are a program's two
+warnings, goes to stderr. stdout and stderr are a program's two output streams.
-output streams. The shell shows both on the terminal, and `>` redirects stdout
+The shell shows both on the terminal, and `>` redirects stdout alone.
-alone.
 
 ### Install
 
@@ -70,9 +69,9 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
 `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
 a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
 string for an untagged commit. That string ends in the commit's hash. A binary
-built from a clone carries the version `git describe --tags --always --dirty`
+built from a clone carries the `git` tag or commit it was built from. `VERSION`
-prints in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev`
+is `dev` when no version is known. To see everything Go recorded in an installed
-when no version is known. To see everything Go recorded in an installed binary:
+binary:
 
 ```sh
 go version -m "$(command -v sfdupes)"
@@ -171,9 +170,8 @@ cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
 where the files differ. The second line prints `hard links` when both paths are
 the same file.
 
-A path holding a special character, such as a tab, is escaped in the reports.
+A path holding a special character, such as a tab, is escaped in the reports
-The character is written as a backslash and a letter, `\t` for a tab (see
+(see "Report output format"). Undo the escapes before using such a path.
-"Report output format"). Undo the escapes before using such a path.
 `printf '%b'` does that, because every backslash in an escaped path starts an
 escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
 newline at the end of a path, print an `x` after the path and remove it
@@ -216,8 +214,8 @@ In order:
 
 1. **Find whole duplicate trees, not just files.** The aim is to find places
    where the same set of files and directories exists at two or more paths. A
-   whole subtree can then be removed at once. Tree-level detection is built on
+   whole subtree can then be removed at once. File-level detection comes first.
-   file-level detection.
+   Tree-level detection is built on top of it.
 2. **Bound the reads per file.** A file is read only when it is new or has
    changed. A file under 100 MiB is then read in full, and a larger one only at
    its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
@@ -252,9 +250,7 @@ sfdupes --version
 sfdupes [command] --help
 ```
 
-A missing or unknown subcommand is a usage error, a mistake in the command line.
+A missing or unknown subcommand prints the usage text on stderr and exits 2.
-`sfdupes` then prints the usage text, a summary of the commands and their flags,
-on stderr and exits 2.
 
 The reference below is also the program's specification.
 
@@ -340,8 +336,7 @@ default. An operand may be a directory or a regular file.
 
 - `scan` with no operand is a usage error. The usage text goes to stderr, and
   the exit code is 2.
-- An operand that does not exist is a fatal error. `scan` then stops before it
+- An operand that does not exist is a fatal error, exit 1.
-  walks anything, and exits 1.
 - Each operand is made absolute before the walk. A `.` or `..` in it is resolved
   as text, and symlinks are not followed. Each record is stored under that
   absolute path, so results do not depend on the working directory.
@@ -349,15 +344,15 @@ default. An operand may be a directory or a regular file.
   "Concurrency").
 - An operand that repeats another, or lies under another, is dropped before the
   walk, without a line. Every file is then reached once. An operand under
-  another is kept when the outer operand's walk would not reach it. That is the
+  another is kept only when the inner operand, or a directory between the two,
-  case when the inner operand, or a directory between the two, is one of these:
+  is one of these:
     - a default exclude (see "Rules for the walk");
+    - a directory on a filesystem that is not persistent, one the kernel makes
+      up or keeps in memory (see "Rules for the walk");
     - with `-x`, a directory on another filesystem than the outer operand.
 
-    It is also the case when a directory between the two is on a filesystem that
+    The outer operand's walk does not enter such a directory, so the inner
-    is not persistent. Such a filesystem is one the kernel makes up or keeps in
+    operand is walked on its own.
-    memory (see "Rules for the walk"). The outer operand's walk does not enter
-    such a directory, so the inner operand is walked on its own.
 
 - An operand that is neither a regular file nor a directory is not scanned, and
   neither is a directory named `.zfs`. `scan` prints a warning naming the path
@@ -389,17 +384,16 @@ operands:
   could verify. A later successful scan adds such files again.
 - Records outside the scanned operands are left alone, so separate trees can be
   scanned on different schedules into one database.
-- The scan keeps every record at or below a path that the walk leaves out for
+- The scan keeps every record at or below a path that the walk leaves out on
-  one of these reasons:
+  purpose. Such records count as outside the scanned operands. The walk leaves
-    - it is a default exclude;
+  out:
-    - it is not on a persistent filesystem;
+    - a default exclude;
-    - with `-x`, it is a directory on another filesystem than its operand.
+    - a path that is not on a persistent filesystem;
+    - with `-x`, a directory on another filesystem than its operand.
 
-    Such records count as outside the scanned operands. So with `-x`, a
+    So with `-x`, a filesystem mounted under an operand can be scanned on its
-    filesystem mounted under an operand can be scanned on its own schedule into
+    own schedule into the same database. An operand at or inside such a path is
-    the same database. An operand may lie at or inside a default exclude, or,
+    walked on its own. Its walk verifies the records below it, so they are
-    with `-x`, at or inside a directory on another filesystem. Such an operand
-    is walked on its own. Its walk verifies the records below it, so they are
     deleted when their files are gone.
 
 #### Phases
@@ -475,11 +469,10 @@ walk practical on large or busy storage pools.
     starts, `scan` lists the mounted filesystems and notes the device ID of each
     one of those types. On Linux the list comes from `/proc/self/mountinfo`,
     which gives each filesystem's own device ID. On macOS it comes from the
-    system's mount list. There the device ID is read from the mount point, the
+    system's mount list, and the device ID is read from the mount point. Where
-    directory the filesystem is mounted on. Where two filesystems share a mount
+    two filesystems share a mount point on macOS, the one mounted last counts.
-    point on macOS, the one mounted last counts. On any other system the list is
+    On any other system the list is empty. A list that `scan` cannot read is a
-    empty. A list that `scan` cannot read is a fatal error, and nothing is
+    fatal error, and nothing is scanned. No flag turns this rule off.
-    scanned. No flag turns this rule off.
 
     The walk never enters a directory on one of those devices, and never records
     a file on one. That holds however the walk reaches the path, through a bind
@@ -869,8 +862,7 @@ Only one `scan` runs against a database at a time.
 - Then, before the walk as well, it writes:
     - the `-wal` and `-shm` files, which stay beside the database while the scan
       runs;
-    - in a new database, the table, the index and the schema version described
+    - in a new database, the table and the index described under "Schema".
-      under "Schema".
 - During the `load` and `walk` phases, no records are written. A first scan's
   database holds no records until the hash phase, however long its walk runs.
 - During the `hash` phase, records are written 10,000 at a time.
@@ -911,8 +903,7 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
   count of nanoseconds cannot hold those. They serve change detection only and
   are not part of the signature.
 - `head`, `tail` and `content` each hold the raw 32 bytes of a SHA-256 (see
-  "Duplicate detection"). `report` and `trees` fail on a record whose hash is
+  "Duplicate detection"). A record whose hash is not 32 bytes is an error.
-  not 32 bytes.
 - `files_signature` lets `report` read the records in signature order without
   sorting the table.
 
@@ -955,9 +946,8 @@ Each display is one line holding:
 
 - the phase and its place among the phases;
 - the progress, written `[m/n] x%` for `m` items done of a total `n`, which is
-  `x` percent of it. A phase that has no total shows the items done instead, as
+  `x` percent of it. Before a phase has a total, the line shows the items done
-  `1234 files`. A phase waiting for its total shows what it waits for, as
+  instead, or what the phase is waiting for;
-  `counting the records`, and no rate;
 - the current rate, in the items the phase counts, such as files or records. It
   covers the items completed over the last 5 seconds, or, when stderr is not a
   terminal, since the line before. It falls to 0 while nothing completes. A
@@ -1010,12 +1000,11 @@ On a terminal:
     ```
 
 - No line reaches the terminal's last column, however narrow the terminal is. A
-  line too wide for the terminal first leaves out its rate, the words of a
+  line too wide for the terminal first leaves out its rate, or the words shown
-  stalled phase, or what the phase is waiting for. It leaves out its bar next.
+  in the rate's place. It leaves out its bar next. It is cut short at the edge
-  It is cut short at the edge only when the rest does not fit on its own. The
+  only when the four remaining parts do not fit on their own. Those are:
-  rest is:
     - the phase and its place;
-    - the progress, when the phase shows one;
+    - the progress;
     - the elapsed time;
     - the time remaining.
 
@@ -1048,12 +1037,12 @@ the count it stopped at.
     - an operand that does not exist, or whose details `scan` cannot read;
     - a list of mounted filesystems that `scan` cannot read;
     - another `scan` already running against the same database;
-    - a database that cannot be created or opened, or whose reads or writes
+    - a database that cannot be created, opened, read or written, its directory
-      fail, its directory and its lock file included;
+      and its lock file included;
     - a relative database path when the working directory cannot be read;
     - a database with another schema version, or one at version 0 that already
       has a `files` table;
-    - a record with a hash that is not 32 bytes, read by `report` or `trees`;
+    - a record whose `head`, `tail` or `content` is not 32 bytes;
     - a missing database for `report`, `trees` or `info`;
     - a failed write to stdout.
 - `2`: a usage error, which is one of these:
@@ -1084,9 +1073,8 @@ This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
 workflow, and the `Makefile` targets call them, except `build` and `clean` (see
-"Build"). Every script runs under plain `sh`, the standard Unix shell, and uses
+"Build"). Every script is POSIX `sh`. It finds the repository root itself, so it
-no `bash` feature. It finds the repository root itself, so it can be run from
+can be run from any directory.
-any directory.
 
 - `script/bootstrap` installs what building and developing need. It can be run
   again at any time, and installed tools are kept. `git`, `make` and `go` are
@@ -1098,16 +1086,14 @@ any directory.
     - `apk`.
 
     Then come `node` and `yarn`. `prettier`, the Markdown formatter, is a `node`
-    program. `yarn` is the package manager that installs it:
+    program, and `yarn` is the package manager that installs it:
     - An installed `node` is used when its major version is 22. A `yarn` already
       on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
       activated through `corepack`, the `yarn` installer that ships with `node`.
-      When there is no `corepack` either, `npm`, the package manager that ships
+      When there is no `corepack` either, `npm` installs it.
-      with `node`, installs it.
+    - Otherwise the script downloads `nvm`, a `node` version manager, and checks
-    - Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
+      the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
-      version manager, and activates `yarn` 1.22.22 under that `node` through
+      activates `yarn` 1.22.22 under that `node` through `corepack`.
-      `corepack`. When `nvm` is missing, the script first downloads it and
-      checks the archive's SHA-256.
 
     `yarn install --frozen-lockfile` then installs the `prettier` that
     `package.json` and `yarn.lock` pin. The linter is never installed. It runs
@@ -1123,22 +1109,18 @@ any directory.
   unsynchronized access to memory from several threads. The suite runs with a
   90-second timeout, and its coverage of the code is measured. On failure it
   reruns the suite verbosely, so that the log names the failing test. It builds
-  only the `test` stage of the `Dockerfile` and writes no image. A stage is one
+  only the `test` stage of the `Dockerfile` and writes no image. The race
-  named part of the `Dockerfile`, which starts from an image of its own and can
+  detector needs cgo, Go's bridge to C code, and a C compiler. The stage
-  be built alone. The race detector needs cgo, Go's bridge to C code, and a C
+  therefore starts from a Debian `golang` image, which has `gcc`. The image is
-  compiler. The stage therefore starts from a Debian `golang` image, which has
+  pinned by its digest, its content hash. The tests run as `nobody`, because
-  the C compiler `gcc`. The image is pinned by its digest, its content hash. The
+  several of them make a file unreadable, and root reads it anyway.
-  tests run as the user `nobody`, because several of them make a file
-  unreadable, and root reads it anyway.
 - `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
   for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
   writes no image. The stage starts from the `golangci/golangci-lint` image,
   pinned by its digest, and runs three build steps:
-    - the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
+    - the `gofmt` check;
-      formatter, would change;
+    - `golangci-lint config verify`;
-    - `golangci-lint config verify`, which checks the linter's settings file
+    - `golangci-lint run`.
-      `.golangci.yml`;
-    - `golangci-lint run`, which lints the Go sources.
 
     A successful build is a clean lint. The linter never runs on the host, so a
     working `docker` is the one thing linting needs.
@@ -1158,19 +1140,17 @@ any directory.
     - `script/test` and `script/lint`, which need `docker`;
     - `script/fmt-check`, which needs what `script/bootstrap` installs.
 - `script/docker` builds the Docker image, tagged with the name from
-  `script/projectname`. It passes the output of
+  `script/projectname`. It passes `git describe --tags --always --dirty`, or
-  `git describe --tags --always --dirty`, or `unknown` when that is empty, to
+  `unknown` when that is empty, as the `VERSION` build argument. The build stage
-  the `Dockerfile` as the build argument `VERSION`. A build argument is a value
+  depends on the `lint` and `test` stages, so this also runs the tests and the
-  that a `docker build` hands to the `Dockerfile`. The build stage depends on
+  linter.
-  the `lint` and `test` stages, so this also runs the tests and the linter.
 - `script/cibuild` runs `script/bootstrap` and then `script/check`. Then it
-  builds the image as `script/docker` does. The Gitea workflow in
+  builds the image as `script/docker` does. The repository's CI workflow on
-  `.gitea/workflows/check.yml` runs it on every push. A successful run means
+  Gitea runs it on every push, and a successful run means every check passed.
-  every check passed. The tests and the linter run twice, once in `script/check`
+  The tests and the linter run twice, once in `script/check` and again as stages
-  and again as stages of the image build.
+  of the image build.
-- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
+- `script/precommit` runs from the `git` pre-commit hook. It fails the commit
-  runs before each commit. It fails the commit when `go mod tidy` would change
+  when `go mod tidy` would change anything. Then it runs `script/check`.
-  anything. Then it runs `script/check`.
 - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
   `script/precommit`.
 
@@ -1214,17 +1194,13 @@ commands of their own:
 
 - The language is Go. The module is `sneak.berlin/go/sfdupes`, and the binary is
   `sfdupes`.
-- The code imports these modules beyond the standard library:
+- The dependencies beyond the standard library are:
     - `github.com/spf13/cobra`, for the command line;
     - `github.com/schollz/progressbar/v3`, for the progress display;
     - `golang.org/x/term`, to tell whether stderr is a terminal;
     - `modernc.org/sqlite`, the SQLite driver;
     - `golang.org/x/sys`, for the scan lock and the list of mounted filesystems.
-
+- `modernc.org/sqlite` is pure Go, so the binary builds without cgo. The
-    No other dependency outside the standard library is added.
-
-- The code stays pure Go, with no C code. Cross-compilation is not a concern.
-  `modernc.org/sqlite` is pure Go too, so the binary builds without cgo. The
   `Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
   because the race detector needs cgo (see `script/test`).
 - `report` and `trees` give the same output for the same database contents,
@@ -1236,14 +1212,12 @@ All of the following, run in this directory, must pass:
 
 1. `make check` passes.
 2. `make docker` succeeds.
-3. The test below behaves as described. It builds a throwaway tree in a new
+3. The test below behaves as described. It builds a throwaway tree in a
-   directory under `/var/tmp`, which is on a persistent filesystem. On many
+   temporary directory. Never test against real data.
-   systems `/tmp` is a `tmpfs`, which `scan` leaves out. Never test against real
-   data.
 
     ```sh
-    d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
+    d=$(mktemp -d)
-    export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
+    export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
     mkdir -p "$d/a" "$d/b"
     head -c 2000 /dev/urandom > "$d/a/one.bin"
     cp "$d/a/one.bin" "$d/b/copy.bin"

TODO.md

@@ -32,7 +32,7 @@
 - `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
   it. It leads with what the program is and how to use it. A reference section
   covers every setting and every subcommand. Development and testing come last.
-  Every statement was checked against the code on `next`. Five comments in
+  Every statement was checked against the code on `next`. Three comments in
   `main_test.go` cite "Settings", where their rules now live (2026-10-26,
   https://git.eeqj.de/sneak/sfdupes/issues/135)
 

main_test.go

@@ -393,7 +393,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) {
 }
 
 func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
-	// README §Settings: the path names the file exactly. With link a
+	// README §Database: the path names the file exactly. With link a
 	// symlink to the directory real/sub, link/../db.sqlite names
 	// real/db.sqlite, not db.sqlite beside link. Every subcommand must
 	// use that file, relative or absolute, and its first line must name
@@ -1122,7 +1122,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
 }
 
 func TestRunDatabasePathUsedAsGiven(t *testing.T) {
-	// README §Settings: the path names the database file exactly. In
+	// README §Database: the path names the database file exactly. In
 	// SQLite's connection string an unescaped ? or # would end the file
 	// name and % would start an escape, and a path starting with //
 	// could be read as a host name. %25 is a valid escape, so unescaped