Rewrite README.md sentence by sentence for a newcomer

The old README was a specification written as one dense document: sentences past 30 words, lists chained with commas, exceptions ahead of the rules, and release history in the user text. This rewrite leads with what the program is, what it is for and how to use it, then gives a reference for the settings, each subcommand, the database, the output formats, the progress display and the exit codes. Non-goals come before the reference; the constraints, the entrypoints and the build come last. Every statement was checked against the code on next and against what the built binary prints on a small tree. The section names the code comments cite are kept, and three comments in main_test.go now cite Settings. TODO.md records the step. Model: fable-5-1
2026-10-26 08:15:26 +00:00
3 changed files with 125 additions and 101 deletions
@@ -20,7 +20,7 @@ proof.
 file takes too long. It has four subcommands. `scan` walks the filesystem and
 keeps its results between runs in an SQLite database, a database held in one
 file. A rescan reads only new and changed files, so `scan` can run every day
-from cron, the system's job scheduler. `report`, `trees` and `info` read the
+from `cron`, the system's job scheduler. `report`, `trees` and `info` read the
 database alone and can run at any time.

 ## Getting Started
@@ -67,9 +67,9 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
 `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
 a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
 string for an untagged commit, which ends in the commit's hash. A binary built
-from a clone carries the git tag or commit it was built from. `VERSION` is `dev`
-only when the binary carries no version at all. To see everything Go recorded in
-an installed binary:
+from a clone carries the `git` tag or commit it was built from. `VERSION` is
+`dev` when no version is known. To see everything Go recorded in an installed
+binary:

 ```sh
 go version -m "$(command -v sfdupes)"
@@ -83,14 +83,15 @@ cd sfdupes
 make build
 ```

-Copy the binary to `/usr/local/bin` for the cron job below.
+Copy the binary to `/usr/local/bin` for the `cron` job below.

 `make docker` builds a Docker image tagged `sfdupes`. The build runs the tests
 and the linter first (see "Build"). Inside the image `sfdupes` runs as root,
-with the database at its default path. Bind-mount `/var/lib/sfdupes` to keep the
-database between runs. Mount the tree to scan at the same path inside the
-container as on the host; read-only is enough. The database then records the
-host's paths, and so do the reports.
+with the database at its default path. Each `-v` flag below makes a bind mount,
+which shows a host directory at a path inside the container. Bind-mount
+`/var/lib/sfdupes` to keep the database between runs. Mount the tree to scan at
+the same path inside the container as on the host; read-only is enough. The
+database then records the host's paths, and so do the reports.

 ```sh
 make docker
@@ -99,7 +100,7 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
 docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
 ```

-### Daily scan from cron
+### Daily scan from `cron`

 Run `scan` as root, so that it can read every file. A path it cannot read is
 skipped with a warning and loses its database record. As the file
@@ -120,18 +121,21 @@ skipped with a warning and loses its database record. As the file
    - a warning for each path it skips;
    - a line for each path the walk leaves out on purpose (see "Rules for the
      walk");
-    - the summary line.
+    - the summary line;
+    - a warning when a report still has the database open as the scan ends (see
+      "Concurrent reads").
 - The log grows with every scan; rotate it like any other log.
- Skipped paths do not fail a scan. It still exits 0, and cron sends no mail.
+- Skipped paths do not fail a scan. It still exits 0, and `cron` sends no mail.
 - A scan that fails exits 1, with the reason among the last lines of the log.
-  `tail` prints those lines, and cron mails them to root if the host can send
+  `tail` prints those lines, and `cron` mails them to root if the host can send
  mail.
 - A scan still running when the next one starts carries on. The new one fails at
-  once, and the lines cron mails include
+  once, and the lines `cron` mails include
  `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
- `report`, `trees` and `info` need only read access to the database. With the
-  usual umask of `022`, the first scan creates it readable by every user, so an
-  unprivileged user can run them against root's database.
+- `report`, `trees` and `info` need only read access to the database. The umask
+  is the setting that removes permissions from the files a process creates.
+  Under the usual umask of `022`, the first scan creates the database readable
+  by every user, so an unprivileged user can run them against root's database.

 ### Reading the reports

@@ -181,7 +185,8 @@ symlink, which `trees` does not see.

 Duplicate finders that hash every file in full do not scale to the target
 environment. That is about ten million files and 150 TB. The disks may be slow
-or busy, as a ZFS storage pool is while it rebuilds a disk.
+or busy. A ZFS storage pool, a set of disks managed as one filesystem, is busy
+while it rebuilds a disk.

 `sfdupes` reads a file under 100 MiB in full. It reads a larger file only at its
 two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
@@ -217,7 +222,7 @@ In order:
 3. **Scan incrementally, analyze offline.** The scan keeps a persistent
   database, and an unchanged file is never read again. `report`, `trees` and
   `info` work from the database alone and never touch the scanned filesystem.
-   `scan` is built to run from cron; the reports run at any time, against what
+   `scan` is built to run from `cron`; the reports run at any time, against what
   the database holds.
 4. **Keep the streams apart.** Everything on stdout is machine-readable data.
   Progress, warnings, summaries, help and usage text go to stderr.
@@ -248,7 +253,7 @@ The reference below is also the program's specification.
 - No byte-for-byte compare, and no deleting or linking of duplicates. The
  reports are advisory; acting on them is your job.
 - No persistence beyond the SQLite database; no export or import formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job.
+- No daemon or filesystem watcher; scheduling rescans is `cron`'s job.

 ## Reference

@@ -266,8 +271,8 @@ The reference below is also the program's specification.
  operand's filesystem, as the same flag does for `du` and `rsync`. By default
  the walk crosses filesystem boundaries.
 - `--no-default-excludes` (`scan`) lets the walk enter the default excludes: the
-  system directories the walk otherwise leaves out, such as `/proc` and `/tmp`
-  on Linux (see "Rules for the walk").
+  system directories the walk otherwise leaves out, such as `/var/tmp` on Linux
+  (see "Rules for the walk").
 - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and exits
  0, writing nothing to stderr.
 - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
@@ -325,16 +330,20 @@ default. An operand may be a directory or a regular file.
 - Each operand is made absolute before the walk. A `.` or `..` in it is resolved
  as text, and symlinks are not followed. Records are keyed by that absolute
  path, so results do not depend on the working directory.
- All operands belong to one scan, and all are walked at once by the same
-  workers (see "Concurrency").
+- All operands belong to one scan, and one walk covers them all at once (see
+  "Concurrency").
 - An operand that repeats another, or lies under another, is dropped before the
  walk, without a line, so every file is reached once. An operand under another
-  is kept when the outer operand's walk would not reach it. That happens when
-  the inner operand, or a directory between the two, is one of these:
+  is kept only when the inner operand, or a directory between the two, is one of
+  these:
    - a default exclude (see "Rules for the walk");
    - a directory on a filesystem that is not persistent: one the kernel makes
      up or keeps in memory (see "Rules for the walk");
    - with `-x`, a directory on another filesystem than the outer operand.
+
+    The outer operand's walk does not enter such a directory, so the inner
+    operand is walked on its own.
+
 - An operand that is a symlink, socket, FIFO or device node is not scanned, and
  neither is a directory named `.zfs`. `scan` prints a warning naming the path
  and what it is, counts it as skipped, and drops the operand. Another operand
@@ -359,10 +368,10 @@ operands:
  same-size rewrite counts as a change whenever the filesystem gives it a later
  mtime, even within the same second.
 - The scan deletes each record under a scanned operand whose file it could not
-  verify. That covers deleted files, and also paths whose details or contents
-  the scan could not read this run. So the database holds only what the latest
-  scan over each path could verify. A later successful scan adds such files
-  again.
+  verify. That covers deleted files, and paths the scan could not stat or read
+  this run. To stat a file is to read its size, type and times without opening
+  it. So the database holds only what the latest scan over each path could
+  verify. A later successful scan adds such files again.
 - Records outside the scanned operands are left alone, so disjoint trees can be
  scanned on different schedules into one database.
 - The scan keeps every record at or below a path that the walk leaves out on
@@ -384,21 +393,20 @@ phases.

 1. `load` reads the existing records. Each record under the operands is kept in
   memory by path, with its size and mtime, for the walk to compare against.
-2. `walk` lists the trees under all operands. Several workers (see
-   "Concurrency") share one queue of directories to read. Each worker reads one
-   directory at a time, hands the subdirectories it finds back to the queue, and
-   stats each regular file as it finds it. To stat a file is to read its size,
-   type and times without opening it. The walk compares each file with its
-   record. An unchanged file needs nothing more. Every new or changed file is
-   passed to the hash phase.
+2. `walk` lists the trees under all operands. Several workers share one queue of
+   directories to read. A worker is one thread of the program, and "Concurrency"
+   says how many there are. Each worker reads one directory at a time, hands the
+   subdirectories it finds back to the queue, and stats each regular file as it
+   finds it. The walk compares each file with its record. An unchanged file
+   needs nothing more. Every new or changed file is passed to the hash phase.
 3. `hash` reads every new or changed file and computes its three hashes. Files
-   are read in inode order. An inode is the filesystem's own record of a file,
-   and files with nearby inode numbers usually lie near each other on disk, so
-   this order keeps a spinning disk from seeking back and forth. Paths that are
-   hard links to one file are read once and share the result, so a backup tree
-   made of hard links costs one read per file. Completed records are committed
-   to the database 10,000 at a time while hashing runs. A scan interrupted after
-   hours therefore keeps everything hashed so far.
+   are read in inode order. An inode is the filesystem's own record of a file.
+   Files with nearby inode numbers usually lie near each other on disk, so this
+   order keeps a spinning disk from seeking back and forth. Paths that are hard
+   links to one file are read once and share the result, so a backup tree made
+   of hard links costs one read per file. Completed records are committed to the
+   database 10,000 at a time while hashing runs. A scan interrupted after hours
+   therefore keeps everything hashed so far.
 4. `update` commits the last partial batch of hashed records, then deletes the
   records the scan did not verify.

@@ -414,8 +422,10 @@ walk practical on large or busy storage pools.
  snapshots, and walking it would list every file once per snapshot.
 - The walk never records the scan's own database files. Those are the database,
  its `.lock` file, and the `-wal` and `-shm` files SQLite keeps beside it while
-  the scan runs (see "Concurrent reads"). It knows them by device and inode, so
-  it passes over them however it reaches them, through a symlinked directory
+  the scan runs (see "Concurrent reads"). It knows them by device ID and inode
+  number. The device ID is the number by which the kernel tells filesystems
+  apart, and the inode number tells the files of one filesystem apart. So the
+  walk passes over them however it reaches them, through a symlinked directory
  included. It prints no line for them and does not count them as skipped. A
  record an earlier scan left for one of them is deleted.
 - By default the walk crosses filesystem boundaries. With `-x`, it never enters
@@ -435,8 +445,7 @@ walk practical on large or busy storage pools.

    `devfs` is the macOS filesystem of `/dev`; the others are Linux names. As it
    starts, `scan` lists the mounted filesystems and notes the device ID of each
-    one of those types. The device ID is the number by which the kernel tells
-    filesystems apart. On Linux the list comes from `/proc/self/mountinfo`,
+    one of those types. On Linux the list comes from `/proc/self/mountinfo`,
    which gives each filesystem's own device ID. On macOS it comes from the
    system's mount list, and the device ID is read from the mount point. Where
    two filesystems share a mount point on macOS, the one mounted last counts.
@@ -496,14 +505,13 @@ walk practical on large or busy storage pools.

 #### Concurrency

-The walk phase and the hash phase each use a pool of workers. A worker is one
-thread of the program, and each pool has `--workers` of them. The walk works on
-several directories at once, and the hash phase on several files at once. On
-spinning disks both phases wait mostly on seeks, the time the disk head takes to
-move. A `--workers` value well above the CPU count can therefore help on storage
-pools with many disks. The main thread of the program owns the database writes
-and the progress counts. Each progress display draws from a thread of its own
-and never blocks the workers.
+The walk phase and the hash phase each use a pool of workers, and each pool has
+`--workers` of them. The walk works on several directories at once, and the hash
+phase on several files at once. On spinning disks both phases wait mostly on
+seeks, the time the disk head takes to move. A `--workers` value well above the
+CPU count can therefore help on storage pools with many disks. The main thread
+of the program owns the database writes and the progress counts. Each progress
+display draws from a thread of its own and never blocks the workers.

 #### What `scan` prints

@@ -512,10 +520,12 @@ and never blocks the workers.
 - one line naming the database (see "Database");
 - for each dropped operand, in operand order, the warning or the line described
  under "Operands". A repeated operand, or one under another, gets neither;
- the progress display of each phase (see "Progress"), with a warning for each
-  path it skips and a line for each path the walk leaves out on purpose, as they
-  happen;
- the summary line.
+- the progress display of each phase (see "Progress"). A warning for each path
+  it skips, and a line for each path the walk leaves out on purpose, come as
+  they happen;
+- the summary line;
+- a warning when a report still has the database open as the scan ends (see
+  "Concurrent reads").

 The summary line reports the files seen this run, by what happened to each, plus
 the skips:
@@ -528,7 +538,8 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),

 #### Stopping a scan

-`scan` stops cleanly on `SIGINT` (Ctrl-C) or `SIGTERM`:
+`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the
+signal `kill` sends by default:

 - The workers stop taking work. Each finishes at most the directory listing or
  the file it is reading, and the progress display is finished.
@@ -540,20 +551,22 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),
 - The line `scan: interrupted after N files` goes to stderr, `N` being the
  number of files the walk reached, and the exit code is 1.

-The next scan skips the records already written and converges as usual.
+The next scan skips the records already written and brings the database up to
+date as usual.

 After the first signal `scan` stops catching signals. A second one ends it at
 once, as any uncaught signal does: the records not yet committed are lost, and
 the database stays valid. `scan` catches `SIGTERM` even when it inherits it as
-ignored. A `SIGINT` that `scan` inherits as ignored, as a background job started
-from a script does, stays ignored.
+ignored. A background job started from a script inherits `SIGINT` as ignored,
+and `scan` leaves it ignored.

 ### `report` mode

-`report` reads every record from the database and takes no positional arguments.
-It never touches the scanned filesystem: it does not stat, open or otherwise
-access any path in the records. Its only I/O is the database, stdout and stderr.
-Its output is the same whether or not the scanned filesystem is still mounted.
+`report` reads every record from the database. It takes no positional arguments:
+nothing may follow `report` on the command line but flags. It never touches the
+scanned filesystem: it does not stat, open or otherwise access any path in the
+records. Its only I/O is the database, stdout and stderr. Its output is the same
+whether or not the scanned filesystem is still mounted.

 Processing:

@@ -625,8 +638,8 @@ Definitions:

 Processing:

- Build the hierarchy, compute every directory's digest, and group the
-  directories by digest. Every group with two or more directories is a
+- `trees` builds the hierarchy, computes every directory's digest, and groups
+  the directories by digest. Every group with two or more directories is a
  duplicate-tree group.
 - Only the outermost duplicate trees are reported. A group is left out when its
  members' parents are distinct directories that all have the same digest: the
@@ -792,17 +805,19 @@ Only one `scan` runs against a database at a time.
  needs the database to itself. A `scan` that starts while a report is still
  reading waits for it up to the busy timeout, then fails. A `scan` that ends
  while a report has the database open prints a warning and leaves the database
-  in WAL mode until the next scan. `report` writes its rows while it reads, so
-  it is still reading while a pager such as `less`, or a stalled pipe, holds up
-  its output.
+  in WAL mode until the next scan. `report` writes its rows while it reads. It
+  is therefore still reading while a pager or a stalled pipe holds up its
+  output. A pager, such as `less`, shows output one screen at a time and waits
+  for a key press.

 #### What `scan` writes, and when

 - As it starts, before the walk: the directory, an empty database file and the
  `.lock` file, each only if missing.
- Then, still before the walk: the schema, the table and index described under
-  "Schema", in a new database, and the `-wal` and `-shm` files, which stay
-  beside the database while the scan runs.
+- Then, still before the walk:
+    - the `-wal` and `-shm` files, which stay beside the database while the scan
+      runs;
+    - in a new database, the table and the index described under "Schema".
 - During the `load` and `walk` phases: no records. A first scan's database holds
  no records until the hash phase, however long its walk runs.
 - During the `hash` phase: records, committed 10,000 at a time.
@@ -988,10 +1003,11 @@ A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
 failed write to stdout, such as a full disk, is reported the same way. Two cases
 never reach `sfdupes` as a failed write:

- When the reader of a stdout pipe exits early, as in `sfdupes report | head`,
-  the next write ends `sfdupes` with `SIGPIPE`, the signal for a write to a
-  closed pipe. It ends quietly and without a summary, as `cat` or `sort` would.
-  The shell reports the signal, status 141 in most shells, not exit 1.
+- The reader of a stdout pipe may exit early, as `head` does in
+  `sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
+  the signal for a write to a closed pipe. It ends quietly and without a
+  summary, as `cat` or `sort` would. The shell reports the signal, status 141 in
+  most shells, not exit 1.
 - When stdout is closed outright, as in `sfdupes report >&-`, the Go runtime
  opens `/dev/null` in its place before `sfdupes` starts. The output is
  discarded, and the run succeeds, as with `> /dev/null`.
@@ -1001,23 +1017,29 @@ never reach `sfdupes` as a failed write:
 This repository follows the
 [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
 standard. The executables in `script/` are the entrypoints of the development
-workflow, and the `Makefile` targets are thin shims that call them. Every script
-is POSIX `sh`, finds the repository root itself, and can be run from any
+workflow, and each `Makefile` target does nothing but call its script. Every
+script is POSIX `sh`, finds the repository root itself, and can be run from any
 directory.

 - `script/bootstrap` installs what building and developing need. It can be run
  again at any time; installed tools are kept. `git`, `make` and `go` are
  checked for presence only. A missing one is installed from the first of `nix`,
-  `apt`, `brew` or `apk` found on the host. An installed `node` is used when its
-  major version is 22. Otherwise the script downloads `nvm`, checks the
-  archive's SHA-256, and installs `node` 22.17.0 through it. It then activates
-  `yarn` 1.22.22 under that `node` through `corepack`. With an installed `node`
-  22, a `yarn` already on `PATH` is used as it is. Otherwise `yarn` 1.22.22 is
-  activated through `corepack`, or installed with `npm` when there is no
-  `corepack`. `yarn install --frozen-lockfile` then installs the `prettier` that
-  `package.json` and `yarn.lock` pin. The linter is never installed; it runs in
-  Docker (see `script/lint`). `docker` is not installed either, and testing,
-  linting and the image build need it. The script ends with `go mod download`.
+  `apt`, `brew` or `apk` found on the host. Then come `node`, the runtime that
+  `prettier` needs, and `yarn`, the package manager that installs `prettier`:
+    - An installed `node` is used when its major version is 22. A `yarn` already
+      on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
+      activated through `corepack`, the `yarn` installer that ships with `node`,
+      or installed with `npm` when there is no `corepack`.
+    - Otherwise the script downloads `nvm`, a `node` version manager, and checks
+      the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
+      activates `yarn` 1.22.22 under that `node` through `corepack`.
+
+    `yarn install --frozen-lockfile` then installs the `prettier` that
+    `package.json` and `yarn.lock` pin. `prettier` is the Markdown formatter.
+    The linter is never installed; it runs in Docker (see `script/lint`).
+    `docker` is not installed either, and testing, linting and the image build
+    need it. The script ends with `go mod download`.
+
 - `script/setup` makes a fresh clone ready for development: `script/bootstrap`,
  then `script/install-precommit`.
 - `script/projectname` prints `sfdupes`. Scripts that need the name call it, so
@@ -1064,7 +1086,7 @@ directory.
  image as `script/docker` does. The Gitea workflow runs it on every push, and a
  successful run means every check passed. The tests and the linter run twice:
  once in `script/check`, and again as stages of the image build.
- `script/precommit` runs from the git pre-commit hook. `go mod tidy` must
+- `script/precommit` runs from the `git` pre-commit hook. `go mod tidy` must
  change nothing, or the commit fails; then `script/check` runs.
 - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
  `script/precommit`.
@@ -1083,8 +1105,9 @@ unchanged package is not linted again.

 ## Build

-The `script/` entrypoints above hold the implementations. The `Makefile` targets
-are shims onto them, except `build`, which carries the compile recipe:
+The `script/` entrypoints above hold the implementations. Each `Makefile` target
+does nothing but call its script, except `build` and `clean`, which carry their
+own recipes:

 - `make` or `make build` builds the `sfdupes` binary with cgo disabled. Building
  is the default target.
@@ -33,7 +33,8 @@
  it. It leads with what the program is and how to use it. A reference section
  covers the settings, the database, the output formats and the exit codes;
  development and testing come last. Every statement was checked against the
-  code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
+  code on `next`. Three comments in `main_test.go` cite "Settings", where their
+  rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)

 - a file or directory removed while the scan runs is not an error, in the walk
  or in the hash phase. The scan prints no line for it and does not count it as
@@ -648,7 +648,7 @@ func TestRunUsageErrors(t *testing.T) {
 }

 func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
-	// README §scan mode: --workers below 1 is a usage error reported in
+	// README §Settings: --workers below 1 is a usage error reported in
 	// one line on stderr, before the scan opens the database.
 	for _, workers := range []string{"0", "-1"} {
 		t.Run(workers, func(t *testing.T) {
@@ -686,8 +686,8 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
 func TestRunHelp(t *testing.T) {
 	t.Parallel()

-	// README §Subcommands: help goes to stderr, exits 0, and leaves
-	// stdout empty.
+	// README §Settings: help goes to stderr, exits 0, and leaves stdout
+	// empty.
 	cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}

 	for _, args := range cases {
@@ -713,8 +713,8 @@ func TestRunHelp(t *testing.T) {
 func TestRunVersion(t *testing.T) {
 	t.Parallel()

-	// README §Subcommands: the version is one line on stdout, with
-	// nothing on stderr, and exits 0.
+	// README §Settings: the version is one line on stdout, with nothing
+	// on stderr, and exits 0.
 	info, _ := debug.ReadBuildInfo()

 	for _, arg := range []string{"--version", "-v"} {