1 Commits
Author SHA1 Message Date
sneak 16a57547b8 Rewrite TODO.md sentence by sentence (closes #135)
check / check (push) Canceled after 0s
TODO.md is the project's own task list, judged by the same rules as
README.md: short sentences, lists as bullets, nothing a newcomer reads
twice. Every completed step is kept with its date and its issue or
branch, and every fact the old entries recorded is kept too, in short
sentences. The 2026-07-22 policy-compliance checklist is folded into
the step that did the work, and the accepted divergence from the
layout policy gets a section of its own. Workflow, Status and Next
Step keep their rules unchanged.

Model: fable-5-1
2026-10-26 07:23:22 +00:00
+59 -76
View File
@@ -39,8 +39,7 @@ are known.
it. It leads with what the program is and how to use it. A reference section it. It leads with what the program is and how to use it. A reference section
covers the settings, the database, the output formats and the exit codes; covers the settings, the database, the output formats and the exit codes;
development and testing come last. Every statement was checked against the development and testing come last. Every statement was checked against the
code on `next`. Three comments in `main_test.go` cite "Settings", where their code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
- A file or directory removed while the scan runs is not an error, in the walk - A file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as or in the hash phase. The scan prints no line for it and does not count it as
@@ -63,9 +62,7 @@ are known.
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
up. Gone are: up. Gone are:
- the 10 MiB and 50 MiB thresholds; - the 10 MiB and 50 MiB thresholds;
- the size census: the sizes of every walked file and of every record - the size census;
outside the operands, by which a file was hashed only when another file
shared its size;
- the records without hashes; - the records without hashes;
- the content phase. - the content phase.
@@ -110,20 +107,20 @@ are known.
- `scan` never records or reads anything on a filesystem that is not persistent: - `scan` never records or reads anything on a filesystem that is not persistent:
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It `tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
matches them by device ID, however the walk or the content phase reaches a matches them by device ID. As the scan starts, it lists the mounted
path. As the scan starts, it lists the mounted filesystems of those types, filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
from `/proc/self/mountinfo` on Linux and from the mount list on macOS. No flag mount list on macOS. No flag turns that off. Below an operand, the walk does
turns that off. Below an operand, the walk does not enter the default excludes not enter the default excludes either, so `sfdupes scan /` skips the kernel's
either, so `sfdupes scan /` skips the kernel's view of the running system and view of the running system and the temporary directories. On Linux those are
the temporary directories. On Linux those are `/proc`, `/sys`, `/dev`, `/run`, `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`. On macOS they are
`/tmp` and `/var/tmp`. On macOS they are `/dev`, `/private/tmp`, `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
`/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/VM` and `/System/Volumes/Data`. A Windows list exists in the
`/System/Volumes/Data`. A Windows list exists in the code but is not used. An code but is not used. An operand at or inside a default exclude is still
operand at or inside a default exclude is still walked. The scan prints a line walked. The scan prints a line for each path it leaves out, without counting
for each path it leaves out, without counting it as skipped, and the records it as skipped, and the records there count as outside the scanned operands, as
there count as outside the scanned operands, as for `-x`. for `-x`. `--no-default-excludes` turns off the default excludes only, and
`--no-default-excludes` turns off the default excludes only, and `scan --help` `scan --help` lists them (2026-10-17,
lists them (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/124) https://git.eeqj.de/sneak/sfdupes/issues/124)
- The rate a progress display shows covers the items completed over the last 5 - The rate a progress display shows covers the items completed over the last 5
seconds, or since the line before when stderr is not a terminal. It falls to 0 seconds, or since the line before when stderr is not a terminal. It falls to 0
@@ -132,15 +129,14 @@ are known.
phase's average. Once no item has completed for 10 seconds, the word `stalled` phase's average. Once no item has completed for 10 seconds, the word `stalled`
and where the phase waits take the rate's place, as `stalled disk/db`. The and where the phase waits take the rate's place, as `stalled disk/db`. The
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
phase that may wait on either. Each phase's word is set in the code that phase that may wait on either. Each phase gives its word where it is defined.
defines the phase. A line too wide for the terminal loses its bar first, then A line too wide for the terminal loses its bar first, then its rate or those
its rate or those words, so the phase, progress, elapsed time and ETA stay words, so the phase, progress, elapsed time and ETA stay whole. On an
whole. On an 80-column terminal, at counts like those of README's example, the 80-column terminal, at counts like those of README's example, the content
content phase's `checks` and `reads` lines were too wide for those words. That phase's checks and reads lose those words at most times in minutes. Every
held at most elapsed times and ETAs in minutes. Every other phase kept them other phase keeps them whenever the elapsed time and the ETA are both in
whenever the elapsed time and the ETA were both in minutes. The content minutes. The content phase's last partial batch is written under a display of
phase's last partial batch is written under a display of its own (2026-10-16, its own (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/125)
https://git.eeqj.de/sneak/sfdupes/issues/125)
- New subcommand `info` prints the database's details on stdout, one per line as - New subcommand `info` prints the database's details on stdout, one per line as
a name, a tab and the value: a name, a tab and the value:
@@ -193,8 +189,8 @@ are known.
narrow, also after the terminal is made narrower during a phase. A line too narrow, also after the terminal is made narrower during a phase. A line too
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
time and ETA whole. It is cut short only when those alone do not fit, as below time and ETA whole. It is cut short only when those alone do not fit, as below
about 70 columns late in a large scan. When stderr is not a terminal, the line about 70 columns late in a large scan. In plain mode the line is printed every
is printed every 5 seconds whether or not an item completed (2026-10-16, 5 seconds whether or not an item completed (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/112) https://git.eeqj.de/sneak/sfdupes/issues/112)
- `scan`, `report` and `trees` first print a line on stderr naming the database - `scan`, `report` and `trees` first print a line on stderr naming the database
@@ -318,8 +314,8 @@ are known.
`files` table is refused with a clear schema-version error (2026-10-04, `files` table is refused with a clear schema-version error (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/11) https://git.eeqj.de/sneak/sfdupes/issues/11)
- README documents install, Docker, a daily `cron` scan and how to read and - README documents install, Docker, a daily cron scan and how to read and check
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home - The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
and copied the sources with `--chown`, so no `chown -R` walked them. Since and copied the sources with `--chown`, so no `chown -R` walked them. Since
@@ -363,12 +359,12 @@ are known.
and warning paths. The root directory's path is `/` (2026-10-03, and warning paths. The root directory's path is `/` (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/7) https://git.eeqj.de/sneak/sfdupes/issues/7)
- A plain `docker build .` stamps the `git` tag or short commit instead of - A plain `docker build .` stamps the git tag or short commit instead of `dev`.
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage `.dockerignore` sends `.git` without `.git/config`. The build stage stamps the
stamps the `VERSION` build argument, else `git describe --tags --always`. It `VERSION` build argument, else `git describe --tags --always`. It fails if the
fails if the context carries `.git` and the version is still empty, `dev` or context carries `.git` and the version is still empty, `dev` or `unknown`. CI
`unknown`. CI checks out the full history (`fetch-depth: 0`), so it stamps the checks out the full history (`fetch-depth: 0`), so it stamps the same value as
same value as `make build` (2026-10-02, branch `next`, `make build` (2026-10-02, branch `next`,
https://git.eeqj.de/sneak/sfdupes/issues/67) https://git.eeqj.de/sneak/sfdupes/issues/67)
- The 1 KiB end-window sampling was replaced with the hashes chosen by file - The 1 KiB end-window sampling was replaced with the hashes chosen by file
@@ -442,19 +438,10 @@ are known.
installing. On a mismatch it printed both paths and both versions and exited installing. On a mismatch it printed both paths and both versions and exited
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version` non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`, call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
`make` and `go` kept presence-only checks. It was verified by bootstrapping `make` and `go` kept presence-only checks. `make check` and `make lint` were
this host from v2.10.1 to v2.12.2, and again to a no-op. Stub runs of the clean at v2.12.2, so v2.10.1 had not been hiding findings on `main`.
script under `dash` covered: Superseded by https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no
- a thirteen-input version-parse matrix; longer installed on a host (2026-08-09, branch `bootstrap-version-check`,
- a shadowed install, which must exit non-zero;
- an install destination not on `PATH`;
- `GOBIN` set;
- a wedged binary, which must hit the timeout.
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
hiding findings on `main`. Superseded by
https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no longer
installed on a host (2026-08-09, branch `bootstrap-version-check`,
https://git.eeqj.de/sneak/sfdupes/issues/24) https://git.eeqj.de/sneak/sfdupes/issues/24)
- The hash worker pool unwinds on the error path. The pool is an owned, - The hash worker pool unwinds on the error path. The pool is an owned,
@@ -467,14 +454,12 @@ are known.
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
cancelled walk yields a partial size census, and every file it never reached cancelled walk yields a partial size census, and every file it never reached
looks vanished to the update phase. That phase's own `BeginTx` also fails on looks vanished to the update phase. That phase's own `BeginTx` also fails on
the cancelled context before deleting anything. The guard is still the barrier the cancelled context before deleting anything, but the guard is the barrier
once an interrupted scan may commit what it has. Tests drive `run(scan)` that still holds once an interrupted scan may commit what it has. Tests drive
against a database whose insert trigger aborts, and assert that the scan fails `run(scan)` against a database whose insert trigger aborts, and assert that
instead of hanging and that the goroutine count returns to its pre-scan the scan fails instead of hanging and that the goroutine count returns to its
baseline. Other tests cancel a scan part-way through the walk, pre-scan baseline. Other tests cancel a scan part-way through the walk and
deterministically, by counting the scan's own checks of `ctx.Done()`. They assert that it stops at the guard with every record intact. Direct tests of
assert that it stops at the guard, holding a partial size census and a
still-populated record index, with every record intact. Direct tests of
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker` `sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
and `hashPhase` cover the remaining cancellation branches of both pools and `hashPhase` cover the remaining cancellation branches of both pools
(2026-08-09, branch `hash-pool-cleanup`, (2026-08-09, branch `hash-pool-cleanup`,
@@ -484,13 +469,13 @@ are known.
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint, are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`, runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
`loadRecords` and `resolveRoots` return errors instead. The single exit point `loadRecords` and `resolveRoots` return errors instead. The single exit point
is `run` in `main.go`. It maps a `fatalError`, anything a subcommand returned, is `run` in `main.go`. It maps a `fatalError` to exit 1 and cobra's own
to exit 1, and cobra's own argument and flag errors to exit 2. So a runtime argument and flag errors to exit 2, so a runtime failure is never reported as
failure is never reported as a usage error, nor printed with the usage text. a usage error or printed with the usage text. `main_test.go` drives the CLI
`main_test.go` drives the CLI in-process and asserts the exit codes from in-process and asserts the exit codes from README "Error handling" and the
README "Error handling" and the stdout/stderr split. That includes a fatal stdout/stderr split. That includes a fatal error after the database is open
error after the database is open leaving no `-wal` or `-shm` file behind, for leaving no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
`scan`, `report` or `trees` (2026-08-09, branch `db-close-on-fatal`, (2026-08-09, branch `db-close-on-fatal`,
https://git.eeqj.de/sneak/sfdupes/issues/4) https://git.eeqj.de/sneak/sfdupes/issues/4)
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as - `golangci-lint` updated to v2.12.2 with the canonical config, merged as
@@ -501,17 +486,17 @@ are known.
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3) `golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
- `Makefile` targets converted to scripts-to-rule-them-all `script/` - Makefile targets converted to scripts-to-rule-them-all `script/` entrypoints,
entrypoints, like the other managed repos. All 12 entrypoints exist: like the other managed repos. All 12 entrypoints exist:
- `bootstrap`, `setup` and `projectname`; - `bootstrap`, `setup` and `projectname`;
- `test`, `lint`, `fmt`, `fmt-check` and `check`; - `test`, `lint`, `fmt`, `fmt-check` and `check`;
- `docker`, `cibuild`, `precommit` and `install-precommit`. - `docker`, `cibuild`, `precommit` and `install-precommit`.
Every `Makefile` target does nothing but call its script (2026-07-26, commit Every Makefile target is a thin shim over them (2026-07-26, commit
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1) `3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
- The binary is the default `make` target: plain `make` builds `sfdupes`, where - The binary is the default Make target: plain `make` builds `sfdupes`, where it
it used to run `check` plus `build`. `make build` remains as an alias used to run `check` plus `build`. `make build` remains as an alias
(2026-07-24, branch `make-default-target`) (2026-07-24, branch `make-default-target`)
- Scan-wide phases, concurrent operands and batched updates. All operands seed - Scan-wide phases, concurrent operands and batched updates. All operands seed
@@ -573,11 +558,9 @@ are known.
- `LICENSE` and the README License section. MIT was assumed from house - `LICENSE` and the README License section. MIT was assumed from house
convention, for the owner to confirm; convention, for the owner to confirm;
- `REPO_POLICIES.md` in the repo root; - `REPO_POLICIES.md` in the repo root;
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so - the canonical `.golangci.yml`, with the 150 findings it raised fixed;
that `make lint` is clean;
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and - the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
`hooks`. `check` depended on `build`, which wrote the binary, and `hooks`, and a `check` target that no longer depends on `build`;
`make check` must not modify files, so `check` no longer depends on it;
- a test suite, which the policy required with a 30-second timeout and the - a test suite, which the policy required with a 30-second timeout and the
conditional `-v` rerun. It covers parsing, grouping, digests, suppression, conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
hashing and the scan pipeline, at 64% coverage; hashing and the scan pipeline, at 64% coverage;
@@ -589,7 +572,7 @@ are known.
name, purpose, category, license and author, then Getting Started, name, purpose, category, license and author, then Getting Started,
Rationale, TODO, License and Author; Rationale, TODO, License and Author;
- the stale README non-goal "no git repository setup and no CI" removed; - the stale README non-goal "no git repository setup and no CI" removed;
- the pre-commit hook, installed with `make hooks` once that target existed. - the pre-commit hook installed.
The code was already `gofmt`-clean, so no standalone formatting commit was The code was already `gofmt`-clean, so no standalone formatting commit was
needed (2026-07-23, branch `repo-policy-compliance`) needed (2026-07-23, branch `repo-policy-compliance`)