Files
sfdupes/TODO.md
T
clawbot 0064eba542
check / check (push) Failing after 3s
Cut the narration from TODO.md and the script and Dockerfile comments (closes #49)
Completed Steps entries keep what landed, the traps, every disclosure
and every record that a check ran; the argument and history go, with
bare issue numbers turned into full links. Comment blocks in script/,
Dockerfile and Dockerfile.lint keep the trap and drop the defence of
past decisions. TODO.md Workflow now branches from next, targets next,
and leaves merging next to main to the owner. Only comments and
Markdown change.

Model: opus-5-5
2026-10-04 20:47:21 +02:00

403 lines
24 KiB
Markdown

# Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
tracker gets filed as an issue first
- branch from `next`
- do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
as the work)
- push the branch and open a PR against `next` whose title ends with
` (closes #N)`
- an independent review gates each merge to `next`; every finding is addressed
or explicitly rebutted on the PR
- only the owner merges `next` to `main`
# Status
- pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
under the `1.0.0` milestone are what remains before the tag, and this file
records history and process, not the queue
# Next Step
- take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
of truth for what is left before 1.0.0. Individual issues are deliberately not
restated here; a copy in this file drifts out of date the moment the tracker
moves
# Completed Steps
- cut the narration from `TODO.md` Completed Steps and from the comments in
`script/` and both Dockerfiles; §Workflow now branches from and merges to
`next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
- `make test-race` runs the test suite under the race detector in a cgo-enabled
container, outside `make check` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/18)
- a bare `docker build .` fails with a message naming `script/cibuild` and
`script/docker` instead of serving the gates from cache (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39)
- `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and
CI checks it; all Markdown reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19)
- `script/lint` writes no image, so a run no longer leaves an untagged one
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
the `report` and `trees` summary lines, and every subcommand going through
`runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
- `.golangci.yml` replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26)
- a test fails when either `hashWorker` cancellation check in `scan.go` is
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
- a database path holding `?`, `#` or `%` opens exactly the file it names
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
- `scan` rejects `--workers` below 1 as a usage error instead of running
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
- a test fails when either walk cancellation check in `scan.go` is removed
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
- test that `scan` refuses a database with another schema version (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/64)
- correct four inaccurate comments in `cancel_test.go` and rename
`walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/33)
- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/17)
- `scan` creates the schema in one transaction; a version-0 database with a
`files` table is refused with a clear schema-version error (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/11)
- README documents install, Docker, a daily cron scan and how to read and check
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
- the `Dockerfile` build stage keeps the Go module cache out of `builder`'s home
and copies the sources with `--chown`, so no `chown -R` walks them
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)
- `--version` prints `sfdupes VERSION` to stdout; README documents it and
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
deletes nothing more, exits 1 (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/5)
- `report` and `trees` stream the records instead of holding them all in memory;
the schema gains the `files_signature` index (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/14)
- progress prints at once on a non-terminal, uses a real terminal test, and
prints warnings through a spinner instead of racing its redraw (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/13)
- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
the records beneath them (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/9)
- `scan` holds a lock on a lock file beside the database for its whole run, so a
second `scan` fails at once with exit 1 (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/53)
- test stdout write failures in `report` and `trees`; README states that
`| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
- `report` and `trees` open the database read-only, and `scan` leaves it out of
WAL mode, so reading needs only read access (2026-10-03, closes
https://git.eeqj.de/sneak/sfdupes/issues/8)
- escape tabs, newlines, carriage returns and backslashes in report, trees and
warning paths; the root directory's path is `/` (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/7)
- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
(2026-10-02, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
without `.git/config`; the build stage stamps the `VERSION` build argument,
else `git describe --tags --always`, and fails if the context carries `.git`
and the version is still empty, `dev` or `unknown`. CI checks out the full
history (`fetch-depth: 0`) so it stamps the same value as `make build`.
- replace the 1 KiB end-window sampling with the head/tail plus content-hash
ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
documents every rung. A file under 10 MiB is hashed in full, and its `head`,
`tail` and `content` all hold that hash. A larger file gets only its 64 KiB
`head` and `tail` in the hash phase; the content phase, after the update
phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
MiB samples at or above) only when its size, `head` and `tail` match another
record's from this scan or an earlier one, and never reads a file gone or
changed since its record was written. `report` and `trees` leave out any
record without a `content` hash. The `content` column is part of the version 1
schema.
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
`.dockerignore` (2026-09-21, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/22)
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag; digest unchanged.
`script/verify-lint-image-pin` still matches the tagless form, and a tag on
one side only is caught as a plain mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling the linter is never installed on a host. `Dockerfile.lint` copies
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
runs `golangci-lint config verify` and `golangci-lint run` as build steps;
`script/lint` builds it. `script/bootstrap` no longer installs or pins the
linter, and warns rather than fails when `docker` is absent;
`ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`.
`script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate
in both files, compares their two `FROM` lines and restates neither pin.
Traps: an unchanged tree lets a lint build pass in under a second having run
no linter, so every gate `RUN` references `ARG CHECK_EPOCH` (BuildKit hashes
the expanded command) and `script/lint` passes `"$(date +%s)-$$"`, the PID
because two runs land in the same second easily. Nothing inside an image build
may shell out to docker, so the `Dockerfile` lint stage calls `golangci-lint`
directly and the build stage runs `make test` and `make fmt-check` instead of
`make check`, through `make` because the Makefile's `export CGO_ENABLED = 0`
only reaches what it invokes. `COPY --from=lint /src/go.sum /dev/null`
replaces the copied linter binary as the only edge making the build stage wait
for lint; dropping it would end fail-fast linting under a still-green build.
`golangci-lint config verify`, included per the ruling, validates from an
embedded schema with no network call, but `go mod download` above the gates
still needs the network on a cold cache. Verified: `make lint` green with no
`golangci-lint` on `PATH`; two back-to-back `script/lint` runs on an untouched
tree both ran the linter (27.7s and 28.7s in the lint step, `COPY . .`
`CACHED` above); a planted unused variable failed `script/lint`, and failed
`make docker` at `[lint 9/9]` with the build stage stopped at
`[builder 3/12]`; the drift guard fails on a tag-only, a digest-only and an
unreadable reference, naming both sides; under `--network none` config verify
passes a valid config and rejects an invalid one; `make docker` green in 5m35s
with all six gates run under one epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`); in the
builder image with the Go test cache off, `--user 0:0` still fails
`TestScanHardlinkRunFailsTogether` where the unprivileged user passes. Noted
for follow-up, not fixed here: `golangci-lint` warns that `gomodguard` is
deprecated since v2.12.0 in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42):
the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
which ends in `go mod download`, so the separate call to it is gone.
`COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap
layer: it is the only edge making this stage depend on the lint stage, so
deleting it would end fail-fast linting silently. A new
`script/verify-linter-pin`, run in the build stage before bootstrap, fails the
build naming both versions unless that copied binary is the version
`script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip.
`$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. Everything
added sits above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still
precede `make check`. Verified: the guard fails the build with both versions
named when the lint stage's linter is faked to another version, and passes an
unmodified build; bootstrap runs clean under Alpine's `sh` and `apk`, finding
the copied linter already at the pin; a second build served the bootstrap and
dependency layers `CACHED` while both gates ran with a fresh epoch; a planted
`unused` finding failed the build at the lint gate in 48.9s with the build
stage's `make check` never starting; and the suite run in the image as
`--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the drop to the
unprivileged user is still needed. That last check needs the Go test cache
off: as root it first reported `ok ... (cached)`, reusing the build-time
result. Build times on a noisy shared host: 2m13s on an unchanged tree, 2m17s
and 4m29s after a source change, 5m14s cold, which breaches the policy
ceiling; `chown -R builder:builder /src /home/builder` walks the module cache
and alone varied from 77s to 210s across those builds, and `main` measured
5m03s cold with a 209s `chown`. Filed as
https://git.eeqj.de/sneak/sfdupes/issues/43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
served them from cache and the build exited 0 having run nothing. Both scripts
now pass `--build-arg CHECK_EPOCH="$(date +%s)"`. `ARG` is per stage and the
gates span two stages, so it is declared in both; BuildKit hashes the expanded
command, so each gate `RUN` echoes the epoch, which also logs it as evidence
the layer ran. It sits below the dependency layers so they stay cached.
Verified under `BUILDKIT_PROGRESS=plain`, each script run twice back to back
on an unchanged tree: all three gates ran on all four runs with a fresh epoch
(`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s then 53.4s), and
thirteen steps were still served `CACHED`. With a planted `unused` finding the
build failed at `make lint` in 36.1s and the build-stage `make check` never
started. Run as root, the same image fails `TestScanHardlinkRunFailsTogether`,
because root reads through the `chmod(0)` the test relies on, so the build
stage must drop to the unprivileged `builder` user. Local fix only;
propagating it to the canonical templates is
https://git.eeqj.de/sneak/prompts/issues/26
- check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
any installed version that is not the pin — older, newer, absent or
unparseable — is reinstalled. `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
bootstrap re-reads the effective version after installing and, on a mismatch,
prints both paths and both versions and exits non-zero; it does not reorder
`PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
presence-only checks. Verified by bootstrapping this host from v2.10.1 to
v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
covering a thirteen-input version-parse matrix, a shadowed install that must
exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
binary that must hit the timeout; `make check` and `make lint` are clean at
v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
pool is now an owned, context-aware `hashPool`: every blocking send in the
feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
`results` until the last goroutine has exited — draining is what frees a
worker already parked on a send. `ctx` is threaded from `cmd.Context()`
through `runScan`, `syncScan`, both worker pools and the whole database layer,
as the first parameter everywhere. The walk pool gets the same treatment plus
a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
census, and every file it never reached looks vanished to the update phase.
That phase's own `BeginTx` also fails on the cancelled context before deleting
anything, but the guard is the barrier that still holds once an interrupted
scan may commit what it has. Tests drive `run(scan)` against a database whose
insert trigger aborts and assert that the scan fails instead of hanging and
that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
cancel a scan part-way through the walk, deterministically, by counting its
own consultations of `ctx.Done()`, and assert that it stops at the guard
holding a partial census and a still-populated record index, with every record
intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
`feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
branches of both pools
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
`db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
`fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
`runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
errors instead. The single exit point is `run` in `main.go`: it maps a
`fatalError` (anything a subcommand returned) to exit 1 and cobra's own
argument and flag errors to exit 2, which keeps a runtime failure from being
reported as a usage error or printing the usage text. New `main_test.go`
drives the CLI in-process and asserts the exit codes from README §Error
handling plus the stdout/stderr split, including that a fatal error raised
after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
`report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
`golangci-v2.12.2`, merged as `38a01bd`, closes
https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
`Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
`funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
the v2 schema, so they are actually applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
like the other managed repos (2026-07-26, commit `3abeacf`, closes
https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
and every Makefile target is now a thin shim over them
- make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
`check` plus `build`); `make build` remains as an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
`scan-wide-phases`): all operands seed the shared walk pool and every pass
runs once over the whole scan, so totals and ETAs are scan-global; the
per-operand walk/hash/update cycles and their stderr announcements are gone;
the update pass commits in batched transactions — the filesystem is
authoritative and the database an eventually-consistent reflection, so
scan-level atomicity is not required
- split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
update per operand — with parallelism only inside each phase; the walk
enumerates paths with per-directory workers and the stat pass lstats them with
per-file workers, restoring the exact total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24, branch
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
whole run's
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
4h on a ZFS server); it is now a per-directory worker-pool traversal that
records size/mtime during the walk (folding away the separate stat pass,
halving metadata I/O), and each `PATH` operand commits in its own transaction
so an interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
disabled) keyed by absolute path that survives between runs — a rescan hashes
only new or changed files (by mtime/size), deletes records for files vanished
from under the scanned operands, and leaves records outside them untouched, so
`scan` can be cronned daily; `report` and `trees` read the database (no
positional arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
journaling plus a single-transaction update keep a report run during a scan
safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on `main`
(2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps
- possible later features (explicitly out of scope per README): full-content
verification of candidates, removal-script helpers
# Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
- [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed from
house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
`make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
`docker`, `hooks`; `check` currently depends on `build`, which writes the
binary (`make check` must not modify files)
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
with a 30-second timeout and the conditional `-v` rerun pattern (suite
covers parsing, grouping, digests, suppression, hashing, and the scan
pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
lint stage, build stage running `make check`
- [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
action pinned by commit SHA)
- [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the
repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists)
Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine for a
small single-binary tool per the Go styleguide; the tracker audit agrees
- `make test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
builds) and the race detector requires cgo, so the detector runs in a separate
cgo-enabled container, `make test-race`, which is not part of `make check`