sfdupes/TODO.md

# Workflow

- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
  tracker gets filed as an issue first
- branch from `next`
- do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
  as the work)
- push the branch and open a PR against `next` whose title ends with
  ` (closes #N)`
- an independent review gates each merge to `next`; every finding is addressed
  or explicitly rebutted on the PR
- only the owner merges `next` to `main`

# Status

- pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
  under the `1.0.0` milestone are what remains before the tag, and this file
  records history and process, not the queue

# Next Step

- take the next issue from the `1.0.0` milestone on the tracker:
  https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
  of truth for what is left before 1.0.0. Individual issues are deliberately not
  restated here; a copy in this file drifts out of date the moment the tracker
  moves

# Completed Steps

- re-vendor the canonical files from `sneak/prompts` commit `dd4027b`, with
  `script/lint`, `script/test` and `REPO_POLICIES.md` from its `next` at
  `c55a0cb`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
  that write no image, and `make test` runs the suite under the race detector,
  so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are
  gone; every `docker build` in `script/` passes `--no-cache`; prettier runs on
  the host, from the node and yarn `script/bootstrap` installs, so the
  `prettier` and `markdown` stages are gone and the build stage installs `git`
  and `make` itself; `.claude/settings.json` is deleted (2026-10-08,
  https://git.eeqj.de/sneak/sfdupes/issues/95)

- cut the narration from `TODO.md` Completed Steps and from the comments in
  `script/` and both Dockerfiles; §Workflow now branches from and merges to
  `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)

- `make test-race` runs the test suite under the race detector in a cgo-enabled
  container, outside `make check` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/18)

- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
  rather than serve the gates from cache (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/39). Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
  `REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
  `script/` pass `--no-cache`, so theirs always run

- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks
  it; all Markdown reformatted (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/19)

- `script/lint` writes no image, so a run no longer leaves an untagged one
  behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)

- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
  the `report` and `trees` summary lines, and every subcommand going through
  `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)

- `.golangci.yml` replaced with the current canonical copy, which uses
  `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/26)

- a test fails when either `hashWorker` cancellation check in `scan.go` is
  removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)

- a database path holding `?`, `#` or `%` opens exactly the file it names
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)

- `scan` rejects `--workers` below 1 as a usage error instead of running
  single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)

- a test fails when either walk cancellation check in `scan.go` is removed
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)

- test that `scan` refuses a database with another schema version (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/64)

- correct four inaccurate comments in `cancel_test.go` and rename
  `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/33)

- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/17)

- `scan` creates the schema in one transaction; a version-0 database with a
  `files` table is refused with a clear schema-version error (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/11)

- README documents install, Docker, a daily cron scan and how to read and check
  the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)

- the `Dockerfile` build stage keeps the Go module cache out of `builder`'s home
  and copies the sources with `--chown`, so no `chown -R` walks them
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)

- `--version` prints `sfdupes VERSION` to stdout; README documents it and
  `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)

- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
  deletes nothing more, exits 1 (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/5)

- `report` and `trees` stream the records instead of holding them all in memory;
  the schema gains the `files_signature` index (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/14)

- progress prints at once on a non-terminal, uses a real terminal test, and
  prints warnings through a spinner instead of racing its redraw (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/13)

- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
  the records beneath them (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/9)

- `scan` holds a lock on a lock file beside the database for its whole run, so a
  second `scan` fails at once with exit 1 (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/53)

- test stdout write failures in `report` and `trees`; README states that
  `| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
  (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)

- `report` and `trees` open the database read-only, and `scan` leaves it out of
  WAL mode, so reading needs only read access (2026-10-03, closes
  https://git.eeqj.de/sneak/sfdupes/issues/8)

- escape tabs, newlines, carriage returns and backslashes in report, trees and
  warning paths; the root directory's path is `/` (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/7)

- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
  (2026-10-02, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
  without `.git/config`; the build stage stamps the `VERSION` build argument,
  else `git describe --tags --always`, and fails if the context carries `.git`
  and the version is still empty, `dev` or `unknown`. CI checks out the full
  history (`fetch-depth: 0`) so it stamps the same value as `make build`.

- replace the 1 KiB end-window sampling with the head/tail plus content-hash
  ladder (2026-09-22, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
  documents every rung. A file under 10 MiB is hashed in full, and its `head`,
  `tail` and `content` all hold that hash. A larger file gets only its 64 KiB
  `head` and `tail` in the hash phase; the content phase, after the update
  phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
  MiB samples at or above) only when its size, `head` and `tail` match another
  record's from this scan or an earlier one, and never reads a file gone or
  changed since its record was written. `report` and `trees` leave out any
  record without a `content` hash. The `content` column is part of the version 1
  schema.

- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
  `.dockerignore` (2026-09-21, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/22)

- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
  `Dockerfile.lint` (2026-08-10, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy
  `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
  without the false `(Debian-based)` note or the tag; digest unchanged.
  `script/verify-lint-image-pin` still matches the tagless form, and a tag on
  one side only is caught as a plain mismatch.

- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
  branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
  owner ruling the linter is never installed on a host. `Dockerfile.lint` copies
  the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
  runs `golangci-lint config verify` and `golangci-lint run` as build steps;
  `script/lint` builds it. `script/bootstrap` no longer installs or pins the
  linter, and warns rather than fails when `docker` is absent;
  `ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`.
  `script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate
  in both files, compares their two `FROM` lines and restates neither pin.
  Traps: nothing inside an image build may shell out to docker, so the
  `Dockerfile` lint stage calls `golangci-lint` directly and the build stage
  runs `make test` and `make fmt-check` instead of `make check`, through `make`
  because the Makefile's `export CGO_ENABLED = 0` only reaches what it invokes.
  `COPY --from=lint /src/go.sum /dev/null` replaces the copied linter binary as
  the only edge making the build stage wait for lint; dropping it would end
  fail-fast linting under a still-green build. `golangci-lint config verify`,
  included per the ruling, validates from an embedded schema with no network
  call, but `go mod download` above the gates still needs the network on a cold
  cache. Verified: `make lint` green with no `golangci-lint` on `PATH`; two
  back-to-back `script/lint` runs on an untouched tree both ran the linter
  (27.7s and 28.7s in the lint step, `COPY . .` `CACHED` above); a planted
  unused variable failed `script/lint`, and failed `make docker` at `[lint 9/9]`
  with the build stage stopped at `[builder 3/12]`; the drift guard fails on a
  tag-only, a digest-only and an unreadable reference, naming both sides; under
  `--network none` config verify passes a valid config and rejects an invalid
  one; `make docker` green in 5m35s with all six gates run (lint 37.6s, test
  25.2s reporting `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not
  `(cached)`); in the builder image with the Go test cache off, `--user 0:0`
  still fails `TestScanHardlinkRunFailsTogether` where the unprivileged user
  passes. Noted for follow-up, not fixed here: `golangci-lint` warns that
  `gomodguard` is deprecated since v2.12.0 in favour of `gomodguard_v2`.

- install the Docker build stage's prerequisites by running `script/bootstrap`
  instead of `apk add --no-cache make` inline (2026-08-09, branch
  `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42):
  the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
  which ends in `go mod download`, so the separate call to it is gone.
  `COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap
  layer: it is the only edge making this stage depend on the lint stage, so
  deleting it would end fail-fast linting silently. A new
  `script/verify-linter-pin`, run in the build stage before bootstrap, fails the
  build naming both versions unless that copied binary is the version
  `script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip.
  `$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. The `chown`
  and `USER builder` still precede `make check`. Verified: the guard fails the
  build with both versions named when the lint stage's linter is faked to
  another version, and passes an unmodified build; bootstrap runs clean under
  Alpine's `sh` and `apk`, finding the copied linter already at the pin; a
  second build served the bootstrap and dependency layers `CACHED` while both
  gates ran; a planted `unused` finding failed the build at the lint gate in
  48.9s with the build stage's `make check` never starting; and the suite run in
  the image as `--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the
  drop to the unprivileged user is still needed. That last check needs the Go
  test cache off: as root it first reported `ok ... (cached)`, reusing the
  build-time result. Build times on a noisy shared host: 2m13s on an unchanged
  tree, 2m17s and 4m29s after a source change, 5m14s cold, which breaches the
  policy ceiling; `chown -R builder:builder /src /home/builder` walks the module
  cache and alone varied from 77s to 210s across those builds, and `main`
  measured 5m03s cold with a 209s `chown`. Filed as
  https://git.eeqj.de/sneak/sfdupes/issues/43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
  `script/docker` cannot report a green they did not earn (2026-08-09, branch
  `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
  `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
  served them from cache and the build exited 0 having run nothing. Every
  `docker build` in `script/` now passes `--no-cache` instead
  (https://git.eeqj.de/sneak/sfdupes/issues/95). Run as root, the tests fail
  `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
  the test relies on, so they run as an unprivileged user
- check the installed golangci-lint version in `script/bootstrap` instead of
  only its presence (2026-08-09, branch `bootstrap-version-check`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
  `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
  any installed version that is not the pin — older, newer, absent or
  unparseable — is reinstalled. `go install` writes into `GOBIN` (or
  `GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
  bootstrap re-reads the effective version after installing and, on a mismatch,
  prints both paths and both versions and exits non-zero; it does not reorder
  `PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
  bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
  presence-only checks. Verified by bootstrapping this host from v2.10.1 to
  v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
  covering a thirteen-input version-parse matrix, a shadowed install that must
  exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
  binary that must hit the timeout; `make check` and `make lint` are clean at
  v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch
  `hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
  pool is now an owned, context-aware `hashPool`: every blocking send in the
  feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
  out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
  `results` until the last goroutine has exited — draining is what frees a
  worker already parked on a send. `ctx` is threaded from `cmd.Context()`
  through `runScan`, `syncScan`, both worker pools and the whole database layer,
  as the first parameter everywhere. The walk pool gets the same treatment plus
  a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
  census, and every file it never reached looks vanished to the update phase.
  That phase's own `BeginTx` also fails on the cancelled context before deleting
  anything, but the guard is the barrier that still holds once an interrupted
  scan may commit what it has. Tests drive `run(scan)` against a database whose
  insert trigger aborts and assert that the scan fails instead of hanging and
  that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
  cancel a scan part-way through the walk, deterministically, by counting its
  own consultations of `ctx.Done()`, and assert that it stops at the guard
  holding a partial census and a still-populated record index, with every record
  intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
  `feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
  branches of both pools
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
  `db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
  `fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
  with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
  `runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
  errors instead. The single exit point is `run` in `main.go`: it maps a
  `fatalError` (anything a subcommand returned) to exit 1 and cobra's own
  argument and flag errors to exit 2, which keeps a runtime failure from being
  reported as a usage error or printing the usage text. New `main_test.go`
  drives the CLI in-process and asserts the exit codes from README §Error
  handling plus the stdout/stderr split, including that a fatal error raised
  after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
  `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
  `golangci-v2.12.2`, merged as `38a01bd`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
  `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
  replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
  `funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
  the v2 schema, so they are actually applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
  like the other managed repos (2026-07-26, commit `3abeacf`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
  exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
  `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
  and every Makefile target is now a thin shim over them
- make the binary the default Make target (2026-07-24, branch
  `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
  `check` plus `build`); `make build` remains as an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
  `scan-wide-phases`): all operands seed the shared walk pool and every pass
  runs once over the whole scan, so totals and ETAs are scan-global; the
  per-operand walk/hash/update cycles and their stderr announcements are gone;
  the update pass commits in batched transactions — the filesystem is
  authoritative and the database an eventually-consistent reflection, so
  scan-level atomicity is not required
- split the stat pass back out of the walk (2026-07-24, branch
  `parallel-phases`): phases are strictly sequential again — walk, stat, hash,
  update per operand — with parallelism only inside each phase; the walk
  enumerates paths with per-directory workers and the stat pass lstats them with
  per-file workers, restoring the exact total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24, branch
  `scan-operand-progress`): with per-operand walk/hash/update cycles, a
  multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
  whole run's
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
  goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
  4h on a ZFS server); it is now a per-directory worker-pool traversal that
  records size/mtime during the walk (folding away the separate stat pass,
  halving metadata I/O), and each `PATH` operand commits in its own transaction
  so an interrupted scan keeps completed operands

- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
  now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
  disabled) keyed by absolute path that survives between runs — a rescan hashes
  only new or changed files (by mtime/size), deletes records for files vanished
  from under the scanned operands, and leaves records outside them untouched, so
  `scan` can be cronned daily; `report` and `trees` read the database (no
  positional arguments) instead of a scan stream. Database at
  `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
  journaling plus a single-transaction update keep a report run during a scan
  safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
  and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
  `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
  `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
  boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch
  `repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on `main`
  (2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history)

# Future Steps

- possible later features (explicitly out of scope per README): full-content
  verification of candidates, removal-script helpers

# Repo Policy Compliance

Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
formatting commit is needed.

- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
      untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
- [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed from
      house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
      `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
      `docker`, `hooks`; `check` currently depends on `build`, which writes the
      binary (`make check` must not modify files)
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
      with a 30-second timeout and the conditional `-v` rerun pattern (suite
      covers parsing, grouping, digests, suppression, hashing, and the scan
      pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
      lint stage, build stage running `make check`
- [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
      action pinned by commit SHA)
- [x] README lacks required sections: Description first line
      (name/purpose/category/license/author), Getting Started, Rationale, TODO,
      License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the
      repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists)

Accepted divergences (no action):

- flat single-package layout with `.go` files in the repo root — fine for a
  small single-binary tool per the Go styleguide; the tracker audit agrees