sfdupes/TODO.md

# Workflow

- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
  tracker gets filed as an issue first
- branch from `next`
- do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
  as the work)
- push the branch and open a PR against `next` whose title ends with
  ` (closes #N)`
- an independent review gates each merge to `next`; every finding is addressed
  or explicitly rebutted on the PR
- only the owner merges `next` to `main`

# Status

- pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
  under the `1.0.0` milestone are what remains before the tag, and this file
  records history and process, not the queue

# Next Step

- take the next issue from the `1.0.0` milestone on the tracker:
  https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
  of truth for what is left before 1.0.0. Individual issues are deliberately not
  restated here; a copy in this file drifts out of date the moment the tracker
  moves

# Completed Steps

- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at
  `e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
  that write no image, and `make test` runs the suite under the race detector,
  so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are
  gone; every `docker build` in `script/` passes a new `CACHEBUST` value, which
  every stage of the `Dockerfile` declares first, so every `RUN` step runs on
  every build, while the steps that compile keep Go's build cache and the tests
  run with `-count=1`; prettier runs on the host, from the node and yarn
  `script/bootstrap` installs, so the `prettier` and `markdown` stages are gone
  and the build stage installs `git` and `make` itself; a new push cancels the
  workflow's older run on the same branch, and a run stops after 20 minutes;
  `.claude/settings.json` is deleted (2026-10-15,
  https://git.eeqj.de/sneak/sfdupes/issues/95)

- with `-x`, an operand under another operand is walked on its own instead of
  being dropped when a directory from it (from its directory, for a regular
  file) up to but not including that operand is on a different filesystem than
  that operand, whose walk then cannot reach it, and the records of files gone
  from below it are deleted (2026-10-14,
  https://git.eeqj.de/sneak/sfdupes/issues/106)

- with `-x`, `scan` keeps the records below a directory on another filesystem
  under its operand, and counts their sizes in the size census, instead of
  deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)

- the hash and content phases open each file without blocking and check that it
  is still a regular file, so one replaced by a FIFO or device node after it was
  statted is warned about and skipped instead of hanging the scan (2026-10-14,
  https://git.eeqj.de/sneak/sfdupes/issues/100)

- the scan lock file is named from the database's path with symlinks resolved,
  after `scan` creates the database file if it is missing, so a scan through a
  symlink to the database fails while another scan holds the lock (2026-10-14,
  https://git.eeqj.de/sneak/sfdupes/issues/101)

- on a terminal, a warning for the last file of a phase no longer erases the
  phase's finished bar: the bar is drawn again below the warning (2026-10-14,
  https://git.eeqj.de/sneak/sfdupes/issues/102)

- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
  `mtime_nsec`, and compares it at that resolution, so a same-size rewrite
  within the same second is re-hashed (2026-10-07,
  https://git.eeqj.de/sneak/sfdupes/issues/12)

- cut the narration from `TODO.md` Completed Steps and from the comments in
  `script/`, `Dockerfile` and `Dockerfile.lint` (gone since
  https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and
  merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)

- `make test-race` ran the test suite under the race detector in a cgo-enabled
  container, outside `make check` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/18). Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
  under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
  `make test-race` is gone

- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
  rather than serve the gates from cache (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/39). Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
  `REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
  `script/` pass a new `CACHEBUST` value, so theirs always run

- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks
  it; all Markdown reformatted (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/19)

- `script/lint` writes no image, so a run no longer leaves an untagged one
  behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)

- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
  the `report` and `trees` summary lines, and every subcommand going through
  `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)

- `.golangci.yml` replaced with the current canonical copy, which uses
  `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/26)

- a test fails when either `hashWorker` cancellation check in `scan.go` is
  removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)

- a database path holding `?`, `#` or `%` opens exactly the file it names
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)

- `scan` rejects `--workers` below 1 as a usage error instead of running
  single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)

- a test fails when either walk cancellation check in `scan.go` is removed
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)

- test that `scan` refuses a database with another schema version (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/64)

- correct four inaccurate comments in `cancel_test.go` and rename
  `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/33)

- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/17)

- `scan` creates the schema in one transaction; a version-0 database with a
  `files` table is refused with a clear schema-version error (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/11)

- README documents install, Docker, a daily cron scan and how to read and check
  the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)

- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home
  and copied the sources with `--chown`, so no `chown -R` walked them
  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
  nothing changes owner: the build stage only compiles, as root, and the tests
  run as `nobody` in the `test` phase

- `--version` prints `sfdupes VERSION` to stdout; README documents it and
  `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)

- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
  deletes nothing more, exits 1 (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/5)

- `report` and `trees` stream the records instead of holding them all in memory;
  the schema gains the `files_signature` index (2026-10-04,
  https://git.eeqj.de/sneak/sfdupes/issues/14)

- progress prints at once on a non-terminal, uses a real terminal test, and
  prints warnings through a spinner instead of racing its redraw (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/13)

- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
  the records beneath them (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/9)

- `scan` holds a lock on a lock file beside the database for its whole run, so a
  second `scan` fails at once with exit 1 (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/53)

- test stdout write failures in `report` and `trees`; README states that
  `| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
  (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)

- `report` and `trees` open the database read-only, and `scan` leaves it out of
  WAL mode, so reading needs only read access (2026-10-03, closes
  https://git.eeqj.de/sneak/sfdupes/issues/8)

- escape tabs, newlines, carriage returns and backslashes in report, trees and
  warning paths; the root directory's path is `/` (2026-10-03,
  https://git.eeqj.de/sneak/sfdupes/issues/7)

- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
  (2026-10-02, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
  without `.git/config`; the build stage stamps the `VERSION` build argument,
  else `git describe --tags --always`, and fails if the context carries `.git`
  and the version is still empty, `dev` or `unknown`. CI checks out the full
  history (`fetch-depth: 0`) so it stamps the same value as `make build`.

- replace the 1 KiB end-window sampling with the head/tail plus content-hash
  ladder (2026-09-22, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
  documents every rung. A file under 10 MiB is hashed in full, and its `head`,
  `tail` and `content` all hold that hash. A larger file gets only its 64 KiB
  `head` and `tail` in the hash phase; the content phase, after the update
  phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
  MiB samples at or above) only when its size, `head` and `tail` match another
  record's from this scan or an earlier one, and never reads a file gone or
  changed since its record was written. `report` and `trees` leave out any
  record without a `content` hash. The `content` column is part of the version 1
  schema.

- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
  `.dockerignore` (2026-09-21, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/22)

- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
  `Dockerfile.lint` (2026-08-10, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy
  `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
  without the false `(Debian-based)` note or the tag. Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
  holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and
  `script/verify-lint-image-pin`, which compared the two pins, are gone.

- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
  branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
  owner ruling the linter is never installed on a host, and
  `golangci-lint config verify` runs before `golangci-lint run`.
  `script/bootstrap` stopped installing or pinning the linter, and
  `script/verify-linter-pin` was retired. Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
  `Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build
  stage depends on through `COPY --from=lint /src/go.sum /dev/null`;
  `Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
  an image build may run docker, so the phase calls `golangci-lint` directly.

- install the Docker build stage's prerequisites by running `script/bootstrap`
  instead of `apk add --no-cache make` inline (2026-08-09, branch
  `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42),
  with `script/verify-linter-pin` failing the build unless the linter copied
  from the lint stage was the version `script/bootstrap` pinned. Builds then
  took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as
  https://git.eeqj.de/sneak/sfdupes/issues/43. Since
  https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
  `make` with `apk add --no-cache` and only compiles; the `lint` and `test`
  phases are the gates
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
  `script/docker` cannot report a green they did not earn (2026-08-09, branch
  `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
  `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
  served them from cache and the build exited 0 having run nothing. The fix was
  a `CHECK_EPOCH` build argument; since
  https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
  passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
  declares first. Run as root, the tests fail
  `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
  the test relies on, so the `test` phase runs them as `nobody`
- check the installed golangci-lint version in `script/bootstrap` instead of
  only its presence (2026-08-09, branch `bootstrap-version-check`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
  `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
  any installed version that is not the pin — older, newer, absent or
  unparseable — is reinstalled. `go install` writes into `GOBIN` (or
  `GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
  bootstrap re-reads the effective version after installing and, on a mismatch,
  prints both paths and both versions and exits non-zero; it does not reorder
  `PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
  bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
  presence-only checks. Verified by bootstrapping this host from v2.10.1 to
  v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
  covering a thirteen-input version-parse matrix, a shadowed install that must
  exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
  binary that must hit the timeout; `make check` and `make lint` are clean at
  v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch
  `hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
  pool is now an owned, context-aware `hashPool`: every blocking send in the
  feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
  out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
  `results` until the last goroutine has exited — draining is what frees a
  worker already parked on a send. `ctx` is threaded from `cmd.Context()`
  through `runScan`, `syncScan`, both worker pools and the whole database layer,
  as the first parameter everywhere. The walk pool gets the same treatment plus
  a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
  census, and every file it never reached looks vanished to the update phase.
  That phase's own `BeginTx` also fails on the cancelled context before deleting
  anything, but the guard is the barrier that still holds once an interrupted
  scan may commit what it has. Tests drive `run(scan)` against a database whose
  insert trigger aborts and assert that the scan fails instead of hanging and
  that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
  cancel a scan part-way through the walk, deterministically, by counting its
  own consultations of `ctx.Done()`, and assert that it stops at the guard
  holding a partial census and a still-populated record index, with every record
  intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
  `feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
  branches of both pools
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
  `db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
  `fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
  with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
  `runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
  errors instead. The single exit point is `run` in `main.go`: it maps a
  `fatalError` (anything a subcommand returned) to exit 1 and cobra's own
  argument and flag errors to exit 2, which keeps a runtime failure from being
  reported as a usage error or printing the usage text. New `main_test.go`
  drives the CLI in-process and asserts the exit codes from README §Error
  handling plus the stdout/stderr split, including that a fatal error raised
  after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
  `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
  `golangci-v2.12.2`, merged as `38a01bd`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
  `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
  replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
  `funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
  the v2 schema, so they are actually applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
  like the other managed repos (2026-07-26, commit `3abeacf`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
  exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
  `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
  and every Makefile target is now a thin shim over them
- make the binary the default Make target (2026-07-24, branch
  `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
  `check` plus `build`); `make build` remains as an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
  `scan-wide-phases`): all operands seed the shared walk pool and every pass
  runs once over the whole scan, so totals and ETAs are scan-global; the
  per-operand walk/hash/update cycles and their stderr announcements are gone;
  the update pass commits in batched transactions — the filesystem is
  authoritative and the database an eventually-consistent reflection, so
  scan-level atomicity is not required
- split the stat pass back out of the walk (2026-07-24, branch
  `parallel-phases`): phases are strictly sequential again — walk, stat, hash,
  update per operand — with parallelism only inside each phase; the walk
  enumerates paths with per-directory workers and the stat pass lstats them with
  per-file workers, restoring the exact total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24, branch
  `scan-operand-progress`): with per-operand walk/hash/update cycles, a
  multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
  whole run's
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
  goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
  4h on a ZFS server); it is now a per-directory worker-pool traversal that
  records size/mtime during the walk (folding away the separate stat pass,
  halving metadata I/O), and each `PATH` operand commits in its own transaction
  so an interrupted scan keeps completed operands

- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
  now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
  disabled) keyed by absolute path that survives between runs — a rescan hashes
  only new or changed files (by mtime/size), deletes records for files vanished
  from under the scanned operands, and leaves records outside them untouched, so
  `scan` can be cronned daily; `report` and `trees` read the database (no
  positional arguments) instead of a scan stream. Database at
  `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
  journaling plus a single-transaction update keep a report run during a scan
  safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
  and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
  `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
  `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
  boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch
  `repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on `main`
  (2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history)

# Future Steps

- possible later features (explicitly out of scope per README): full-content
  verification of candidates, removal-script helpers

# Repo Policy Compliance

Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
formatting commit is needed.

- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
      untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
- [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed from
      house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
      `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
      `docker`, `hooks`; `check` currently depends on `build`, which writes the
      binary (`make check` must not modify files)
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
      with a 30-second timeout and the conditional `-v` rerun pattern (suite
      covers parsing, grouping, digests, suppression, hashing, and the scan
      pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
      lint stage, build stage running `make check`
- [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
      action pinned by commit SHA)
- [x] README lacks required sections: Description first line
      (name/purpose/category/license/author), Getting Started, Rationale, TODO,
      License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the
      repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists)

Accepted divergences (no action):

- flat single-package layout with `.go` files in the repo root — fine for a
  small single-binary tool per the Go styleguide; the tracker audit agrees