sfdupes/TODO.md

Workflow

• take an issue from the 1.0.0 milestone on the tracker; work not yet on the tracker gets filed as an issue first
• branch (from main)
• do the work, with tests, in small focused commits
• record it at the top of Completed Steps (TODO.md changes in the same commit as the work)
• push the branch and open a PR whose title ends with (closes #N)
• an independent review gates the merge; every finding is addressed or explicitly rebutted on the PR
• merge to main once the review passes

Status

• pre-1.0
• the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues under the 1.0.0 milestone are what remains before the tag, and this file records history and process, not the queue

Next Step

• take the next issue from the 1.0.0 milestone on the tracker: https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source of truth for what is left before 1.0.0. Individual issues are deliberately not restated here; a copy in this file drifts out of date the moment the tracker moves

Completed Steps

• update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch golangci-v2.12.2, merged as 38a01bd, closes #3): bumped the pinned linter in the Dockerfile lint stage and script/bootstrap from v2.12.1 to v2.12.2, and replaced .golangci.yml with the canonical file — the linter settings (lll, funlen, cyclop, dupl thresholds) now live under linters.settings per the v2 schema, so they are actually applied; no new lint findings surfaced

• convert Makefile targets to scripts-to-rule-them-all script/ entrypoints like the other managed repos (2026-07-26, commit 3abeacf, closes #1): all 12 script/ entrypoints exist (bootstrap, setup, projectname, test, lint, fmt, fmt-check, check, docker, cibuild, precommit, install-precommit) and every Makefile target is now a thin shim over them, matching the other managed repos

• make the binary the default Make target (2026-07-24, branch make-default-target): plain make now builds sfdupes (previously it ran check plus build); make build remains as an alias

• scan-wide phases, concurrent operands, batched updates (2026-07-24, branch scan-wide-phases): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required

• split the stat pass back out of the walk (2026-07-24, branch parallel-phases): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar

• announce each operand on stderr before its passes (2026-07-24, branch scan-operand-progress): with per-operand walk/hash/update cycles, a multi-operand run (e.g. scan /srv/*) showed pass totals that looked like the whole run's — an operator watching operand 3 of 14 hash 300k files concluded 20M files were being skipped

• parallel walk (2026-07-24, branch parallel-walk): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each PATH operand commits in its own transaction so an interrupted scan keeps completed operands

• persistent scan database (2026-07-24, branch persistent-database): scan now maintains a SQLite database (modernc.org/sqlite, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so scan can be cronned daily; report and trees read the database (no positional arguments) instead of a scan stream. Database at /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE; WAL journaling plus a single-transaction update keep a report run during a scan safe

• add the origin remote (git@git.eeqj.de:sneak/sfdupes.git), tag v0.0.1, and push main plus tags (2026-07-23)

• scan CLI rework (2026-07-23, branch scan-required-paths): required PATH... operands via cobra flags replacing the /srv -root default; new -x/--one-file-system flag (GNU convention) to stop at filesystem boundaries, which are crossed by default

• bring the repo into full policy compliance (2026-07-23, branch repo-policy-compliance; checklist below)

• git init with README-only first commit; code baseline committed on main (2026-07-22)

• implement scan, report, and trees subcommands (pre-git history)

Future Steps

• possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers

Repo Policy Compliance

Audited 2026-07-22 against REPO_POLICIES.md (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed.

• .gitignore missing — the compiled sfdupes binary and files.dat sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
• .editorconfig missing
• LICENSE missing and README has no License section (MIT assumed from house convention — user to confirm)
• REPO_POLICIES.md missing from repo root
• .golangci.yml missing (install canonical copy); code must then pass make lint (150 findings fixed; make lint is clean)
• Makefile lacks required targets test, lint, fmt, fmt-check, docker, hooks; check currently depends on build, which writes the binary (make check must not modify files)
• no tests — go test ./... has nothing to run; policy requires real tests with a 30-second timeout and the conditional -v rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage)
• Dockerfile missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running make check
• .dockerignore missing
• .gitea/workflows/check.yml missing (docker build . on push, checkout action pinned by commit SHA)
• README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author
• README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI
• pre-commit hook not installed (make hooks once the target exists)

Accepted divergences (no action):

• flat single-package layout with .go files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees
• go test runs without -race — the repo mandates CGO_ENABLED=0 (pure-Go builds) and the race detector requires cgo