sfdupes/TODO.md

Workflow

• branch (from main)
• do the work in Next Step
• move Next Step to the top of Completed Steps
• move the top item of Future Steps into Next Step
• commit (TODO.md changes in the same commit as the work)
• merge to main if the branch is not protected, otherwise open a PR
• push

Status

• pre-1.0

Next Step

• convert Makefile targets to scripts-to-rule-them-all script/ entrypoints like the other managed repos

Completed Steps

• scan-wide phases, concurrent operands, batched updates (2026-07-24, branch scan-wide-phases): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required

• split the stat pass back out of the walk (2026-07-24, branch parallel-phases): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar

• announce each operand on stderr before its passes (2026-07-24, branch scan-operand-progress): with per-operand walk/hash/update cycles, a multi-operand run (e.g. scan /srv/*) showed pass totals that looked like the whole run's — an operator watching operand 3 of 14 hash 300k files concluded 20M files were being skipped

• parallel walk (2026-07-24, branch parallel-walk): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each PATH operand commits in its own transaction so an interrupted scan keeps completed operands

• persistent scan database (2026-07-24, branch persistent-database): scan now maintains a SQLite database (modernc.org/sqlite, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so scan can be cronned daily; report and trees read the database (no positional arguments) instead of a scan stream. Database at /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE; WAL journaling plus a single-transaction update keep a report run during a scan safe

• add the origin remote (git@git.eeqj.de:sneak/sfdupes.git), tag v0.0.1, and push main plus tags (2026-07-23)

• scan CLI rework (2026-07-23, branch scan-required-paths): required PATH... operands via cobra flags replacing the /srv -root default; new -x/--one-file-system flag (GNU convention) to stop at filesystem boundaries, which are crossed by default

• bring the repo into full policy compliance (2026-07-23, branch repo-policy-compliance; checklist below)

• git init with README-only first commit; code baseline committed on main (2026-07-22)

• implement scan, report, and trees subcommands (pre-git history)

Future Steps

• possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers

Repo Policy Compliance

Audited 2026-07-22 against REPO_POLICIES.md (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed.

• .gitignore missing — the compiled sfdupes binary and files.dat sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
• .editorconfig missing
• LICENSE missing and README has no License section (MIT assumed from house convention — user to confirm)
• REPO_POLICIES.md missing from repo root
• .golangci.yml missing (install canonical copy); code must then pass make lint (150 findings fixed; make lint is clean)
• Makefile lacks required targets test, lint, fmt, fmt-check, docker, hooks; check currently depends on build, which writes the binary (make check must not modify files)
• no tests — go test ./... has nothing to run; policy requires real tests with a 30-second timeout and the conditional -v rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage)
• Dockerfile missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running make check
• .dockerignore missing
• .gitea/workflows/check.yml missing (docker build . on push, checkout action pinned by commit SHA)
• README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author
• README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI
• pre-commit hook not installed (make hooks once the target exists)

Accepted divergences (no action):

• flat single-package layout with .go files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees
• go test runs without -race — the repo mandates CGO_ENABLED=0 (pure-Go builds) and the race detector requires cgo