sfdupes/TODO.md

Workflow

• take an issue from the 1.0.0 milestone on the tracker; work not yet on the tracker gets filed as an issue first
• branch from next
• do the work, with tests, in small focused commits
• record it at the top of Completed Steps (TODO.md changes in the same commit as the work)
• push the branch and open a PR against next whose title ends with (closes #N)
• an independent review gates each merge to next; every finding is addressed or explicitly rebutted on the PR
• only the owner merges next to main

Status

• pre-1.0
• the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues under the 1.0.0 milestone are what remains before the tag, and this file records history and process, not the queue

Next Step

• take the next issue from the 1.0.0 milestone on the tracker: https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source of truth for what is left before 1.0.0. Individual issues are deliberately not restated here; a copy in this file drifts out of date the moment the tracker moves

Completed Steps

• re-vendor the canonical files and model scripts from sneak/prompts next at c55a0cb: golangci-lint v2.14.0; lint and test are phases of the Dockerfile that write no image, and make test runs the suite under the race detector, so Dockerfile.lint, script/verify-lint-image-pin and make test-race are gone; every docker build in script/ passes --no-cache; prettier runs on the host, from the node and yarn script/bootstrap installs, so the prettier and markdown stages are gone and the build stage installs git and make itself; a new push cancels the workflow's older run on the same branch, and a run stops after 20 minutes; .claude/settings.json is deleted (2026-10-08, #95)

• scan records mtime to the nanosecond, as whole seconds in mtime plus mtime_nsec, and compares it at that resolution, so a same-size rewrite within the same second is re-hashed (2026-10-07, #12)

• cut the narration from TODO.md Completed Steps and from the comments in script/, Dockerfile and Dockerfile.lint (gone since #95); §Workflow now branches from and merges to next (2026-10-04, #49)

• make test-race ran the test suite under the race detector in a cgo-enabled container, outside make check (2026-10-04, #18). Since #95 make test itself runs the suite under the race detector, in the Dockerfile's Debian-based test phase, and make test-race is gone

• a bare docker build . failed, naming script/cibuild and script/docker, rather than serve the gates from cache (2026-10-04, #39). Since #95 a bare build succeeds, as REPO_POLICIES.md requires, and may serve the gates from cache; the builds in script/ pass --no-cache, so theirs always run

• make fmt and make fmt-check run prettier over all Markdown, and CI checks it; all Markdown reformatted (2026-10-04, #19)

• script/lint writes no image, so a run no longer leaves an untagged one behind (2026-10-04, #48)

• tests cover a missing database, scan keeping stdout empty, its skip warning, the report and trees summary lines, and every subcommand going through runE (2026-10-04, #16)

• .golangci.yml replaced with the current canonical copy, which uses gomodguard_v2, so lint no longer prints a deprecation warning (2026-10-04, #26)

• a test fails when either hashWorker cancellation check in scan.go is removed (2026-10-04, #83)

• a database path holding ?, # or % opens exactly the file it names (2026-10-04, #55)

• scan rejects --workers below 1 as a usage error instead of running single-threaded (2026-10-04, #10)

• a test fails when either walk cancellation check in scan.go is removed (2026-10-04, #81)

• test that scan refuses a database with another schema version (2026-10-04, #64)

• correct four inaccurate comments in cancel_test.go and rename walkCancelInFlightDirs to walkCancelInFlightFiles (2026-10-04, #33)

• test the -x filesystem-boundary rules in subdirJob (2026-10-04, #17)

• scan creates the schema in one transaction; a version-0 database with a files table is refused with a clear schema-version error (2026-10-04, #11)

• README documents install, Docker, a daily cron scan and how to read and check the reports (2026-10-04, #54)

• the Dockerfile build stage kept the Go module cache out of builder's home and copied the sources with --chown, so no chown -R walked them (2026-10-04, #43). Since #95 there is no builder user and nothing changes owner: the build stage only compiles, as root, and the tests run as nobody in the test phase

• --version prints sfdupes VERSION to stdout; README documents it and --help (2026-10-04, #15)

• scan stops cleanly on SIGINT or SIGTERM: commits what it has hashed, deletes nothing more, exits 1 (2026-10-04, #5)

• report and trees stream the records instead of holding them all in memory; the schema gains the files_signature index (2026-10-04, #14)

• progress prints at once on a non-terminal, uses a real terminal test, and prints warnings through a spinner instead of racing its redraw (2026-10-03, #13)

• warn about and skip symlink, socket, FIFO, device and .zfs operands, keeping the records beneath them (2026-10-03, #9)

• scan holds a lock on a lock file beside the database for its whole run, so a second scan fails at once with exit 1 (2026-10-03, #53)

• test stdout write failures in report and trees; README states that | head ends sfdupes by SIGPIPE and >&- writes to /dev/null (2026-10-03, #30)

• report and trees open the database read-only, and scan leaves it out of WAL mode, so reading needs only read access (2026-10-03, closes #8)

• escape tabs, newlines, carriage returns and backslashes in report, trees and warning paths; the root directory's path is / (2026-10-03, #7)

• stamp the git tag or short commit in a plain docker build . instead of dev (2026-10-02, branch next, closes #67): .dockerignore sends .git without .git/config; the build stage stamps the VERSION build argument, else git describe --tags --always, and fails if the context carries .git and the version is still empty, dev or unknown. CI checks out the full history (fetch-depth: 0) so it stamps the same value as make build.

• replace the 1 KiB end-window sampling with the head/tail plus content-hash ladder (2026-09-22, branch next, closes #61); README "Duplicate detection" documents every rung. A file under 10 MiB is hashed in full, and its head, tail and content all hold that hash. A larger file gets only its 64 KiB head and tail in the hash phase; the content phase, after the update phase, reads it for content (the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above) only when its size, head and tail match another record's from this scan or an earlier one, and never reads a file gone or changed since its record was written. report and trees leave out any record without a content hash. The content column is part of the version 1 schema.

• remove the dead files.dat references from Makefile, .gitignore and .dockerignore (2026-09-21, branch next, closes #22)

• fix the lint-image pin comments and FROM form in Dockerfile and Dockerfile.lint (2026-08-10, branch next, closes #25): both pins became the policy # image:vX.Y.Z, YYYY-MM-DD comment over a bare FROM image@sha256:..., without the false (Debian-based) note or the tag. Since #95 the Dockerfile's lint phase holds the only golangci-lint pin, in that form, so Dockerfile.lint and script/verify-lint-image-pin, which compared the two pins, are gone.

• run all linting in Docker via Dockerfile.lint and script/lint (2026-08-10, branch next, closes #46): per the owner ruling the linter is never installed on a host, and golangci-lint config verify runs before golangci-lint run. script/bootstrap stopped installing or pinning the linter, and script/verify-linter-pin was retired. Since #95 both commands run in the Dockerfile's lint phase, which script/lint builds alone and the build stage depends on through COPY --from=lint /src/go.sum /dev/null; Dockerfile.lint and script/verify-lint-image-pin are gone. Nothing inside an image build may run docker, so the phase calls golangci-lint directly.

• install the Docker build stage's prerequisites by running script/bootstrap instead of apk add --no-cache make inline (2026-08-09, branch dockerfile-bootstrap, closes #42), with script/verify-linter-pin failing the build unless the linter copied from the lint stage was the version script/bootstrap pinned. Builds then took up to 5m14s cold, mostly in a chown -R of the module cache, filed as #43. Since #95 the build stage installs git and make with apk add --no-cache and only compiles; the lint and test phases are the gates

• bust the Docker layer cache for the gate steps, so script/cibuild and script/docker cannot report a green they did not earn (2026-08-09, branch cibuild-cache-bust, closes #32): the Dockerfile copies the tree before its gates, so on an unchanged tree Docker served them from cache and the build exited 0 having run nothing. The fix was a CHECK_EPOCH build argument; since #95 every docker build in script/ passes --no-cache instead. Run as root, the tests fail TestScanHardlinkRunFailsTogether, because root reads through the chmod(0) the test relies on, so the test phase runs them as nobody

• check the installed golangci-lint version in script/bootstrap instead of only its presence (2026-08-09, branch bootstrap-version-check, closes #24): the version lives only in GOLANGCI_LINT_VERSION, with the go install module ref derived from it, and any installed version that is not the pin — older, newer, absent or unparseable — is reinstalled. go install writes into GOBIN (or GOPATH/bin) while make lint runs the first golangci-lint on PATH, so bootstrap re-reads the effective version after installing and, on a mismatch, prints both paths and both versions and exits non-zero; it does not reorder PATH or delete anyone's binary. The --version call keeps its stderr and is bounded by timeout(1) where that exists. git, make and go keep presence-only checks. Verified by bootstrapping this host from v2.10.1 to v2.12.2 and again to a no-op, and by stub runs of the script under dash covering a thirteen-input version-parse matrix, a shadowed install that must exit non-zero, an install destination not on PATH, GOBIN set, and a wedged binary that must hit the timeout; make check and make lint are clean at v2.12.2, so v2.10.1 was not hiding any findings on main

• unwind the hash worker pool on the error path (2026-08-09, branch hash-pool-cleanup, closes #6): the pool is now an owned, context-aware hashPool: every blocking send in the feeder and the workers selects on ctx.Done(), jobs is closed on every path out, and hashPhase defers pool.stop(), which cancels and then drains results until the last goroutine has exited — draining is what frees a worker already parked on a send. ctx is threaded from cmd.Context() through runScan, syncScan, both worker pools and the whole database layer, as the first parameter everywhere. The walk pool gets the same treatment plus a ctx.Err() guard after the walk: a cancelled walk yields a partial size census, and every file it never reached looks vanished to the update phase. That phase's own BeginTx also fails on the cancelled context before deleting anything, but the guard is the barrier that still holds once an interrupted scan may commit what it has. Tests drive run(scan) against a database whose insert trigger aborts and assert that the scan fails instead of hanging and that runtime.NumGoroutine() polls back to its pre-scan baseline; others cancel a scan part-way through the walk, deterministically, by counting its own consultations of ctx.Done(), and assert that it stops at the guard holding a partial census and a still-populated record index, with every record intact. Direct tests of sendEvent, the walk workers, dispatchDirs, feedHashJobs, hashWorker and hashPhase cover the remaining cancellation branches of both pools

• guarantee the database is closed on every fatal exit path (2026-08-09, branch db-close-on-fatal, closes #4): fatalf and its os.Exit(1) are gone, so the deferred db.Close() — and with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; runScan, runReport, runTrees, loadRecords and resolveRoots return errors instead. The single exit point is run in main.go: it maps a fatalError (anything a subcommand returned) to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a runtime failure from being reported as a usage error or printing the usage text. New main_test.go drives the CLI in-process and asserts the exit codes from README §Error handling plus the stdout/stderr split, including that a fatal error raised after the database is open leaves no -wal/-shm sidecar behind for scan, report or trees

• update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch golangci-v2.12.2, merged as 38a01bd, closes #3): bumped the pinned linter in the Dockerfile lint stage and script/bootstrap from v2.12.1 to v2.12.2, and replaced .golangci.yml with the canonical file — the linter settings (lll, funlen, cyclop, dupl thresholds) now live under linters.settings per the v2 schema, so they are actually applied; no new lint findings surfaced

• convert Makefile targets to scripts-to-rule-them-all script/ entrypoints like the other managed repos (2026-07-26, commit 3abeacf, closes #1): all 12 script/ entrypoints exist (bootstrap, setup, projectname, test, lint, fmt, fmt-check, check, docker, cibuild, precommit, install-precommit) and every Makefile target is now a thin shim over them

• make the binary the default Make target (2026-07-24, branch make-default-target): plain make now builds sfdupes (previously it ran check plus build); make build remains as an alias

• scan-wide phases, concurrent operands, batched updates (2026-07-24, branch scan-wide-phases): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required

• split the stat pass back out of the walk (2026-07-24, branch parallel-phases): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar

• announce each operand on stderr before its passes (2026-07-24, branch scan-operand-progress): with per-operand walk/hash/update cycles, a multi-operand run (e.g. scan /srv/*) showed pass totals that looked like the whole run's

• parallel walk (2026-07-24, branch parallel-walk): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each PATH operand commits in its own transaction so an interrupted scan keeps completed operands

• persistent scan database (2026-07-24, branch persistent-database): scan now maintains a SQLite database (modernc.org/sqlite, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so scan can be cronned daily; report and trees read the database (no positional arguments) instead of a scan stream. Database at /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE; WAL journaling plus a single-transaction update keep a report run during a scan safe

• add the origin remote (git@git.eeqj.de:sneak/sfdupes.git), tag v0.0.1, and push main plus tags (2026-07-23)

• scan CLI rework (2026-07-23, branch scan-required-paths): required PATH... operands via cobra flags replacing the /srv -root default; new -x/--one-file-system flag (GNU convention) to stop at filesystem boundaries, which are crossed by default

• bring the repo into full policy compliance (2026-07-23, branch repo-policy-compliance; checklist below)

• git init with README-only first commit; code baseline committed on main (2026-07-22)

• implement scan, report, and trees subcommands (pre-git history)

Future Steps

• possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers

Repo Policy Compliance

Audited 2026-07-22 against REPO_POLICIES.md (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed.

• .gitignore missing — the compiled sfdupes binary and files.dat sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
• .editorconfig missing
• LICENSE missing and README has no License section (MIT assumed from house convention — user to confirm)
• REPO_POLICIES.md missing from repo root
• .golangci.yml missing (install canonical copy); code must then pass make lint (150 findings fixed; make lint is clean)
• Makefile lacks required targets test, lint, fmt, fmt-check, docker, hooks; check currently depends on build, which writes the binary (make check must not modify files)
• no tests — go test ./... has nothing to run; policy requires real tests with a 30-second timeout and the conditional -v rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage)
• Dockerfile missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running make check
• .dockerignore missing
• .gitea/workflows/check.yml missing (docker build . on push, checkout action pinned by commit SHA)
• README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author
• README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI
• pre-commit hook not installed (make hooks once the target exists)

Accepted divergences (no action):

• flat single-package layout with .go files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees