sfdupes/TODO.md

Workflow

• take an issue from the 1.0.0 milestone on the tracker; work not yet on the tracker gets filed as an issue first
• branch from next
• do the work, with tests, in small focused commits
• record it at the top of Completed Steps (TODO.md changes in the same commit as the work)
• push the branch and open a PR against next whose title ends with (closes #N)
• an independent review gates each merge to next; every finding is addressed or explicitly rebutted on the PR
• only the owner merges next to main

Status

• pre-1.0
• the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues under the 1.0.0 milestone are what remains before the tag, and this file records history and process, not the queue

Next Step

• take the next issue from the 1.0.0 milestone on the tracker: https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source of truth for what is left before 1.0.0. Individual issues are deliberately not restated here; a copy in this file drifts out of date the moment the tracker moves

Completed Steps

• cut the narration from TODO.md Completed Steps and from the comments in script/ and both Dockerfiles; §Workflow now branches from and merges to next (2026-10-04, #49)

• make test-race runs the test suite under the race detector in a cgo-enabled container, outside make check (2026-10-04, #18)

• a bare docker build . fails with a message naming script/cibuild and script/docker instead of serving the gates from cache (2026-10-04, #39)

• make fmt and make fmt-check run prettier over all Markdown, in Docker, and CI checks it; all Markdown reformatted (2026-10-04, #19)

• script/lint writes no image, so a run no longer leaves an untagged one behind (2026-10-04, #48)

• tests cover a missing database, scan keeping stdout empty, its skip warning, the report and trees summary lines, and every subcommand going through runE (2026-10-04, #16)

• .golangci.yml replaced with the current canonical copy, which uses gomodguard_v2, so lint no longer prints a deprecation warning (2026-10-04, #26)

• a test fails when either hashWorker cancellation check in scan.go is removed (2026-10-04, #83)

• a database path holding ?, # or % opens exactly the file it names (2026-10-04, #55)

• scan rejects --workers below 1 as a usage error instead of running single-threaded (2026-10-04, #10)

• a test fails when either walk cancellation check in scan.go is removed (2026-10-04, #81)

• test that scan refuses a database with another schema version (2026-10-04, #64)

• correct four inaccurate comments in cancel_test.go and rename walkCancelInFlightDirs to walkCancelInFlightFiles (2026-10-04, #33)

• test the -x filesystem-boundary rules in subdirJob (2026-10-04, #17)

• scan creates the schema in one transaction; a version-0 database with a files table is refused with a clear schema-version error (2026-10-04, #11)

• README documents install, Docker, a daily cron scan and how to read and check the reports (2026-10-04, #54)

• the Dockerfile build stage keeps the Go module cache out of builder's home and copies the sources with --chown, so no chown -R walks them (2026-10-04, #43)

• --version prints sfdupes VERSION to stdout; README documents it and --help (2026-10-04, #15)

• scan stops cleanly on SIGINT or SIGTERM: commits what it has hashed, deletes nothing more, exits 1 (2026-10-04, #5)

• report and trees stream the records instead of holding them all in memory; the schema gains the files_signature index (2026-10-04, #14)

• progress prints at once on a non-terminal, uses a real terminal test, and prints warnings through a spinner instead of racing its redraw (2026-10-03, #13)

• warn about and skip symlink, socket, FIFO, device and .zfs operands, keeping the records beneath them (2026-10-03, #9)

• scan holds a lock on a lock file beside the database for its whole run, so a second scan fails at once with exit 1 (2026-10-03, #53)

• test stdout write failures in report and trees; README states that | head ends sfdupes by SIGPIPE and >&- writes to /dev/null (2026-10-03, #30)

• report and trees open the database read-only, and scan leaves it out of WAL mode, so reading needs only read access (2026-10-03, closes #8)

• escape tabs, newlines, carriage returns and backslashes in report, trees and warning paths; the root directory's path is / (2026-10-03, #7)

• stamp the git tag or short commit in a plain docker build . instead of dev (2026-10-02, branch next, closes #67): .dockerignore sends .git without .git/config; the build stage stamps the VERSION build argument, else git describe --tags --always, and fails if the context carries .git and the version is still empty, dev or unknown. CI checks out the full history (fetch-depth: 0) so it stamps the same value as make build.

• replace the 1 KiB end-window sampling with the head/tail plus content-hash ladder (2026-09-22, branch next, closes #61); README "Duplicate detection" documents every rung. A file under 10 MiB is hashed in full, and its head, tail and content all hold that hash. A larger file gets only its 64 KiB head and tail in the hash phase; the content phase, after the update phase, reads it for content (the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above) only when its size, head and tail match another record's from this scan or an earlier one, and never reads a file gone or changed since its record was written. report and trees leave out any record without a content hash. The content column is part of the version 1 schema.

• remove the dead files.dat references from Makefile, .gitignore and .dockerignore (2026-09-21, branch next, closes #22)

• fix the lint-image pin comments and FROM form in Dockerfile and Dockerfile.lint (2026-08-10, branch next, closes #25): both pins are now the policy # image:vX.Y.Z, YYYY-MM-DD comment over a bare FROM image@sha256:..., without the false (Debian-based) note or the tag; digest unchanged. script/verify-lint-image-pin still matches the tagless form, and a tag on one side only is caught as a plain mismatch.

• run all linting in Docker via Dockerfile.lint and script/lint (2026-08-10, branch next, closes #46): per the owner ruling the linter is never installed on a host. Dockerfile.lint copies the repo into the digest-pinned golangci/golangci-lint:v2.12.2 image and runs golangci-lint config verify and golangci-lint run as build steps; script/lint builds it. script/bootstrap no longer installs or pins the linter, and warns rather than fails when docker is absent; ENV PATH=/home/builder/go/bin:$PATH went with its go install. script/verify-linter-pin is retired; script/verify-lint-image-pin, a gate in both files, compares their two FROM lines and restates neither pin. Traps: an unchanged tree lets a lint build pass in under a second having run no linter, so every gate RUN references ARG CHECK_EPOCH (BuildKit hashes the expanded command) and script/lint passes "$(date +%s)-$$", the PID because two runs land in the same second easily. Nothing inside an image build may shell out to docker, so the Dockerfile lint stage calls golangci-lint directly and the build stage runs make test and make fmt-check instead of make check, through make because the Makefile's export CGO_ENABLED = 0 only reaches what it invokes. COPY --from=lint /src/go.sum /dev/null replaces the copied linter binary as the only edge making the build stage wait for lint; dropping it would end fail-fast linting under a still-green build. golangci-lint config verify, included per the ruling, validates from an embedded schema with no network call, but go mod download above the gates still needs the network on a cold cache. Verified: make lint green with no golangci-lint on PATH; two back-to-back script/lint runs on an untouched tree both ran the linter (27.7s and 28.7s in the lint step, COPY . . CACHED above); a planted unused variable failed script/lint, and failed make docker at [lint 9/9] with the build stage stopped at [builder 3/12]; the drift guard fails on a tag-only, a digest-only and an unreadable reference, naming both sides; under --network none config verify passes a valid config and rejects an invalid one; make docker green in 5m35s with all six gates run under one epoch (lint 37.6s, test 25.2s reporting ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%, not (cached)); in the builder image with the Go test cache off, --user 0:0 still fails TestScanHardlinkRunFailsTogether where the unprivileged user passes. Noted for follow-up, not fixed here: golangci-lint warns that gomodguard is deprecated since v2.12.0 in favour of gomodguard_v2.

• install the Docker build stage's prerequisites by running script/bootstrap instead of apk add --no-cache make inline (2026-08-09, branch dockerfile-bootstrap, closes #42): the stage copies script/ plus go.mod/go.sum and runs script/bootstrap, which ends in go mod download, so the separate call to it is gone. COPY --from=lint /usr/bin/golangci-lint stays and moves above the bootstrap layer: it is the only edge making this stage depend on the lint stage, so deleting it would end fail-fast linting silently. A new script/verify-linter-pin, run in the build stage before bootstrap, fails the build naming both versions unless that copied binary is the version script/bootstrap pins; a pin it cannot read is a hard failure, not a skip. $GOPATH/bin joins PATH, where bootstrap's go install lands. Everything added sits above ARG CHECK_EPOCH, and the chown and USER builder still precede make check. Verified: the guard fails the build with both versions named when the lint stage's linter is faked to another version, and passes an unmodified build; bootstrap runs clean under Alpine's sh and apk, finding the copied linter already at the pin; a second build served the bootstrap and dependency layers CACHED while both gates ran with a fresh epoch; a planted unused finding failed the build at the lint gate in 48.9s with the build stage's make check never starting; and the suite run in the image as --user 0:0 fails TestScanHardlinkRunFailsTogether, so the drop to the unprivileged user is still needed. That last check needs the Go test cache off: as root it first reported ok ... (cached), reusing the build-time result. Build times on a noisy shared host: 2m13s on an unchanged tree, 2m17s and 4m29s after a source change, 5m14s cold, which breaches the policy ceiling; chown -R builder:builder /src /home/builder walks the module cache and alone varied from 77s to 210s across those builds, and main measured 5m03s cold with a 209s chown. Filed as #43

• bust the Docker layer cache for the gate steps, so script/cibuild and script/docker cannot report a green they did not earn (2026-08-09, branch cibuild-cache-bust, closes #32): the Dockerfile copies the tree before its gates, so on an unchanged tree Docker served them from cache and the build exited 0 having run nothing. Both scripts now pass --build-arg CHECK_EPOCH="$(date +%s)". ARG is per stage and the gates span two stages, so it is declared in both; BuildKit hashes the expanded command, so each gate RUN echoes the epoch, which also logs it as evidence the layer ran. It sits below the dependency layers so they stay cached. Verified under BUILDKIT_PROGRESS=plain, each script run twice back to back on an unchanged tree: all three gates ran on all four runs with a fresh epoch (script/cibuild 78.8s then 61.1s; script/docker 61.1s then 53.4s), and thirteen steps were still served CACHED. With a planted unused finding the build failed at make lint in 36.1s and the build-stage make check never started. Run as root, the same image fails TestScanHardlinkRunFailsTogether, because root reads through the chmod(0) the test relies on, so the build stage must drop to the unprivileged builder user. Local fix only; propagating it to the canonical templates is sneak/prompts#26

• check the installed golangci-lint version in script/bootstrap instead of only its presence (2026-08-09, branch bootstrap-version-check, closes #24): the version lives only in GOLANGCI_LINT_VERSION, with the go install module ref derived from it, and any installed version that is not the pin — older, newer, absent or unparseable — is reinstalled. go install writes into GOBIN (or GOPATH/bin) while make lint runs the first golangci-lint on PATH, so bootstrap re-reads the effective version after installing and, on a mismatch, prints both paths and both versions and exits non-zero; it does not reorder PATH or delete anyone's binary. The --version call keeps its stderr and is bounded by timeout(1) where that exists. git, make and go keep presence-only checks. Verified by bootstrapping this host from v2.10.1 to v2.12.2 and again to a no-op, and by stub runs of the script under dash covering a thirteen-input version-parse matrix, a shadowed install that must exit non-zero, an install destination not on PATH, GOBIN set, and a wedged binary that must hit the timeout; make check and make lint are clean at v2.12.2, so v2.10.1 was not hiding any findings on main

• unwind the hash worker pool on the error path (2026-08-09, branch hash-pool-cleanup, closes #6): the pool is now an owned, context-aware hashPool: every blocking send in the feeder and the workers selects on ctx.Done(), jobs is closed on every path out, and hashPhase defers pool.stop(), which cancels and then drains results until the last goroutine has exited — draining is what frees a worker already parked on a send. ctx is threaded from cmd.Context() through runScan, syncScan, both worker pools and the whole database layer, as the first parameter everywhere. The walk pool gets the same treatment plus a ctx.Err() guard after the walk: a cancelled walk yields a partial size census, and every file it never reached looks vanished to the update phase. That phase's own BeginTx also fails on the cancelled context before deleting anything, but the guard is the barrier that still holds once an interrupted scan may commit what it has. Tests drive run(scan) against a database whose insert trigger aborts and assert that the scan fails instead of hanging and that runtime.NumGoroutine() polls back to its pre-scan baseline; others cancel a scan part-way through the walk, deterministically, by counting its own consultations of ctx.Done(), and assert that it stops at the guard holding a partial census and a still-populated record index, with every record intact. Direct tests of sendEvent, the walk workers, dispatchDirs, feedHashJobs, hashWorker and hashPhase cover the remaining cancellation branches of both pools

• guarantee the database is closed on every fatal exit path (2026-08-09, branch db-close-on-fatal, closes #4): fatalf and its os.Exit(1) are gone, so the deferred db.Close() — and with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; runScan, runReport, runTrees, loadRecords and resolveRoots return errors instead. The single exit point is run in main.go: it maps a fatalError (anything a subcommand returned) to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a runtime failure from being reported as a usage error or printing the usage text. New main_test.go drives the CLI in-process and asserts the exit codes from README §Error handling plus the stdout/stderr split, including that a fatal error raised after the database is open leaves no -wal/-shm sidecar behind for scan, report or trees

• update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch golangci-v2.12.2, merged as 38a01bd, closes #3): bumped the pinned linter in the Dockerfile lint stage and script/bootstrap from v2.12.1 to v2.12.2, and replaced .golangci.yml with the canonical file — the linter settings (lll, funlen, cyclop, dupl thresholds) now live under linters.settings per the v2 schema, so they are actually applied; no new lint findings surfaced

• convert Makefile targets to scripts-to-rule-them-all script/ entrypoints like the other managed repos (2026-07-26, commit 3abeacf, closes #1): all 12 script/ entrypoints exist (bootstrap, setup, projectname, test, lint, fmt, fmt-check, check, docker, cibuild, precommit, install-precommit) and every Makefile target is now a thin shim over them

• make the binary the default Make target (2026-07-24, branch make-default-target): plain make now builds sfdupes (previously it ran check plus build); make build remains as an alias

• scan-wide phases, concurrent operands, batched updates (2026-07-24, branch scan-wide-phases): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required

• split the stat pass back out of the walk (2026-07-24, branch parallel-phases): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar

• announce each operand on stderr before its passes (2026-07-24, branch scan-operand-progress): with per-operand walk/hash/update cycles, a multi-operand run (e.g. scan /srv/*) showed pass totals that looked like the whole run's

• parallel walk (2026-07-24, branch parallel-walk): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each PATH operand commits in its own transaction so an interrupted scan keeps completed operands

• persistent scan database (2026-07-24, branch persistent-database): scan now maintains a SQLite database (modernc.org/sqlite, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so scan can be cronned daily; report and trees read the database (no positional arguments) instead of a scan stream. Database at /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE; WAL journaling plus a single-transaction update keep a report run during a scan safe

• add the origin remote (git@git.eeqj.de:sneak/sfdupes.git), tag v0.0.1, and push main plus tags (2026-07-23)

• scan CLI rework (2026-07-23, branch scan-required-paths): required PATH... operands via cobra flags replacing the /srv -root default; new -x/--one-file-system flag (GNU convention) to stop at filesystem boundaries, which are crossed by default

• bring the repo into full policy compliance (2026-07-23, branch repo-policy-compliance; checklist below)

• git init with README-only first commit; code baseline committed on main (2026-07-22)

• implement scan, report, and trees subcommands (pre-git history)

Future Steps

• possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers

Repo Policy Compliance

Audited 2026-07-22 against REPO_POLICIES.md (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed.

• .gitignore missing — the compiled sfdupes binary and files.dat sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
• .editorconfig missing
• LICENSE missing and README has no License section (MIT assumed from house convention — user to confirm)
• REPO_POLICIES.md missing from repo root
• .golangci.yml missing (install canonical copy); code must then pass make lint (150 findings fixed; make lint is clean)
• Makefile lacks required targets test, lint, fmt, fmt-check, docker, hooks; check currently depends on build, which writes the binary (make check must not modify files)
• no tests — go test ./... has nothing to run; policy requires real tests with a 30-second timeout and the conditional -v rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage)
• Dockerfile missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running make check
• .dockerignore missing
• .gitea/workflows/check.yml missing (docker build . on push, checkout action pinned by commit SHA)
• README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author
• README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI
• pre-commit hook not installed (make hooks once the target exists)

Accepted divergences (no action):

• flat single-package layout with .go files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees
• make test runs without -race — the repo mandates CGO_ENABLED=0 (pure-Go builds) and the race detector requires cgo, so the detector runs in a separate cgo-enabled container, make test-race, which is not part of make check