sneak/sfdupes
1
0
Fork 0
Code Issues 23 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
73841c9989abce8121876634127b935a97eee195
sfdupes/TODO.md
sneak 73841c9989
All checks were successful
check / check (push) Successful in 57s
Details
Guarantee the database is closed on every fatal exit path (closes #4) 
fatalf called os.Exit(1), which does not run deferred functions, so
every defer db.Close() was dead on the fatal path: the SQLite WAL was
left uncheckpointed and the -wal/-shm sidecars were left for the next
process to recover. It also made those paths impossible to exercise
in-process.

fatalf is gone. runScan, runReport, runTrees, loadRecords and
resolveRoots return their errors, so the deferred close always runs,
and the only exit point is run() in main.go.

Mapping errors to exit codes needs care: cobra prints the error and
the command's usage text for anything RunE returns, and main mapped
every Execute() error to exit 2. A runtime failure is not a usage
problem, so the runE adapter silences both for the subcommands and
marks their errors fatalError; run() reports a fatalError as
"sfdupes: ..." on stderr and exits 1, and leaves everything else --
cobra's own argument, flag and unknown-command errors, which cobra has
already reported with its usage text -- on exit 2. The bare
"sfdupes" invocation still prints usage and exits 2.

Exit codes and message text are unchanged: 0 on success even with
per-file warnings, 1 fatal, 2 usage, per README section "Error
handling and exit codes". Everything on stdout is still data only.

main_test.go drives the CLI in-process and covers all three: a fatal
error raised after the database is open (a database with no files
table) closes it and leaves no -wal or -shm behind for scan, report
and trees; a nonexistent PATH operand is fatal, not usage, and prints
no usage text; the usage errors still exit 2; and a scan that skipped
an unreadable file still exits 0.
2026-08-09 02:27:07 +00:00

8.2 KiB
Raw Blame History

Workflow

  • take an issue from the 1.0.0 milestone on the tracker; work not yet on the tracker gets filed as an issue first
  • branch (from main)
  • do the work, with tests, in small focused commits
  • record it at the top of Completed Steps (TODO.md changes in the same commit as the work)
  • push the branch and open a PR whose title ends with (closes #N)
  • an independent review gates the merge; every finding is addressed or explicitly rebutted on the PR
  • merge to main once the review passes

Status

  • pre-1.0
  • the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues under the 1.0.0 milestone are what remains before the tag, and this file records history and process, not the queue

Next Step

  • take the next issue from the 1.0.0 milestone on the tracker: https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source of truth for what is left before 1.0.0. Individual issues are deliberately not restated here; a copy in this file drifts out of date the moment the tracker moves

Completed Steps

  • guarantee the database is closed on every fatal exit path (2026-08-09, branch db-close-on-fatal, closes #4): fatalf and its os.Exit(1) are gone, so the deferred db.Close() — and with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; runScan, runReport, runTrees, loadRecords and resolveRoots return errors instead. The single exit point is run in main.go: it maps a fatalError (anything a subcommand returned) to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a runtime failure from being reported as a usage error or printing the usage text. New main_test.go drives the CLI in-process and asserts the exit codes from README §Error handling plus the stdout/stderr split, including that a fatal error raised after the database is open leaves no -wal/-shm sidecar behind for scan, report or trees

  • update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch golangci-v2.12.2, merged as 38a01bd, closes #3): bumped the pinned linter in the Dockerfile lint stage and script/bootstrap from v2.12.1 to v2.12.2, and replaced .golangci.yml with the canonical file — the linter settings (lll, funlen, cyclop, dupl thresholds) now live under linters.settings per the v2 schema, so they are actually applied; no new lint findings surfaced

  • convert Makefile targets to scripts-to-rule-them-all script/ entrypoints like the other managed repos (2026-07-26, commit 3abeacf, closes #1): all 12 script/ entrypoints exist (bootstrap, setup, projectname, test, lint, fmt, fmt-check, check, docker, cibuild, precommit, install-precommit) and every Makefile target is now a thin shim over them, matching the other managed repos

  • make the binary the default Make target (2026-07-24, branch make-default-target): plain make now builds sfdupes (previously it ran check plus build); make build remains as an alias

  • scan-wide phases, concurrent operands, batched updates (2026-07-24, branch scan-wide-phases): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required

  • split the stat pass back out of the walk (2026-07-24, branch parallel-phases): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar

  • announce each operand on stderr before its passes (2026-07-24, branch scan-operand-progress): with per-operand walk/hash/update cycles, a multi-operand run (e.g. scan /srv/*) showed pass totals that looked like the whole run's — an operator watching operand 3 of 14 hash 300k files concluded 20M files were being skipped

  • parallel walk (2026-07-24, branch parallel-walk): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each PATH operand commits in its own transaction so an interrupted scan keeps completed operands

  • persistent scan database (2026-07-24, branch persistent-database): scan now maintains a SQLite database (modernc.org/sqlite, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so scan can be cronned daily; report and trees read the database (no positional arguments) instead of a scan stream. Database at /var/lib/sfdupes/db.sqlite, overridable via SFDUPES_DATABASE; WAL journaling plus a single-transaction update keep a report run during a scan safe

  • add the origin remote (git@git.eeqj.de:sneak/sfdupes.git), tag v0.0.1, and push main plus tags (2026-07-23)

  • scan CLI rework (2026-07-23, branch scan-required-paths): required PATH... operands via cobra flags replacing the /srv -root default; new -x/--one-file-system flag (GNU convention) to stop at filesystem boundaries, which are crossed by default

  • bring the repo into full policy compliance (2026-07-23, branch repo-policy-compliance; checklist below)

  • git init with README-only first commit; code baseline committed on main (2026-07-22)

  • implement scan, report, and trees subcommands (pre-git history)

Future Steps

  • possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers

Repo Policy Compliance

Audited 2026-07-22 against REPO_POLICIES.md (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed.

  • .gitignore missing — the compiled sfdupes binary and files.dat sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
  • .editorconfig missing
  • LICENSE missing and README has no License section (MIT assumed from house convention — user to confirm)
  • REPO_POLICIES.md missing from repo root
  • .golangci.yml missing (install canonical copy); code must then pass make lint (150 findings fixed; make lint is clean)
  • Makefile lacks required targets test, lint, fmt, fmt-check, docker, hooks; check currently depends on build, which writes the binary (make check must not modify files)
  • no tests — go test ./... has nothing to run; policy requires real tests with a 30-second timeout and the conditional -v rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage)
  • Dockerfile missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running make check
  • .dockerignore missing
  • .gitea/workflows/check.yml missing (docker build . on push, checkout action pinned by commit SHA)
  • README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author
  • README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI
  • pre-commit hook not installed (make hooks once the target exists)

Accepted divergences (no action):

  • flat single-package layout with .go files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees
  • go test runs without -race — the repo mandates CGO_ENABLED=0 (pure-Go builds) and the race detector requires cgo
Reference in New Issue View Git Blame Copy Permalink