sneak
73841c9989
All checks were successful
check / check (push) Successful in 57s
fatalf called os.Exit(1), which does not run deferred functions, so every defer db.Close() was dead on the fatal path: the SQLite WAL was left uncheckpointed and the -wal/-shm sidecars were left for the next process to recover. It also made those paths impossible to exercise in-process. fatalf is gone. runScan, runReport, runTrees, loadRecords and resolveRoots return their errors, so the deferred close always runs, and the only exit point is run() in main.go. Mapping errors to exit codes needs care: cobra prints the error and the command's usage text for anything RunE returns, and main mapped every Execute() error to exit 2. A runtime failure is not a usage problem, so the runE adapter silences both for the subcommands and marks their errors fatalError; run() reports a fatalError as "sfdupes: ..." on stderr and exits 1, and leaves everything else -- cobra's own argument, flag and unknown-command errors, which cobra has already reported with its usage text -- on exit 2. The bare "sfdupes" invocation still prints usage and exits 2. Exit codes and message text are unchanged: 0 on success even with per-file warnings, 1 fatal, 2 usage, per README section "Error handling and exit codes". Everything on stdout is still data only. main_test.go drives the CLI in-process and covers all three: a fatal error raised after the database is open (a database with no files table) closes it and leaves no -wal or -shm behind for scan, report and trees; a nonexistent PATH operand is fatal, not usage, and prints no usage text; the usage errors still exit 2; and a scan that skipped an unreadable file still exits 0.
163 lines
8.2 KiB
Markdown
163 lines
8.2 KiB
Markdown
# Workflow
|
|
|
|
- take an issue from the `1.0.0` milestone on the tracker; work not
|
|
yet on the tracker gets filed as an issue first
|
|
- branch (from `main`)
|
|
- do the work, with tests, in small focused commits
|
|
- record it at the top of Completed Steps (`TODO.md` changes in the
|
|
same commit as the work)
|
|
- push the branch and open a PR whose title ends with
|
|
` (closes #N)`
|
|
- an independent review gates the merge; every finding is addressed
|
|
or explicitly rebutted on the PR
|
|
- merge to `main` once the review passes
|
|
|
|
# Status
|
|
|
|
- pre-1.0
|
|
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
|
|
open issues under the `1.0.0` milestone are what remains before
|
|
the tag, and this file records history and process, not the queue
|
|
|
|
# Next Step
|
|
|
|
- take the next issue from the `1.0.0` milestone on the tracker:
|
|
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
|
|
the source of truth for what is left before 1.0.0. Individual
|
|
issues are deliberately not restated here; a copy in this file
|
|
drifts out of date the moment the tracker moves
|
|
|
|
# Completed Steps
|
|
|
|
- guarantee the database is closed on every fatal exit path
|
|
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
|
|
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
|
|
it the SQLite WAL checkpoint — now actually runs when a subcommand
|
|
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
|
|
`resolveRoots` return errors instead. The single exit point is `run`
|
|
in `main.go`: it maps a `fatalError` (anything a subcommand
|
|
returned) to exit 1 and cobra's own argument and flag errors to exit
|
|
2, which keeps a runtime failure from being reported as a usage
|
|
error or printing the usage text. New `main_test.go` drives the CLI
|
|
in-process and asserts the exit codes from README §Error handling
|
|
plus the stdout/stderr split, including that a fatal error raised
|
|
after the database is open leaves no `-wal`/`-shm` sidecar behind
|
|
for `scan`, `report` or `trees`
|
|
- update golangci-lint to v2.12.2 with the canonical config
|
|
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
|
|
closes #3): bumped the pinned linter in the `Dockerfile` lint
|
|
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
|
|
`.golangci.yml` with the canonical file — the linter settings
|
|
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
|
`linters.settings` per the v2 schema, so they are actually
|
|
applied; no new lint findings surfaced
|
|
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
|
entrypoints like the other managed repos (2026-07-26, commit
|
|
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
|
|
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
|
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
|
`install-precommit`) and every Makefile target is now a thin shim
|
|
over them, matching the other managed repos
|
|
- make the binary the default Make target (2026-07-24, branch
|
|
`make-default-target`): plain `make` now builds `sfdupes`
|
|
(previously it ran `check` plus `build`); `make build` remains as
|
|
an alias
|
|
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
|
|
branch `scan-wide-phases`): all operands seed the shared walk pool
|
|
and every pass runs once over the whole scan, so totals and ETAs
|
|
are scan-global; the per-operand walk/hash/update cycles and their
|
|
stderr announcements are gone; the update pass commits in batched
|
|
transactions — the filesystem is authoritative and the database an
|
|
eventually-consistent reflection, so scan-level atomicity is not
|
|
required
|
|
- split the stat pass back out of the walk (2026-07-24, branch
|
|
`parallel-phases`): phases are strictly sequential again — walk,
|
|
stat, hash, update per operand — with parallelism only inside each
|
|
phase; the walk enumerates paths with per-directory workers and the
|
|
stat pass lstats them with per-file workers, restoring the exact
|
|
total/ETA stat bar
|
|
- announce each operand on stderr before its passes (2026-07-24,
|
|
branch `scan-operand-progress`): with per-operand walk/hash/update
|
|
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
|
|
that looked like the whole run's — an operator watching operand 3 of
|
|
14 hash 300k files concluded 20M files were being skipped
|
|
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
|
was a single goroutine and took hours at ~20M files on a busy pool
|
|
(observed: 22M files in 4h on a ZFS server); it is now a
|
|
per-directory worker-pool traversal that records size/mtime during
|
|
the walk (folding away the separate stat pass, halving metadata
|
|
I/O), and each `PATH` operand commits in its own transaction so an
|
|
interrupted scan keeps completed operands
|
|
|
|
- persistent scan database (2026-07-24, branch `persistent-database`):
|
|
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
|
Go, cgo stays disabled) keyed by absolute path that survives between
|
|
runs — a rescan hashes only new or changed files (by mtime/size),
|
|
deletes records for files vanished from under the scanned operands,
|
|
and leaves records outside them untouched, so `scan` can be cronned
|
|
daily; `report` and `trees` read the database (no positional
|
|
arguments) instead of a scan stream. Database at
|
|
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
|
WAL journaling plus a single-transaction update keep a report run
|
|
during a scan safe
|
|
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
|
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
|
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
|
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
|
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
|
at filesystem boundaries, which are crossed by default
|
|
- bring the repo into full policy compliance (2026-07-23, branch
|
|
`repo-policy-compliance`; checklist below)
|
|
- `git init` with README-only first commit; code baseline committed on
|
|
`main` (2026-07-22)
|
|
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
|
|
|
# Future Steps
|
|
|
|
- possible later features (explicitly out of scope per README):
|
|
full-content verification of candidates, removal-script helpers
|
|
|
|
# Repo Policy Compliance
|
|
|
|
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
|
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
|
standalone formatting commit is needed.
|
|
|
|
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
|
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
|
artifacts plus secrets patterns
|
|
- [x] `.editorconfig` missing
|
|
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
|
from house convention — user to confirm)
|
|
- [x] `REPO_POLICIES.md` missing from repo root
|
|
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
|
pass `make lint` (150 findings fixed; `make lint` is clean)
|
|
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
|
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
|
`build`, which writes the binary (`make check` must not modify
|
|
files)
|
|
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
|
real tests with a 30-second timeout and the conditional `-v`
|
|
rerun pattern (suite covers parsing, grouping, digests,
|
|
suppression, hashing, and the scan pipeline; 64% coverage)
|
|
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
|
fail-fast lint stage, build stage running `make check`
|
|
- [x] `.dockerignore` missing
|
|
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
|
checkout action pinned by commit SHA)
|
|
- [x] README lacks required sections: Description first line
|
|
(name/purpose/category/license/author), Getting Started,
|
|
Rationale, TODO, License, Author
|
|
- [x] README non-goal "no git repository setup and no CI" is stale now
|
|
that the repo is under git with CI
|
|
- [x] pre-commit hook not installed (`make hooks` once the target
|
|
exists)
|
|
|
|
Accepted divergences (no action):
|
|
|
|
- flat single-package layout with `.go` files in the repo root — fine
|
|
for a small single-binary tool per the Go styleguide; the tracker
|
|
audit agrees
|
|
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
|
(pure-Go builds) and the race detector requires cgo
|