sneak
1399249957
All checks were successful
check / check (push) Successful in 1m2s
hashPhase returned the moment recordRun failed and left the pool running: the feeder parked forever on a full jobs channel and every worker on a full results channel. Until #4 landed the process exited before that mattered; now that runScan returns an error and unwinds, the goroutines are a real leak. The pool is now an owned hashPool. Its context is derived from the scan's, every blocking send in the feeder and the workers selects on ctx.Done(), the feeder closes jobs on every path out so the workers' range always terminates, and hashPhase defers pool.stop(), which cancels and then drains results until the last goroutine has exited. Draining is the half that matters: a worker already parked on a send cannot observe the cancellation until a receiver frees it. ctx comes from cmd.Context() and is threaded through runScan, syncScan, both worker pools and the database layer as the first parameter throughout, so graceful interrupt handling has a path to hook into rather than a pool to rewrite. The walk pool never leaked, because walkPhase always drains its events to close, but it has the same unbounded-send shape and gets the same treatment, together with a ctx.Err() guard after the walk: a cancelled walk leaves a partial size census, and the update phase would read every file it never reached as vanished and delete its record. Tests drive the scan entry point against a database whose insert trigger aborts, with a fixture large enough that the failure lands partway through the hash phase with more runs queued than either pool channel can hold, and assert that the scan fails instead of hanging and that runtime.NumGoroutine polls back to its pre-scan baseline.
186 lines
9.7 KiB
Markdown
186 lines
9.7 KiB
Markdown
# Workflow
|
|
|
|
- take an issue from the `1.0.0` milestone on the tracker; work not
|
|
yet on the tracker gets filed as an issue first
|
|
- branch (from `main`)
|
|
- do the work, with tests, in small focused commits
|
|
- record it at the top of Completed Steps (`TODO.md` changes in the
|
|
same commit as the work)
|
|
- push the branch and open a PR whose title ends with
|
|
` (closes #N)`
|
|
- an independent review gates the merge; every finding is addressed
|
|
or explicitly rebutted on the PR
|
|
- merge to `main` once the review passes
|
|
|
|
# Status
|
|
|
|
- pre-1.0
|
|
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
|
|
open issues under the `1.0.0` milestone are what remains before
|
|
the tag, and this file records history and process, not the queue
|
|
|
|
# Next Step
|
|
|
|
- take the next issue from the `1.0.0` milestone on the tracker:
|
|
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
|
|
the source of truth for what is left before 1.0.0. Individual
|
|
issues are deliberately not restated here; a copy in this file
|
|
drifts out of date the moment the tracker moves
|
|
|
|
# Completed Steps
|
|
|
|
- unwind the hash worker pool on the error path (2026-08-09, branch
|
|
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
|
|
moment `recordRun` failed and abandon the pool — the feeder parked
|
|
forever on a full `jobs` channel and every worker on a full
|
|
`results` channel. That only stopped being invisible when #4 landed
|
|
and `runScan` began unwinding instead of calling `os.Exit`. The
|
|
pool is now an owned, context-aware `hashPool`: every blocking send
|
|
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
|
closed on every path out, and `hashPhase` defers `pool.stop()`,
|
|
which cancels and then drains `results` until the last goroutine
|
|
has exited — draining is what frees a worker already parked on a
|
|
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
|
|
`syncScan`, both worker pools and the whole database layer (it is
|
|
the first parameter everywhere), so #5 can hand this path a signal
|
|
and needs to add nothing else. The walk pool never leaked, because
|
|
`walkPhase` always drains its events to close, but it has the same
|
|
unbounded-send shape and #5 will give it an early return, so it
|
|
gets the same treatment plus a `ctx.Err()` guard after the walk: a
|
|
cancelled walk yields a partial size census, and the update phase
|
|
would read every unreached file as vanished and delete its record.
|
|
New tests drive `run(scan)` against a database whose insert trigger
|
|
aborts, and assert both that the scan fails instead of hanging and
|
|
that `runtime.NumGoroutine()` polls back to its pre-scan baseline
|
|
- guarantee the database is closed on every fatal exit path
|
|
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
|
|
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
|
|
it the SQLite WAL checkpoint — now actually runs when a subcommand
|
|
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
|
|
`resolveRoots` return errors instead. The single exit point is `run`
|
|
in `main.go`: it maps a `fatalError` (anything a subcommand
|
|
returned) to exit 1 and cobra's own argument and flag errors to exit
|
|
2, which keeps a runtime failure from being reported as a usage
|
|
error or printing the usage text. New `main_test.go` drives the CLI
|
|
in-process and asserts the exit codes from README §Error handling
|
|
plus the stdout/stderr split, including that a fatal error raised
|
|
after the database is open leaves no `-wal`/`-shm` sidecar behind
|
|
for `scan`, `report` or `trees`
|
|
- update golangci-lint to v2.12.2 with the canonical config
|
|
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
|
|
closes #3): bumped the pinned linter in the `Dockerfile` lint
|
|
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
|
|
`.golangci.yml` with the canonical file — the linter settings
|
|
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
|
`linters.settings` per the v2 schema, so they are actually
|
|
applied; no new lint findings surfaced
|
|
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
|
entrypoints like the other managed repos (2026-07-26, commit
|
|
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
|
|
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
|
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
|
`install-precommit`) and every Makefile target is now a thin shim
|
|
over them, matching the other managed repos
|
|
- make the binary the default Make target (2026-07-24, branch
|
|
`make-default-target`): plain `make` now builds `sfdupes`
|
|
(previously it ran `check` plus `build`); `make build` remains as
|
|
an alias
|
|
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
|
|
branch `scan-wide-phases`): all operands seed the shared walk pool
|
|
and every pass runs once over the whole scan, so totals and ETAs
|
|
are scan-global; the per-operand walk/hash/update cycles and their
|
|
stderr announcements are gone; the update pass commits in batched
|
|
transactions — the filesystem is authoritative and the database an
|
|
eventually-consistent reflection, so scan-level atomicity is not
|
|
required
|
|
- split the stat pass back out of the walk (2026-07-24, branch
|
|
`parallel-phases`): phases are strictly sequential again — walk,
|
|
stat, hash, update per operand — with parallelism only inside each
|
|
phase; the walk enumerates paths with per-directory workers and the
|
|
stat pass lstats them with per-file workers, restoring the exact
|
|
total/ETA stat bar
|
|
- announce each operand on stderr before its passes (2026-07-24,
|
|
branch `scan-operand-progress`): with per-operand walk/hash/update
|
|
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
|
|
that looked like the whole run's — an operator watching operand 3 of
|
|
14 hash 300k files concluded 20M files were being skipped
|
|
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
|
was a single goroutine and took hours at ~20M files on a busy pool
|
|
(observed: 22M files in 4h on a ZFS server); it is now a
|
|
per-directory worker-pool traversal that records size/mtime during
|
|
the walk (folding away the separate stat pass, halving metadata
|
|
I/O), and each `PATH` operand commits in its own transaction so an
|
|
interrupted scan keeps completed operands
|
|
|
|
- persistent scan database (2026-07-24, branch `persistent-database`):
|
|
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
|
Go, cgo stays disabled) keyed by absolute path that survives between
|
|
runs — a rescan hashes only new or changed files (by mtime/size),
|
|
deletes records for files vanished from under the scanned operands,
|
|
and leaves records outside them untouched, so `scan` can be cronned
|
|
daily; `report` and `trees` read the database (no positional
|
|
arguments) instead of a scan stream. Database at
|
|
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
|
WAL journaling plus a single-transaction update keep a report run
|
|
during a scan safe
|
|
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
|
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
|
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
|
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
|
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
|
at filesystem boundaries, which are crossed by default
|
|
- bring the repo into full policy compliance (2026-07-23, branch
|
|
`repo-policy-compliance`; checklist below)
|
|
- `git init` with README-only first commit; code baseline committed on
|
|
`main` (2026-07-22)
|
|
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
|
|
|
# Future Steps
|
|
|
|
- possible later features (explicitly out of scope per README):
|
|
full-content verification of candidates, removal-script helpers
|
|
|
|
# Repo Policy Compliance
|
|
|
|
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
|
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
|
standalone formatting commit is needed.
|
|
|
|
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
|
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
|
artifacts plus secrets patterns
|
|
- [x] `.editorconfig` missing
|
|
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
|
from house convention — user to confirm)
|
|
- [x] `REPO_POLICIES.md` missing from repo root
|
|
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
|
pass `make lint` (150 findings fixed; `make lint` is clean)
|
|
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
|
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
|
`build`, which writes the binary (`make check` must not modify
|
|
files)
|
|
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
|
real tests with a 30-second timeout and the conditional `-v`
|
|
rerun pattern (suite covers parsing, grouping, digests,
|
|
suppression, hashing, and the scan pipeline; 64% coverage)
|
|
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
|
fail-fast lint stage, build stage running `make check`
|
|
- [x] `.dockerignore` missing
|
|
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
|
checkout action pinned by commit SHA)
|
|
- [x] README lacks required sections: Description first line
|
|
(name/purpose/category/license/author), Getting Started,
|
|
Rationale, TODO, License, Author
|
|
- [x] README non-goal "no git repository setup and no CI" is stale now
|
|
that the repo is under git with CI
|
|
- [x] pre-commit hook not installed (`make hooks` once the target
|
|
exists)
|
|
|
|
Accepted divergences (no action):
|
|
|
|
- flat single-package layout with `.go` files in the repo root — fine
|
|
for a small single-binary tool per the Go styleguide; the tracker
|
|
audit agrees
|
|
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
|
(pure-Go builds) and the race detector requires cgo
|