1a38570301
All checks were successful
check / check (push) Successful in 1m32s
TestSyncScanCancelledWalkKeepsRecords handed syncScan a context that was already cancelled. loadIndex is the first thing syncScan does, and its QueryContext fails on that context, so the scan returned before startWalk was ever called: no walk ran, no pool started, no write path was reachable, and all three of the test's assertions held for the wrong reason. The post-walk ctx.Err() guard, which is the highest-stakes line in the change, had no coverage at all — a panic in its body, or deleting it outright, left the suite green. Replace it with TestSyncScanCancelledMidWalkKeepsRecords, which cancels during the walk and so reaches the guard holding a genuinely partial census and a still-populated record index. The cancellation is driven by the scan's own progress rather than by a timer: walkClock is a context that cancels itself once its Done method has been consulted a set number of times, and since every blocking channel operation in the walk selects on Done — one consultation per event, a couple per directory, against the index load's fixed three — a threshold set to a quarter of the fixture's file count lands the cancellation deep inside the walk on every run. The census settles at around 380 of 2000 files, leaving some 1600 records that a complete-looking census would have handed to the update phase as deletions. The already-cancelled case is kept, renamed to what it actually tests and with its goroutine assertion dropped, since nothing that could leak is ever started. Direct tests cover the remaining cancellation branches of both pools: sendEvent abandoning a blocked send, walk workers dropping queued directories, a walk worker abandoning its subdirectory hand-off, dispatchDirs closing jobs on its way out, feedHashJobs doing the same, hashWorker dropping queued runs, and hashPhase leaving its result loop. Each is deterministic — the channels involved are unbuffered, unread or pre-filled, so the cancellation case is the only one that can be ready. Also correct two overstated claims. The hashLeakFiles comment described a mechanism that does not occur: the surplus is absorbed exactly by the two pool channels plus the workers in flight, so the feeder drains and exits, and what an abandoned pool leaves parked is the workers and the goroutine waiting on them. And the guard is defence in depth, not the sole barrier against data loss: the update phase's BeginTx fails on the same cancelled context before deleting anything today. The guard is what keeps that true once an interrupted scan is allowed to commit what it has.
197 lines
10 KiB
Markdown
197 lines
10 KiB
Markdown
# Workflow
|
|
|
|
- take an issue from the `1.0.0` milestone on the tracker; work not
|
|
yet on the tracker gets filed as an issue first
|
|
- branch (from `main`)
|
|
- do the work, with tests, in small focused commits
|
|
- record it at the top of Completed Steps (`TODO.md` changes in the
|
|
same commit as the work)
|
|
- push the branch and open a PR whose title ends with
|
|
` (closes #N)`
|
|
- an independent review gates the merge; every finding is addressed
|
|
or explicitly rebutted on the PR
|
|
- merge to `main` once the review passes
|
|
|
|
# Status
|
|
|
|
- pre-1.0
|
|
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
|
|
open issues under the `1.0.0` milestone are what remains before
|
|
the tag, and this file records history and process, not the queue
|
|
|
|
# Next Step
|
|
|
|
- take the next issue from the `1.0.0` milestone on the tracker:
|
|
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
|
|
the source of truth for what is left before 1.0.0. Individual
|
|
issues are deliberately not restated here; a copy in this file
|
|
drifts out of date the moment the tracker moves
|
|
|
|
# Completed Steps
|
|
|
|
- unwind the hash worker pool on the error path (2026-08-09, branch
|
|
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
|
|
moment `recordRun` failed and abandon the pool — the feeder parked
|
|
forever on a full `jobs` channel and every worker on a full
|
|
`results` channel. That only stopped being invisible when #4 landed
|
|
and `runScan` began unwinding instead of calling `os.Exit`. The
|
|
pool is now an owned, context-aware `hashPool`: every blocking send
|
|
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
|
closed on every path out, and `hashPhase` defers `pool.stop()`,
|
|
which cancels and then drains `results` until the last goroutine
|
|
has exited — draining is what frees a worker already parked on a
|
|
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
|
|
`syncScan`, both worker pools and the whole database layer (it is
|
|
the first parameter everywhere), so #5 can hand this path a signal
|
|
and needs to add nothing else. The walk pool never leaked, because
|
|
`walkPhase` always drains its events to close, but it has the same
|
|
unbounded-send shape and #5 will give it an early return, so it
|
|
gets the same treatment plus a `ctx.Err()` guard after the walk: a
|
|
cancelled walk yields a partial size census, and every file it never
|
|
reached looks vanished to the update phase. That phase's own
|
|
`BeginTx` fails on the same cancelled context before deleting
|
|
anything, so the guard is defence in depth rather than the only
|
|
barrier — but it is the one that survives #5 deciding an interrupted
|
|
scan may commit what it has. Tests drive `run(scan)` against a
|
|
database whose insert trigger aborts, and assert both that the scan
|
|
fails instead of hanging and that `runtime.NumGoroutine()` polls
|
|
back to its pre-scan baseline; a second set cancels a scan part-way
|
|
through the walk — deterministically, by counting the scan's own
|
|
consultations of `ctx.Done()` rather than racing a timer — and
|
|
asserts that it stops at the guard holding a partial census and a
|
|
still-populated record index, with every record intact. The
|
|
remaining cancellation branches of both pools are covered by direct
|
|
tests of `sendEvent`, the walk workers, `dispatchDirs`,
|
|
`feedHashJobs`, `hashWorker` and `hashPhase`
|
|
- guarantee the database is closed on every fatal exit path
|
|
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
|
|
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
|
|
it the SQLite WAL checkpoint — now actually runs when a subcommand
|
|
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
|
|
`resolveRoots` return errors instead. The single exit point is `run`
|
|
in `main.go`: it maps a `fatalError` (anything a subcommand
|
|
returned) to exit 1 and cobra's own argument and flag errors to exit
|
|
2, which keeps a runtime failure from being reported as a usage
|
|
error or printing the usage text. New `main_test.go` drives the CLI
|
|
in-process and asserts the exit codes from README §Error handling
|
|
plus the stdout/stderr split, including that a fatal error raised
|
|
after the database is open leaves no `-wal`/`-shm` sidecar behind
|
|
for `scan`, `report` or `trees`
|
|
- update golangci-lint to v2.12.2 with the canonical config
|
|
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
|
|
closes #3): bumped the pinned linter in the `Dockerfile` lint
|
|
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
|
|
`.golangci.yml` with the canonical file — the linter settings
|
|
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
|
`linters.settings` per the v2 schema, so they are actually
|
|
applied; no new lint findings surfaced
|
|
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
|
entrypoints like the other managed repos (2026-07-26, commit
|
|
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
|
|
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
|
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
|
`install-precommit`) and every Makefile target is now a thin shim
|
|
over them, matching the other managed repos
|
|
- make the binary the default Make target (2026-07-24, branch
|
|
`make-default-target`): plain `make` now builds `sfdupes`
|
|
(previously it ran `check` plus `build`); `make build` remains as
|
|
an alias
|
|
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
|
|
branch `scan-wide-phases`): all operands seed the shared walk pool
|
|
and every pass runs once over the whole scan, so totals and ETAs
|
|
are scan-global; the per-operand walk/hash/update cycles and their
|
|
stderr announcements are gone; the update pass commits in batched
|
|
transactions — the filesystem is authoritative and the database an
|
|
eventually-consistent reflection, so scan-level atomicity is not
|
|
required
|
|
- split the stat pass back out of the walk (2026-07-24, branch
|
|
`parallel-phases`): phases are strictly sequential again — walk,
|
|
stat, hash, update per operand — with parallelism only inside each
|
|
phase; the walk enumerates paths with per-directory workers and the
|
|
stat pass lstats them with per-file workers, restoring the exact
|
|
total/ETA stat bar
|
|
- announce each operand on stderr before its passes (2026-07-24,
|
|
branch `scan-operand-progress`): with per-operand walk/hash/update
|
|
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
|
|
that looked like the whole run's — an operator watching operand 3 of
|
|
14 hash 300k files concluded 20M files were being skipped
|
|
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
|
was a single goroutine and took hours at ~20M files on a busy pool
|
|
(observed: 22M files in 4h on a ZFS server); it is now a
|
|
per-directory worker-pool traversal that records size/mtime during
|
|
the walk (folding away the separate stat pass, halving metadata
|
|
I/O), and each `PATH` operand commits in its own transaction so an
|
|
interrupted scan keeps completed operands
|
|
|
|
- persistent scan database (2026-07-24, branch `persistent-database`):
|
|
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
|
Go, cgo stays disabled) keyed by absolute path that survives between
|
|
runs — a rescan hashes only new or changed files (by mtime/size),
|
|
deletes records for files vanished from under the scanned operands,
|
|
and leaves records outside them untouched, so `scan` can be cronned
|
|
daily; `report` and `trees` read the database (no positional
|
|
arguments) instead of a scan stream. Database at
|
|
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
|
WAL journaling plus a single-transaction update keep a report run
|
|
during a scan safe
|
|
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
|
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
|
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
|
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
|
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
|
at filesystem boundaries, which are crossed by default
|
|
- bring the repo into full policy compliance (2026-07-23, branch
|
|
`repo-policy-compliance`; checklist below)
|
|
- `git init` with README-only first commit; code baseline committed on
|
|
`main` (2026-07-22)
|
|
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
|
|
|
# Future Steps
|
|
|
|
- possible later features (explicitly out of scope per README):
|
|
full-content verification of candidates, removal-script helpers
|
|
|
|
# Repo Policy Compliance
|
|
|
|
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
|
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
|
standalone formatting commit is needed.
|
|
|
|
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
|
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
|
artifacts plus secrets patterns
|
|
- [x] `.editorconfig` missing
|
|
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
|
from house convention — user to confirm)
|
|
- [x] `REPO_POLICIES.md` missing from repo root
|
|
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
|
pass `make lint` (150 findings fixed; `make lint` is clean)
|
|
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
|
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
|
`build`, which writes the binary (`make check` must not modify
|
|
files)
|
|
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
|
real tests with a 30-second timeout and the conditional `-v`
|
|
rerun pattern (suite covers parsing, grouping, digests,
|
|
suppression, hashing, and the scan pipeline; 64% coverage)
|
|
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
|
fail-fast lint stage, build stage running `make check`
|
|
- [x] `.dockerignore` missing
|
|
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
|
checkout action pinned by commit SHA)
|
|
- [x] README lacks required sections: Description first line
|
|
(name/purpose/category/license/author), Getting Started,
|
|
Rationale, TODO, License, Author
|
|
- [x] README non-goal "no git repository setup and no CI" is stale now
|
|
that the repo is under git with CI
|
|
- [x] pre-commit hook not installed (`make hooks` once the target
|
|
exists)
|
|
|
|
Accepted divergences (no action):
|
|
|
|
- flat single-package layout with `.go` files in the repo root — fine
|
|
for a small single-binary tool per the Go styleguide; the tracker
|
|
audit agrees
|
|
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
|
(pure-Go builds) and the race detector requires cgo
|