sneak/sfdupes
1
0
Fork 0
Code Issues 27 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Reformat all Markdown with prettier
check / check (push) Successful in 2m2s
Details

Browse Source 
Mechanical result of `make fmt` now that prettier runs over Markdown:
README.md and TODO.md rewrapped to the house settings (4-space,
proseWrap always). REPO_POLICIES.md was already compliant. No prose was
changed by hand in this commit.

Model: opus-4-8
This commit is contained in:
clawbot
2026-09-21 07:32:12 +00:00
parent 2dd4b8c401
commit 282146a409
2 changed files with 733 additions and 843 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+399 -461
View File
File diff suppressed because it is too large Load Diff
TODO.md
+334 -382
View File
@@ -1,434 +1,386 @@
# Workflow # Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not - take an issue from the `1.0.0` milestone on the tracker; work not yet on the
yet on the tracker gets filed as an issue first tracker gets filed as an issue first
- branch (from `main`) - branch (from `main`)
- do the work, with tests, in small focused commits - do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the - record it at the top of Completed Steps (`TODO.md` changes in the same commit
same commit as the work) as the work)
- push the branch and open a PR whose title ends with - push the branch and open a PR whose title ends with ` (closes #N)`
` (closes #N)` - an independent review gates the merge; every finding is addressed or
- an independent review gates the merge; every finding is addressed explicitly rebutted on the PR
or explicitly rebutted on the PR
- merge to `main` once the review passes - merge to `main` once the review passes
# Status # Status
- pre-1.0 - pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the - the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
open issues under the `1.0.0` milestone are what remains before under the `1.0.0` milestone are what remains before the tag, and this file
the tag, and this file records history and process, not the queue records history and process, not the queue
# Next Step # Next Step
- take the next issue from the `1.0.0` milestone on the tracker: - take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
the source of truth for what is left before 1.0.0. Individual of truth for what is left before 1.0.0. Individual issues are deliberately not
issues are deliberately not restated here; a copy in this file restated here; a copy in this file drifts out of date the moment the tracker
drifts out of date the moment the tracker moves moves
# Completed Steps # Completed Steps
- restore Markdown formatting in `script/fmt`/`fmt-check` and reformat - restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all
all Markdown to the house prettier settings (2026-09-21, closes Markdown to the house prettier settings (2026-09-21, closes
https://git.eeqj.de/sneak/sfdupes/issues/19) https://git.eeqj.de/sneak/sfdupes/issues/19)
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes `Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the `(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
redundant tag, so both pins are the policy `# image:vX.Y.Z, so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest `FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
unchanged. `script/verify-lint-image-pin` parses those `FROM` lines parses those `FROM` lines and still matches the tagless form; its advice line
and still matches the tagless form; its advice line lost the now lost the now meaningless "tag and digest". With no tag in either reference, a
meaningless "tag and digest". With no tag in either reference, a tag-only disagreement no longer exists — a one-sided tag is caught as a plain
tag-only disagreement no longer exists — a one-sided tag is caught as mismatch.
a plain mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` - run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
(2026-08-10, branch `next`, closes branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the owner ruling, the linter runs inside a container invoked through the `script/`
linter runs inside a container invoked through the `script/` entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
entrypoint and is never installed on a host. New root the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
`Dockerfile.lint` COPYs the repo into the digest-pinned runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
`golangci/golangci-lint:v2.12.2` image and runs a successful build IS a clean lint; `script/lint` is reduced to building it.
`golangci-lint config verify` and `golangci-lint run` as build `script/bootstrap` loses the `go install`, the pin constants, the version
steps, so a successful build IS a clean lint; `script/lint` is parser and `verify_golangci_lint` outright rather than hardening them — with
reduced to building it. `script/bootstrap` loses the `go install`, nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
the pin constants, the version parser and `verify_golangci_lint` motivated them has no subject — and now warns rather than fails when `docker`
outright rather than hardening them — with nothing linting on the is absent. Two traps handled. A lint build on an unchanged tree returns
host, the `$GOPATH/bin` versus `PATH` problem that motivated them has success in well under a second having run no linter, which is
no subject — and now warns rather than fails when `docker` is absent.
Two traps handled. A lint build on an unchanged tree returns success
in well under a second having run no linter, which is
https://git.eeqj.de/sneak/sfdupes/issues/32 and https://git.eeqj.de/sneak/sfdupes/issues/32 and
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
`Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
inside every gate `RUN` (BuildKit hashes the expanded command, not the expanded command, not the declaration) and `script/lint` passes
the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the `"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
PID matters because two lint runs land inside the same second easily. same second easily. And nothing inside an image build may shell out to docker,
And nothing inside an image build may shell out to docker, so the so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
instead of `make lint`, and its build stage runs `make test` and instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate (`make`, not `make fmt-check` instead of the `make check` aggregate (`make`, not the
the scripts bare, because the Makefile's `export CGO_ENABLED = 0` scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
only reaches what it invokes). `COPY --from=lint` what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
`/usr/bin/golangci-lint` is replaced by `COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the forcing BuildKit to finish linting before the build stage starts, and dropping
only edge forcing BuildKit to finish linting before the build stage it without replacing the edge would have ended fail-fast linting silently
starts, and dropping it without replacing the edge would have ended under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
fail-fast linting silently under a still-green build. That is edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
canonical `REPO_POLICIES.md:107`'s ordering edge, restored. `go install` that justified it. `script/verify-linter-pin` is retired, deleted
`ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install` along with its README entry, because both of its subjects ceased to exist in
that justified it. `script/verify-linter-pin` is retired, deleted the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
along with its README entry, because both of its subjects ceased to in `script/bootstrap`, and there is now neither a binary crossing between
exist in the same change: it compared a linter binary against stages nor a version pin in bootstrap. The drift it guarded has not gone away,
`GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now it has moved — the linter is still pinned twice, now as the `FROM` line of
neither a binary crossing between stages nor a version pin in `Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
bootstrap. The drift it guarded has not gone away, it has moved — the nothing syncing them, which is exactly what
linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
with nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`, replacement is one new `script/verify-lint-image-pin`, run as a gate in both
run as a gate in both files, which compares the two references to files, which compares the two references to each other and deliberately
each other and deliberately restates neither: a hardcoded expected restates neither: a hardcoded expected digest would be a third copy and the
digest would be a third copy and the same drift one file further out. same drift one file further out. `golangci-lint config verify` is included per
`golangci-lint config verify` is included per the ruling, and the the ruling, and the concern about its unpinned live HTTPS schema fetch was
concern about its unpinned live HTTPS schema fetch was measured measured rather than assumed — under `--network none` the pinned binary both
rather than assumed — under `--network none` the pinned binary both passes a valid config and rejects an invalid one with the jsonschema error, so
passes a valid config and rejects an invalid one with the jsonschema it validates from an embedded schema and makes no network call of its own. The
error, so it validates from an embedded schema and makes no network README scopes that to the gate steps rather than to linting as a whole:
call of its own. The README scopes that to the gate steps rather `Dockerfile.lint` runs `go mod download` above them, so a cold cache still
than to linting as a whole: `Dockerfile.lint` runs `go mod download` needs the network and only a warm one lints offline. Verified: `make lint`
above them, so a cold cache still needs the network and only a warm green with every `PATH` directory containing a `golangci-lint` removed
one lints offline. Verified: `make lint` green with every `PATH`
directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`; (`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs `command -v golangci-lint` empty); two consecutive `script/lint` runs on an
on an untouched tree both executed the linter, 27.7s and 28.7s in the untouched tree both executed the linter, 27.7s and 28.7s in the lint step
lint step under distinct epochs with the `COPY . .` layer `CACHED` under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
above them, at 42.2s and 41.8s wall clock — the no-cache rule was not and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
weakened to shorten that. Negative control: a planted Negative control: a planted `var unusedIssue46Sentinel = 1` failed
`var unusedIssue46Sentinel = 1` failed `script/lint` with `script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and `report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
failed `make docker` at `[lint 9/9]` with the build stage stopped at `make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
`[builder 3/12]` — `COPY --from=lint`, `script/bootstrap`, the test — `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
gate and `make build` all zero occurrences — then reverted clean. The zero occurrences — then reverted clean. The drift guard fails on a tag-only
drift guard fails on a tag-only disagreement, on a digest-only disagreement, on a digest-only disagreement, and on an unreadable reference,
disagreement, and on an unreadable reference, naming both sides. naming both sides. `make docker` green in 5m35s with all six gates executing
`make docker` green in 5m35s with all six gates executing under one under one epoch (lint 37.6s, test 25.2s reporting
epoch (lint 37.6s, test 25.2s reporting `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). non-root quirk still holds: in the builder image with the Go test cache off,
The non-root quirk still holds: in the builder image with the Go test `--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether` unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
(exit 1) where the unprivileged user passes (exit 0). Noted for `golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
follow-up, not fixed here: `golangci-lint` warns that the in favour of `gomodguard_v2`.
`gomodguard` linter is deprecated since v2.12.0 in favour of
`gomodguard_v2`.
- install the Docker build stage's prerequisites by running - install the Docker build stage's prerequisites by running `script/bootstrap`
`script/bootstrap` instead of `apk add --no-cache make` inline instead of `apk add --no-cache make` inline (2026-08-09, branch
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical `dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
`REPO_POLICIES.md:97` requires it, and the inline install left the it, and the inline install left the build stage maintaining its own notion of
build stage maintaining its own notion of the toolchain — exactly the toolchain — exactly the divergence #24 exists to close, one layer down.
the divergence #24 exists to close, one layer down. The stage now The stage now copies `script/` plus `go.mod`/`go.sum` and runs
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`, `script/bootstrap`, which ends in `go mod download`, so the separate
which ends in `go mod download`, so the separate invocation of that invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves and moves above the bootstrap layer. It is the only edge making this stage
above the bootstrap layer. It is the only edge making this stage depend on the lint stage, so deleting it as redundant would end fail-fast
depend on the lint stage, so deleting it as redundant would end linting silently. Letting bootstrap install its own linter here would have
fail-fast linting silently. Letting bootstrap install its own linter reintroduced the second toolchain and paid for a from-source build of it. What
here would have reintroduced the second toolchain and paid for a makes the two stages provably one toolchain rather than two that happen to
from-source build of it. What makes the two stages provably one agree is a new `script/verify-linter-pin`, run in the build stage on the
toolchain rather than two that happen to agree is a new binary that arrives from the lint stage, before bootstrap: it fails the build
`script/verify-linter-pin`, run in the build stage on the binary naming both versions unless that binary is the version `script/bootstrap`
that arrives from the lint stage, before bootstrap: it fails the pins. Bootstrap's own check could not serve that purpose — it reinstalls its
build naming both versions unless that binary is the version pin from source and then verifies whatever `PATH` resolves, so drift
`script/bootstrap` pins. Bootstrap's own check could not serve that self-heals silently and a lint stage image bumped on its own would lint at the
purpose — it reinstalls its pin from source and then verifies new version while `make check` ran at the old one, green. The linter version
whatever `PATH` resolves, so drift self-heals silently and a lint is pinned in two independent places (the lint stage image digest and
stage image bumped on its own would lint at the new version while
`make check` ran at the old one, green. The linter version is pinned
in two independent places (the lint stage image digest and
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
half-applied bump is now a build failure. The pin is read out of half-applied bump is now a build failure. The pin is read out of
`script/bootstrap`, which stays the single source of truth; a pin `script/bootstrap`, which stays the single source of truth; a pin that cannot
that cannot be read is a hard failure, not a skip. The check needs be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
no `CHECK_EPOCH`: its only inputs are the copied binary and only inputs are the copied binary and `script/`, so Docker invalidates the
`script/`, so Docker invalidates the layer exactly when a cached layer exactly when a cached result would stop being true, and it is documented
result would stop being true, and it is documented with the other with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
entrypoints in the README. `$GOPATH/bin` joins `PATH` because that is where bootstrap's `go install` lands and bootstrap verifies its
that is where bootstrap's `go install` lands and bootstrap verifies installs against what `PATH` resolves — nothing in the image is shadowed by
its installs against what `PATH` resolves — nothing in the image is it, the directory does not exist until bootstrap runs. Everything added sits
shadowed by it, the directory does not exist until bootstrap runs. above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and `make check`. Verified: the guard fails the build with both versions named
`USER builder` still precede `make check`. Verified: the guard fails when the lint stage's linter is faked to a different version, and an
the build with both versions named when the lint stage's linter is unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
faked to a different version, and an unmodified build still passes its `apk` branch, installing `git` and `make` and finding the copied linter
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch, already at the pin; a second build served the bootstrap and dependency layers
installing `git` and `make` and finding the copied `CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
linter already at the pin; a second build served the bootstrap and failed the build at the lint gate in 48.9s with the build stage's `make check`
dependency layers `CACHED` while both gates ran with a fresh epoch; never starting; and the suite run in the image as `--user 0:0` fails
a planted `unused` finding failed the build at the lint gate in `TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
48.9s with the build stage's `make check` never starting; and the still load-bearing. That last check needs the Go test cache disabled — the
suite run in the image as `--user 0:0` fails first attempt reported `ok ... (cached)` as root, reusing the result the
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged build-time run had left in the shared cache, which would have read as a pass.
user is still load-bearing. That last check needs the Go test cache Build wall time, on a shared host running many concurrent builds and so noisy:
disabled — the first attempt reported `ok ... (cached)` as root, 2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
reusing the result the build-time run had left in the shared cache, change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
which would have read as a pass. Build wall time, on a shared host because of this change — `chown -R builder:builder /src /home/builder` walks
running many concurrent builds and so noisy: 2m13s on an unchanged the module cache and re-runs on every source change, and it alone varied
tree, 2m17s and 4m29s for two builds after a source change, 5m14s between 77s and 210s across those four builds, which is also the whole spread
cold. Only the cold one breaches the policy ceiling, and not because in the totals. The same cold measurement against `main` is 5m03s with a 209s
of this change — `chown -R builder:builder /src /home/builder` walks `chown`. Filed as #43
the module cache and re-runs on every source change, and it alone - bust the Docker layer cache for the gate steps, so `script/cibuild` and
varied between 77s and 210s across those four builds, which is also `script/docker` cannot report a green they did not earn (2026-08-09, branch
the whole spread in the totals. The same cold measurement against `cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
`main` is 5m03s with a 209s `chown`. Filed as #43 invocations with no cache control, and the `Dockerfile` copies the tree before
- bust the Docker layer cache for the gate steps, so `script/cibuild` running its gates, so on an unchanged tree Docker served those layers from
and `script/docker` cannot report a green they did not earn cache and the build exited 0 having executed nothing. That is not hypothetical
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts here — every merge this repo has done is a non-fast-forward merge of an
were bare `docker build` invocations with no cache control, and the undiverged branch, so each merge commit's tree is byte-identical to the branch
`Dockerfile` copies the tree before running its gates, so on an head's and each merge CI run was almost certainly a full cache hit; and PR
unchanged tree Docker served those layers from cache and the build #31's reviewer found `make docker` returning success as a 17-layer cache hit,
exited 0 having executed nothing. That is not hypothetical here — catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
every merge this repo has done is a non-fast-forward merge of an scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
undiverged branch, so each merge commit's tree is byte-identical to break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
the branch head's and each merge CI run was almost certainly a full across two — `make fmt-check` and `make lint` in the lint stage, `make check`
cache hit; and PR #31's reviewer found `make docker` returning in the build stage — so a single declaration would have left one stage
success as a 17-layer cache hit, catching it only by being silently cacheable; it is declared in both. And BuildKit hashes the expanded
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break nothing: each gate `RUN` echoes the epoch, which also puts the value in the
it. `ARG` is scoped per stage and this `Dockerfile` has three gates build log as evidence the layer really ran. Placement is below the dependency
across two — `make fmt-check` and `make lint` in the lint stage, layers on purpose — a build that goes cold every time would be a different
`make check` in the build stage — so a single declaration would have bug, not a fix. Verified by running each script twice back to back on an
left one stage silently cacheable; it is declared in both. And unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
BuildKit hashes the expanded command, not the declaration, so a all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN` 61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
echoes the epoch, which also puts the value in the build log as `CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
evidence the layer really ran. Placement is below the dependency the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
layers on purpose — a build that goes cold every time would be a the lint stage, and the binary copy into the runtime stage. The lint stage
different bug, not a fix. Verified by running each script twice back still gates the build stage: with a deliberate `unused` finding planted in the
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all tree, the build failed at `make lint` in 36.1s and the build-stage
three gates executed on all four runs, each with a fresh epoch in `make check` never started. The build stage also still drops to the
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s unprivileged `builder` user before `make check`, which the suite depends on
then 53.4s), and twelve steps were still served `CACHED` in the rather than merely prefers: forcing the same image to run the tests as root
steady state — both `go mod download`s, `apk add`, `adduser`, the fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy the `chmod(0)` the test uses to prove hard links are read once. This is the
out of the lint stage, and the binary copy into the runtime stage. local fix only; propagating it to the canonical templates is `prompts` #26
The lint stage still gates the build stage: with a deliberate - check the installed golangci-lint version in `script/bootstrap` instead of
`unused` finding planted in the tree, the build failed at only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
`make lint` in 36.1s and the build-stage `make check` never started. `missing golangci-lint` meant any linter already on `PATH` satisfied the
The build stage also still drops to the unprivileged `builder` user check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
before `make check`, which the suite depends on rather than merely on every host that already had one — this host ran v2.10.1 against a v2.12.2
prefers: forcing the same image to run the tests as root fails pin, `make check` went green, and `make docker` then rejected the same commit
`TestScanHardlinkRunFailsTogether`, because root reads straight with findings the local gate never saw. The version now lives in one place,
through the `chmod(0)` the test uses to prove hard links are read `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
once. This is the local fix only; propagating it to the canonical bump cannot half-apply; a `golangci_lint_version` helper parses
templates is `prompts` #26 `golangci-lint --version` (taking the field after the word `version` and
- check the installed golangci-lint version in `script/bootstrap` tolerating an optional leading `v`, which the module ref carries and the
instead of only its presence (2026-08-09, branch binary's output does not), and any version that is not the pin — older, newer,
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant absent or unparseable — is reinstalled. The install is then verified against
any linter already on `PATH` satisfied the check, so the pin was never the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
consulted and the v2.12.2 bump from #3 was inert on every host that `GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
already had one — this host ran v2.10.1 against a v2.12.2 pin, `PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
`make check` went green, and `make docker` then rejected the same the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
commit with findings the local gate never saw. The version now lives the install and leave the local gate disagreeing with CI under an affirmative
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module `bootstrap complete`. Bootstrap now re-reads the effective version after
ref derived from it so a bump cannot half-apply; a installing and, on a mismatch, prints both paths and both versions to stderr
`golangci_lint_version` helper parses `golangci-lint --version` and exits non-zero instead of claiming success; it does not reorder anyone's
(taking the field after the word `version` and tolerating an optional
leading `v`, which the module ref carries and the binary's output does
not), and any version that is not the pin — older, newer, absent or
unparseable — is reinstalled. The install is then verified against the
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
stage makes — would swallow the install and leave the local gate
disagreeing with CI under an affirmative `bootstrap complete`.
Bootstrap now re-reads the effective version after installing and, on
a mismatch, prints both paths and both versions to stderr and exits
non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr `PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than connected, so a present-but-broken binary says why rather than reinstalling
reinstalling forever in silence, and is bounded by `timeout(1)` where forever in silence, and is bounded by `timeout(1)` where that exists, so a
that exists, so a wedged binary cannot hang bootstrap. `git`, `make` wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
and `go` keep their presence-only checks and now say why in a presence-only checks and now say why in a comment: they are host
comment: they are host package-manager tools the repo deliberately package-manager tools the repo deliberately does not pin, with `go.mod`
does not pin, with `go.mod` governing the language version and the governing the language version and the digest-pinned images covering
digest-pinned images covering reproducible builds. Verified on this reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to v2.12.2 and running it again to a no-op, plus stub runs of the real script
a no-op, plus stub runs of the real script under `dash` covering a under `dash` covering a thirteen-input parse matrix (absent, older, newer,
thirteen-input parse matrix (absent, older, newer, host-style, host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor impostor binary, `(devel)`, trailing `version`), a shadowed install that must
binary, `(devel)`, trailing `version`), a shadowed install that must exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
exit non-zero, an install destination not on `PATH` at all, `GOBIN` wedged binary that must hit the timeout; `make check` and `make lint` are
set, and a wedged binary that must hit the timeout; `make check` and clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch - unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the `hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
moment `recordRun` failed and abandon the pool — the feeder parked `recordRun` failed and abandon the pool — the feeder parked forever on a full
forever on a full `jobs` channel and every worker on a full `jobs` channel and every worker on a full `results` channel. That only stopped
`results` channel. That only stopped being invisible when #4 landed being invisible when #4 landed and `runScan` began unwinding instead of
and `runScan` began unwinding instead of calling `os.Exit`. The calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
pool is now an owned, context-aware `hashPool`: every blocking send blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
in the feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
closed on every path out, and `hashPhase` defers `pool.stop()`, and then drains `results` until the last goroutine has exited — draining is
which cancels and then drains `results` until the last goroutine what frees a worker already parked on a send. `ctx` is threaded from
has exited — draining is what frees a worker already parked on a `cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
send. `ctx` is threaded from `cmd.Context()` through `runScan`, database layer (it is the first parameter everywhere), so #5 can hand this
`syncScan`, both worker pools and the whole database layer (it is path a signal and needs to add nothing else. The walk pool never leaked,
the first parameter everywhere), so #5 can hand this path a signal because `walkPhase` always drains its events to close, but it has the same
and needs to add nothing else. The walk pool never leaked, because unbounded-send shape and #5 will give it an early return, so it gets the same
`walkPhase` always drains its events to close, but it has the same treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
unbounded-send shape and #5 will give it an early return, so it partial size census, and every file it never reached looks vanished to the
gets the same treatment plus a `ctx.Err()` guard after the walk: a update phase. That phase's own `BeginTx` fails on the same cancelled context
cancelled walk yields a partial size census, and every file it never before deleting anything, so the guard is defence in depth rather than the
reached looks vanished to the update phase. That phase's own only barrier — but it is the one that survives #5 deciding an interrupted scan
`BeginTx` fails on the same cancelled context before deleting may commit what it has. Tests drive `run(scan)` against a database whose
anything, so the guard is defence in depth rather than the only insert trigger aborts, and assert both that the scan fails instead of hanging
barrier — but it is the one that survives #5 deciding an interrupted and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
scan may commit what it has. Tests drive `run(scan)` against a second set cancels a scan part-way through the walk — deterministically, by
database whose insert trigger aborts, and assert both that the scan counting the scan's own consultations of `ctx.Done()` rather than racing a
fails instead of hanging and that `runtime.NumGoroutine()` polls timer — and asserts that it stops at the guard holding a partial census and a
back to its pre-scan baseline; a second set cancels a scan part-way still-populated record index, with every record intact. The remaining
through the walk — deterministically, by counting the scan's own cancellation branches of both pools are covered by direct tests of
consultations of `ctx.Done()` rather than racing a timer — and `sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
asserts that it stops at the guard holding a partial census and a and `hashPhase`
still-populated record index, with every record intact. The - guarantee the database is closed on every fatal exit path (2026-08-09, branch
remaining cancellation branches of both pools are covered by direct `db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
tests of `sendEvent`, the walk workers, `dispatchDirs`, the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
`feedHashJobs`, `hashWorker` and `hashPhase` actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
- guarantee the database is closed on every fatal exit path `loadRecords` and `resolveRoots` return errors instead. The single exit point
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
it the SQLite WAL checkpoint — now actually runs when a subcommand runtime failure from being reported as a usage error or printing the usage
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and text. New `main_test.go` drives the CLI in-process and asserts the exit codes
`resolveRoots` return errors instead. The single exit point is `run` from README §Error handling plus the stdout/stderr split, including that a
in `main.go`: it maps a `fatalError` (anything a subcommand fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
returned) to exit 1 and cobra's own argument and flag errors to exit behind for `scan`, `report` or `trees`
2, which keeps a runtime failure from being reported as a usage - update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
error or printing the usage text. New `main_test.go` drives the CLI `golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
in-process and asserts the exit codes from README §Error handling in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
plus the stdout/stderr split, including that a fatal error raised and replaced `.golangci.yml` with the canonical file — the linter settings
after the database is open leaves no `-wal`/`-shm` sidecar behind
for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
closes #3): bumped the pinned linter in the `Dockerfile` lint
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
`.golangci.yml` with the canonical file — the linter settings
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`linters.settings` per the v2 schema, so they are actually `linters.settings` per the v2 schema, so they are actually applied; no new
applied; no new lint findings surfaced lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` - convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
entrypoints like the other managed repos (2026-07-26, commit like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
`3abeacf`, closes #1): all 12 `script/` entrypoints exist `script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`, `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`) and every Makefile target is now a thin shim over them,
`install-precommit`) and every Makefile target is now a thin shim matching the other managed repos
over them, matching the other managed repos
- make the binary the default Make target (2026-07-24, branch - make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes` `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
(previously it ran `check` plus `build`); `make build` remains as `check` plus `build`); `make build` remains as an alias
an alias - scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
- scan-wide phases, concurrent operands, batched updates (2026-07-24, `scan-wide-phases`): all operands seed the shared walk pool and every pass
branch `scan-wide-phases`): all operands seed the shared walk pool runs once over the whole scan, so totals and ETAs are scan-global; the
and every pass runs once over the whole scan, so totals and ETAs per-operand walk/hash/update cycles and their stderr announcements are gone;
are scan-global; the per-operand walk/hash/update cycles and their the update pass commits in batched transactions — the filesystem is
stderr announcements are gone; the update pass commits in batched authoritative and the database an eventually-consistent reflection, so
transactions — the filesystem is authoritative and the database an scan-level atomicity is not required
eventually-consistent reflection, so scan-level atomicity is not
required
- split the stat pass back out of the walk (2026-07-24, branch - split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk, `parallel-phases`): phases are strictly sequential again — walk, stat, hash,
stat, hash, update per operand — with parallelism only inside each update per operand — with parallelism only inside each phase; the walk
phase; the walk enumerates paths with per-directory workers and the enumerates paths with per-directory workers and the stat pass lstats them with
stat pass lstats them with per-file workers, restoring the exact per-file workers, restoring the exact total/ETA stat bar
total/ETA stat bar - announce each operand on stderr before its passes (2026-07-24, branch
- announce each operand on stderr before its passes (2026-07-24, `scan-operand-progress`): with per-operand walk/hash/update cycles, a
branch `scan-operand-progress`): with per-operand walk/hash/update multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals whole run's — an operator watching operand 3 of 14 hash 300k files concluded
that looked like the whole run's — an operator watching operand 3 of 20M files were being skipped
14 hash 300k files concluded 20M files were being skipped - parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
was a single goroutine and took hours at ~20M files on a busy pool 4h on a ZFS server); it is now a per-directory worker-pool traversal that
(observed: 22M files in 4h on a ZFS server); it is now a records size/mtime during the walk (folding away the separate stat pass,
per-directory worker-pool traversal that records size/mtime during halving metadata I/O), and each `PATH` operand commits in its own transaction
the walk (folding away the separate stat pass, halving metadata so an interrupted scan keeps completed operands
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`): - persistent scan database (2026-07-24, branch `persistent-database`): `scan`
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
Go, cgo stays disabled) keyed by absolute path that survives between disabled) keyed by absolute path that survives between runs — a rescan hashes
runs — a rescan hashes only new or changed files (by mtime/size), only new or changed files (by mtime/size), deletes records for files vanished
deletes records for files vanished from under the scanned operands, from under the scanned operands, and leaves records outside them untouched, so
and leaves records outside them untouched, so `scan` can be cronned `scan` can be cronned daily; `report` and `trees` read the database (no
daily; `report` and `trees` read the database (no positional positional arguments) instead of a scan stream. Database at
arguments) instead of a scan stream. Database at `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; journaling plus a single-transaction update keep a report run during a scan
WAL journaling plus a single-transaction update keep a report run safe
during a scan safe - add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag and push `main` plus tags (2026-07-23)
`v0.0.1`, and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root` `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
default; new `-x`/`--one-file-system` flag (GNU convention) to stop `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
at filesystem boundaries, which are crossed by default boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch - bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below) `repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on - `git init` with README-only first commit; code baseline committed on `main`
`main` (2026-07-22) (2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history) - implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps # Future Steps
- possible later features (explicitly out of scope per README): - possible later features (explicitly out of scope per README): full-content
full-content verification of candidates, removal-script helpers verification of candidates, removal-script helpers
# Repo Policy Compliance # Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
standalone formatting commit is needed. formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and - [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
`files.dat` sit untracked in the tree; needs OS/editor/Go untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
artifacts plus secrets patterns
- [x] `.editorconfig` missing - [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed - [x] `LICENSE` missing and README has no License section (MIT assumed from
from house convention — user to confirm) house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root - [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then - [x] `.golangci.yml` missing (install canonical copy); code must then pass
pass `make lint` (150 findings fixed; `make lint` is clean) `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, - [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
`fmt-check`, `docker`, `hooks`; `check` currently depends on `docker`, `hooks`; `check` currently depends on `build`, which writes the
`build`, which writes the binary (`make check` must not modify binary (`make check` must not modify files)
files) - [x] no tests — `go test ./...` has nothing to run; policy requires real tests
- [x] no tests — `go test ./...` has nothing to run; policy requires with a 30-second timeout and the conditional `-v` rerun pattern (suite
real tests with a 30-second timeout and the conditional `-v` covers parsing, grouping, digests, suppression, hashing, and the scan
rerun pattern (suite covers parsing, grouping, digests, pipeline; 64% coverage)
suppression, hashing, and the scan pipeline; 64% coverage) - [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: lint stage, build stage running `make check`
fail-fast lint stage, build stage running `make check`
- [x] `.dockerignore` missing - [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, - [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
checkout action pinned by commit SHA) action pinned by commit SHA)
- [x] README lacks required sections: Description first line - [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started, (name/purpose/category/license/author), Getting Started, Rationale, TODO,
Rationale, TODO, License, Author License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now - [x] README non-goal "no git repository setup and no CI" is stale now that the
that the repo is under git with CI repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target - [x] pre-commit hook not installed (`make hooks` once the target exists)
exists)
Accepted divergences (no action): Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine - flat single-package layout with `.go` files in the repo root — fine for a
for a small single-binary tool per the Go styleguide; the tracker small single-binary tool per the Go styleguide; the tracker audit agrees
audit agrees - `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` builds) and the race detector requires cgo
(pure-Go builds) and the race detector requires cgo