Cut the narration from TODO.md and the script and Dockerfile comments (closes #49)
check / check (push) Failing after 3s
check / check (push) Failing after 3s
Completed Steps entries keep what landed, the traps, every disclosure and every record that a check ran; the argument and history go, with bare issue numbers turned into full links. Comment blocks in script/, Dockerfile and Dockerfile.lint keep the trap and drop the defence of past decisions. TODO.md Workflow now branches from next, targets next, and leaves merging next to main to the owner. Only comments and Markdown change. Model: opus-5-5
This commit is contained in:
+37
-64
@@ -6,47 +6,36 @@ COPY go.mod go.sum ./
|
|||||||
RUN go mod download
|
RUN go mod download
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Cache-buster for the gate layers, and only for them. Docker
|
# Cache-buster for the gate layers, and only for them: on an unchanged
|
||||||
# invalidates COPY only when the copied content changes, so on an
|
# tree Docker would serve the gates below from cache and the build would
|
||||||
# unchanged tree the gates below would be served from cache and the
|
# exit 0 having run nothing. script/cibuild and script/docker pass a
|
||||||
# build would exit 0 having run nothing. script/cibuild and
|
# fresh CHECK_EPOCH; a build without one, such as a bare
|
||||||
# script/docker pass a fresh CHECK_EPOCH on every invocation. A build
|
# `docker build .`, fails at the check right after the ARG.
|
||||||
# that passes none, such as a bare `docker build .`, fails at the check
|
|
||||||
# right after the ARG instead of quietly serving the gates from cache.
|
|
||||||
#
|
#
|
||||||
# Two properties this depends on. ARG is per-stage, so the markdown and
|
# ARG is per-stage, so the markdown and build stages declare it again.
|
||||||
# build stages below declare it again; one declaration here would leave
|
# Each gate RUN must reference the value: BuildKit hashes the expanded
|
||||||
# their gates cacheable. And each gate RUN must reference the value,
|
# command, so a declared but unreferenced ARG invalidates nothing. Keep
|
||||||
# because BuildKit hashes the expanded command: a declared but
|
# it below the dependency layers so they stay cached.
|
||||||
# unreferenced ARG invalidates nothing.
|
|
||||||
#
|
|
||||||
# It sits below the dependency layers deliberately. Everything above it
|
|
||||||
# (the pinned base image, go mod download) keeps its cache; only the
|
|
||||||
# gates go cold.
|
|
||||||
ARG CHECK_EPOCH
|
ARG CHECK_EPOCH
|
||||||
RUN if [ -z "${CHECK_EPOCH}" ]; then \
|
RUN if [ -z "${CHECK_EPOCH}" ]; then \
|
||||||
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
|
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
|
||||||
exit 1; \
|
exit 1; \
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# The linter is invoked directly here, not through `make lint`. That
|
# These gates call the tools directly, not through `make lint` or
|
||||||
# target now runs `docker build -f Dockerfile.lint`, and a docker build
|
# `make fmt-check`: both run docker, which cannot run inside a docker
|
||||||
# cannot run a docker build: routing the gate through make would mean
|
# build. This step is the gofmt half of `make fmt-check`; the markdown
|
||||||
# nesting docker inside this image. Same reason `make check` is gone
|
# stage is its prettier half. gofmt's output is assigned to a variable
|
||||||
# from the build stage below, and `make fmt-check` from both stages: it
|
# first so that its own exit status, as when it cannot parse a file,
|
||||||
# runs prettier through docker too. Its gofmt half is the step below,
|
# still fails the step.
|
||||||
# its Markdown half the markdown stage further down. gofmt's output is
|
|
||||||
# assigned to a variable first so that its own exit status, as when it
|
|
||||||
# cannot parse a file, still fails the step.
|
|
||||||
RUN echo "gate gofmt, epoch ${CHECK_EPOCH}" && \
|
RUN echo "gate gofmt, epoch ${CHECK_EPOCH}" && \
|
||||||
files="$(gofmt -s -l .)" && \
|
files="$(gofmt -s -l .)" && \
|
||||||
if [ -n "$files" ]; then \
|
if [ -n "$files" ]; then \
|
||||||
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
|
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# The FROM above and the one in Dockerfile.lint pin the same linter
|
# Fails the build when the FROM above and the one in Dockerfile.lint pin
|
||||||
# twice, and nothing else keeps them in sync; this fails the build when
|
# different linter images.
|
||||||
# they disagree. See the script for why it restates neither pin.
|
|
||||||
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
|
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
|
||||||
script/verify-lint-image-pin
|
script/verify-lint-image-pin
|
||||||
|
|
||||||
@@ -93,16 +82,13 @@ FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c
|
|||||||
|
|
||||||
# We never build or run as root. Create an unprivileged user and point
|
# We never build or run as root. Create an unprivileged user and point
|
||||||
# HOME and the build cache at its home so go build and go test can write
|
# HOME and the build cache at its home so go build and go test can write
|
||||||
# it when we drop to it below. $GOPATH/bin is deliberately not on PATH:
|
# it when we drop to it below.
|
||||||
# script/bootstrap no longer `go install`s anything (the linter runs
|
|
||||||
# from a pinned image, never from a host install), so nothing lands
|
|
||||||
# there and adding it would only widen what this image resolves.
|
|
||||||
#
|
#
|
||||||
# The module cache is kept outside that home, at the base image's
|
# The module cache stays at the base image's default /go/pkg/mod and
|
||||||
# default /go/pkg/mod, and belongs to root: script/bootstrap fills it as
|
# belongs to root: script/bootstrap fills it as root. Do not move it
|
||||||
# root. Do not move it into the home and hand it over with `chown -R`:
|
# into the home and hand it over with `chown -R`: that walks every file
|
||||||
# that walks every file in it, which took from about 80 s to over ten
|
# in it, which took from about 80 s to over ten minutes on a shared
|
||||||
# minutes on a shared host, depending on load.
|
# host, depending on load.
|
||||||
RUN adduser -D -u 1000 builder
|
RUN adduser -D -u 1000 builder
|
||||||
ENV HOME=/home/builder
|
ENV HOME=/home/builder
|
||||||
ENV GOPATH=/home/builder/go
|
ENV GOPATH=/home/builder/go
|
||||||
@@ -111,24 +97,16 @@ ENV GOCACHE=/home/builder/.cache/go-build
|
|||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# No-op file copies whose only purpose is the build-graph edge: they are
|
# No-op file copies whose only purpose is the build-graph edge: they
|
||||||
# what make this stage depend on the lint and markdown stages, and so
|
# make this stage depend on the lint and markdown stages, so BuildKit
|
||||||
# what forces BuildKit to finish gofmt, the pin guard, lint and prettier
|
# finishes those gates before compilation and tests start. Remove one
|
||||||
# before compilation and tests start. Remove one and the fail-fast
|
# and the build silently stops gating on that stage and still exits 0.
|
||||||
# design dies silently — the build stops gating on that stage and still
|
|
||||||
# exits 0. The first replaces a copy of the linter binary itself, which
|
|
||||||
# is no longer wanted here: nothing in this stage runs the linter,
|
|
||||||
# because `make lint` is now a docker build and a docker build cannot
|
|
||||||
# run inside one.
|
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
COPY --from=markdown /src/go.sum /dev/null
|
COPY --from=markdown /src/go.sum /dev/null
|
||||||
|
|
||||||
# Install development prerequisites the same way a developer does,
|
# Install development prerequisites the same way a developer does. Only
|
||||||
# rather than duplicating the installs inline. Only script/ and the
|
# script/ and the dependency manifests are copied first, so this layer
|
||||||
# dependency manifests are copied first, nothing else, so this layer
|
# stays cached until they change. Bootstrap ends in `go mod download`.
|
||||||
# stays cached until the scripts or the dependencies change — bootstrap
|
|
||||||
# ends in `go mod download`, which is why there is no separate
|
|
||||||
# invocation of it here.
|
|
||||||
COPY script/ script/
|
COPY script/ script/
|
||||||
COPY go.mod go.sum ./
|
COPY go.mod go.sum ./
|
||||||
RUN script/bootstrap
|
RUN script/bootstrap
|
||||||
@@ -151,22 +129,17 @@ RUN chown builder:builder /src /go/pkg/mod/cache/download && \
|
|||||||
COPY --chown=builder:builder . .
|
COPY --chown=builder:builder . .
|
||||||
USER builder
|
USER builder
|
||||||
|
|
||||||
# Fail the build unless the branch is green. Runs as non-root so the
|
# Fail the build unless the branch is green. Runs as non-root: root
|
||||||
# permission-denied test paths are exercised legitimately (root would
|
# would bypass the chmod(0) the permission-denied tests rely on.
|
||||||
# bypass the chmod(0) the tests rely on).
|
|
||||||
#
|
#
|
||||||
# The gate is `make test`, not `make check`: that aggregate runs
|
# The gate is `make test`, not `make check`, which runs docker; lint and
|
||||||
# `script/lint` and `script/fmt-check`, which both run docker, and
|
# the format checks ran in the lint and markdown stages above. `make`,
|
||||||
# nothing inside an image build may shell out to docker. Lint and the
|
# not the script directly, because the Makefile's
|
||||||
# format checks are not skipped by this — they ran in the lint and
|
|
||||||
# markdown stages above, which this stage's COPY --from lines make
|
|
||||||
# prerequisites. `make`, not the script directly, because the Makefile's
|
|
||||||
# `export CGO_ENABLED = 0` applies only to what it invokes.
|
# `export CGO_ENABLED = 0` applies only to what it invokes.
|
||||||
#
|
#
|
||||||
# Third per-stage declaration of the gate cache-buster and its check;
|
# Third per-stage declaration of the gate cache-buster and its check;
|
||||||
# see the lint stage above for why one is not enough. It is placed after
|
# see the lint stage above. It is placed after USER so the drop to the
|
||||||
# USER so the drop to the unprivileged user still happens before the
|
# unprivileged user still happens before the checks run.
|
||||||
# checks run.
|
|
||||||
ARG CHECK_EPOCH
|
ARG CHECK_EPOCH
|
||||||
RUN if [ -z "${CHECK_EPOCH}" ]; then \
|
RUN if [ -z "${CHECK_EPOCH}" ]; then \
|
||||||
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
|
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
|
||||||
|
|||||||
+21
-35
@@ -1,14 +1,12 @@
|
|||||||
# Lint-only image: this is how the linter runs, everywhere. The repo is
|
# Lint-only image, built by script/lint: the repo is copied into the
|
||||||
# COPYed into the pinned golangci-lint image and the linter runs as a
|
# pinned golangci-lint image and the linter runs as a build step, so a
|
||||||
# build step, so a successful build IS a clean lint. golangci-lint is
|
# successful build is a clean lint. No bind mount, so it works when the
|
||||||
# never installed on a host — one toolchain, pinned by digest, identical
|
# docker daemon is remote.
|
||||||
# on a laptop and in CI — and this works even when the docker daemon is
|
|
||||||
# remote and bind mounts are impossible.
|
|
||||||
#
|
#
|
||||||
# script/lint builds this file. It is a separate image from the lint
|
# It is separate from the main Dockerfile's lint stage because
|
||||||
# stage of the main Dockerfile because script/lint must not depend on
|
# script/lint must not depend on the rest of that build; the two FROM
|
||||||
# the rest of that build; the two FROM lines are kept identical by
|
# lines are kept identical by script/verify-lint-image-pin, run as a
|
||||||
# script/verify-lint-image-pin, run as a gate below.
|
# gate below.
|
||||||
# golangci/golangci-lint:v2.12.2, 2026-08-07
|
# golangci/golangci-lint:v2.12.2, 2026-08-07
|
||||||
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
|
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
|
||||||
|
|
||||||
@@ -20,38 +18,26 @@ RUN go mod download
|
|||||||
|
|
||||||
COPY . .
|
COPY . .
|
||||||
|
|
||||||
# Cache-buster for the gate layers, and only for them. Caching of the
|
# Cache-buster for the gate layers, and only for them; caching of the
|
||||||
# lint run is waived by ruling: COPY is invalidated only by changed
|
# lint run is waived by ruling. On an unchanged tree the gates below
|
||||||
# content, so on an unchanged tree the gates below would be served from
|
# would be served from cache and this build would exit 0 in under a
|
||||||
# cache and this build would exit 0 in under a second having run no
|
# second having run no linter. script/lint passes a fresh value on every
|
||||||
# linter at all. That exact false green has bitten this repo twice
|
|
||||||
# already (#32, #39). script/lint passes a fresh value on every
|
|
||||||
# invocation.
|
# invocation.
|
||||||
#
|
#
|
||||||
# Each gate RUN must reference the value, because BuildKit hashes the
|
# Each gate RUN must reference the value: BuildKit hashes the expanded
|
||||||
# expanded command and not the ARG declaration: a declared but
|
# command, so a declared but unreferenced ARG invalidates nothing. Keep
|
||||||
# unreferenced ARG invalidates nothing. The ARG sits below the
|
# it below the dependency layers so they stay cached.
|
||||||
# dependency layers deliberately — everything above it keeps its cache,
|
|
||||||
# only the gates go cold.
|
|
||||||
ARG CHECK_EPOCH
|
ARG CHECK_EPOCH
|
||||||
|
|
||||||
# The linter version is pinned in two places, here and in the main
|
# Fails the build when the FROM above and the main Dockerfile's lint
|
||||||
# Dockerfile's lint stage. Nothing else keeps them in sync, so a
|
# stage pin different linter images.
|
||||||
# half-applied bump is a build failure; see the script.
|
|
||||||
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
|
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
|
||||||
script/verify-lint-image-pin
|
script/verify-lint-image-pin
|
||||||
|
|
||||||
# Validates .golangci.yml against golangci-lint's JSON schema. The
|
# Validates .golangci.yml against golangci-lint's JSON schema, which the
|
||||||
# concern about this step was that it fetches that schema over a live,
|
# pinned binary embeds: measured under `--network none`, it passes a
|
||||||
# unpinned HTTPS call; measured on the pinned image, it does not. The
|
# valid config and rejects an invalid one. No gate step makes a network
|
||||||
# binary carries the schema for its own version, so under
|
# call, but `go mod download` above needs the network on a cold cache.
|
||||||
# `--network none` this both passes on a valid config and still rejects
|
|
||||||
# an invalid one with the jsonschema error. That holds for the gate
|
|
||||||
# steps generally — none of them makes a network call — but not for
|
|
||||||
# this build as a whole: `go mod download` above needs the network on a
|
|
||||||
# cold cache, and under `--network none` a first build fails there
|
|
||||||
# before reaching any gate. That layer stays cached, so only a warm
|
|
||||||
# cache lints offline, until go.mod or go.sum changes.
|
|
||||||
RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
|
RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
|
||||||
golangci-lint config verify --config .golangci.yml
|
golangci-lint config verify --config .golangci.yml
|
||||||
|
|
||||||
|
|||||||
@@ -2,14 +2,15 @@
|
|||||||
|
|
||||||
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
|
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
|
||||||
tracker gets filed as an issue first
|
tracker gets filed as an issue first
|
||||||
- branch (from `main`)
|
- branch from `next`
|
||||||
- do the work, with tests, in small focused commits
|
- do the work, with tests, in small focused commits
|
||||||
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
|
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
|
||||||
as the work)
|
as the work)
|
||||||
- push the branch and open a PR whose title ends with ` (closes #N)`
|
- push the branch and open a PR against `next` whose title ends with
|
||||||
- an independent review gates the merge; every finding is addressed or
|
` (closes #N)`
|
||||||
explicitly rebutted on the PR
|
- an independent review gates each merge to `next`; every finding is addressed
|
||||||
- merge to `main` once the review passes
|
or explicitly rebutted on the PR
|
||||||
|
- only the owner merges `next` to `main`
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
@@ -28,6 +29,10 @@
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- cut the narration from `TODO.md` Completed Steps and from the comments in
|
||||||
|
`script/` and both Dockerfiles; §Workflow now branches from and merges to
|
||||||
|
`next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
|
||||||
|
|
||||||
- `make test-race` runs the test suite under the race detector in a cgo-enabled
|
- `make test-race` runs the test suite under the race detector in a cgo-enabled
|
||||||
container, outside `make check` (2026-10-04,
|
container, outside `make check` (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
||||||
@@ -121,29 +126,24 @@
|
|||||||
|
|
||||||
- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
|
- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
|
||||||
(2026-10-02, branch `next`, closes
|
(2026-10-02, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` now sends
|
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
|
||||||
`.git`, without `.git/config`, and the `Dockerfile` build stage takes the
|
without `.git/config`; the build stage stamps the `VERSION` build argument,
|
||||||
`VERSION` build argument when one is given, otherwise
|
else `git describe --tags --always`, and fails if the context carries `.git`
|
||||||
`git describe --tags --always` of that `.git`. The build fails if the context
|
and the version is still empty, `dev` or `unknown`. CI checks out the full
|
||||||
carries `.git` and the version still comes out empty, `dev` or `unknown`. The
|
history (`fetch-depth: 0`) so it stamps the same value as `make build`.
|
||||||
CI checkout step fetches the full history (`fetch-depth: 0`) so CI sees the
|
|
||||||
tag and stamps the same value as `make build`.
|
|
||||||
|
|
||||||
- replace the 1 KiB end-window sampling with the head/tail plus content-hash
|
- replace the 1 KiB end-window sampling with the head/tail plus content-hash
|
||||||
ladder (2026-09-22, branch `next`, closes
|
ladder (2026-09-22, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is hashed in
|
https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
|
||||||
full and compared directly, with no end-window step — its `head`, `tail`, and
|
documents every rung. A file under 10 MiB is hashed in full, and its `head`,
|
||||||
`content` all hold the whole-file hash. A file at 10 MiB or above gets only
|
`tail` and `content` all hold that hash. A larger file gets only its 64 KiB
|
||||||
the 64 KiB `head` and `tail` in the hash phase; a new content phase, after the
|
`head` and `tail` in the hash phase; the content phase, after the update
|
||||||
update phase, reads it for its `content` hash — the whole file below 50 MiB,
|
phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
|
||||||
gigabyte-spaced 1 MiB samples at or above — only when its size, `head`, and
|
MiB samples at or above) only when its size, `head` and `tail` match another
|
||||||
`tail` match another record's, from the same scan or stored by an earlier one,
|
record's from this scan or an earlier one, and never reads a file gone or
|
||||||
so a stored file gains its content hash when it gains a match. A file that is
|
changed since its record was written. `report` and `trees` leave out any
|
||||||
gone or has changed since its record was written is not read. The `content`
|
record without a `content` hash. The `content` column is part of the version 1
|
||||||
column is part of the version 1 schema. `report` and `trees` group by the
|
schema.
|
||||||
extended signature and leave out any record without a `content` hash, so the
|
|
||||||
ladder is applied across the whole database. README "Duplicate detection"
|
|
||||||
documents every rung including the probabilistic large-file path.
|
|
||||||
|
|
||||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
|
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
|
||||||
`.dockerignore` (2026-09-21, branch `next`, closes
|
`.dockerignore` (2026-09-21, branch `next`, closes
|
||||||
@@ -151,265 +151,162 @@
|
|||||||
|
|
||||||
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||||
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
|
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy
|
||||||
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
|
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||||
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
|
without the false `(Debian-based)` note or the tag; digest unchanged.
|
||||||
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
|
`script/verify-lint-image-pin` still matches the tagless form, and a tag on
|
||||||
parses those `FROM` lines and still matches the tagless form; its advice line
|
one side only is caught as a plain mismatch.
|
||||||
lost the now meaningless "tag and digest". With no tag in either reference, a
|
|
||||||
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
|
|
||||||
mismatch.
|
|
||||||
|
|
||||||
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
|
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
|
||||||
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
|
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
|
||||||
owner ruling, the linter runs inside a container invoked through the `script/`
|
owner ruling the linter is never installed on a host. `Dockerfile.lint` copies
|
||||||
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
|
|
||||||
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
|
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
|
||||||
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
|
runs `golangci-lint config verify` and `golangci-lint run` as build steps;
|
||||||
a successful build IS a clean lint; `script/lint` is reduced to building it.
|
`script/lint` builds it. `script/bootstrap` no longer installs or pins the
|
||||||
`script/bootstrap` loses the `go install`, the pin constants, the version
|
linter, and warns rather than fails when `docker` is absent;
|
||||||
parser and `verify_golangci_lint` outright rather than hardening them — with
|
`ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`.
|
||||||
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
|
`script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate
|
||||||
motivated them has no subject — and now warns rather than fails when `docker`
|
in both files, compares their two `FROM` lines and restates neither pin.
|
||||||
is absent. Two traps handled. A lint build on an unchanged tree returns
|
Traps: an unchanged tree lets a lint build pass in under a second having run
|
||||||
success in well under a second having run no linter, which is
|
no linter, so every gate `RUN` references `ARG CHECK_EPOCH` (BuildKit hashes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/32 and
|
the expanded command) and `script/lint` passes `"$(date +%s)-$$"`, the PID
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
|
because two runs land in the same second easily. Nothing inside an image build
|
||||||
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
|
may shell out to docker, so the `Dockerfile` lint stage calls `golangci-lint`
|
||||||
the expanded command, not the declaration) and `script/lint` passes
|
directly and the build stage runs `make test` and `make fmt-check` instead of
|
||||||
`"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
|
`make check`, through `make` because the Makefile's `export CGO_ENABLED = 0`
|
||||||
same second easily. And nothing inside an image build may shell out to docker,
|
only reaches what it invokes. `COPY --from=lint /src/go.sum /dev/null`
|
||||||
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
|
replaces the copied linter binary as the only edge making the build stage wait
|
||||||
instead of `make lint`, and its build stage runs `make test` and
|
for lint; dropping it would end fail-fast linting under a still-green build.
|
||||||
`make fmt-check` instead of the `make check` aggregate (`make`, not the
|
`golangci-lint config verify`, included per the ruling, validates from an
|
||||||
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
|
embedded schema with no network call, but `go mod download` above the gates
|
||||||
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
|
still needs the network on a cold cache. Verified: `make lint` green with no
|
||||||
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
|
`golangci-lint` on `PATH`; two back-to-back `script/lint` runs on an untouched
|
||||||
forcing BuildKit to finish linting before the build stage starts, and dropping
|
tree both ran the linter (27.7s and 28.7s in the lint step, `COPY . .`
|
||||||
it without replacing the edge would have ended fail-fast linting silently
|
`CACHED` above); a planted unused variable failed `script/lint`, and failed
|
||||||
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
|
`make docker` at `[lint 9/9]` with the build stage stopped at
|
||||||
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
|
`[builder 3/12]`; the drift guard fails on a tag-only, a digest-only and an
|
||||||
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
|
unreadable reference, naming both sides; under `--network none` config verify
|
||||||
along with its README entry, because both of its subjects ceased to exist in
|
passes a valid config and rejects an invalid one; `make docker` green in 5m35s
|
||||||
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
|
with all six gates run under one epoch (lint 37.6s, test 25.2s reporting
|
||||||
in `script/bootstrap`, and there is now neither a binary crossing between
|
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`); in the
|
||||||
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
|
builder image with the Go test cache off, `--user 0:0` still fails
|
||||||
it has moved — the linter is still pinned twice, now as the `FROM` line of
|
`TestScanHardlinkRunFailsTogether` where the unprivileged user passes. Noted
|
||||||
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
|
for follow-up, not fixed here: `golangci-lint` warns that `gomodguard` is
|
||||||
nothing syncing them, which is exactly what
|
deprecated since v2.12.0 in favour of `gomodguard_v2`.
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
|
|
||||||
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
|
|
||||||
files, which compares the two references to each other and deliberately
|
|
||||||
restates neither: a hardcoded expected digest would be a third copy and the
|
|
||||||
same drift one file further out. `golangci-lint config verify` is included per
|
|
||||||
the ruling, and the concern about its unpinned live HTTPS schema fetch was
|
|
||||||
measured rather than assumed — under `--network none` the pinned binary both
|
|
||||||
passes a valid config and rejects an invalid one with the jsonschema error, so
|
|
||||||
it validates from an embedded schema and makes no network call of its own. The
|
|
||||||
README scopes that to the gate steps rather than to linting as a whole:
|
|
||||||
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
|
|
||||||
needs the network and only a warm one lints offline. Verified: `make lint`
|
|
||||||
green with every `PATH` directory containing a `golangci-lint` removed
|
|
||||||
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
|
|
||||||
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
|
|
||||||
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
|
|
||||||
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
|
|
||||||
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
|
|
||||||
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
|
|
||||||
`script/lint` with
|
|
||||||
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
|
|
||||||
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
|
|
||||||
— `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
|
|
||||||
zero occurrences — then reverted clean. The drift guard fails on a tag-only
|
|
||||||
disagreement, on a digest-only disagreement, and on an unreadable reference,
|
|
||||||
naming both sides. `make docker` green in 5m35s with all six gates executing
|
|
||||||
under one epoch (lint 37.6s, test 25.2s reporting
|
|
||||||
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
|
|
||||||
non-root quirk still holds: in the builder image with the Go test cache off,
|
|
||||||
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
|
|
||||||
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
|
|
||||||
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
|
|
||||||
in favour of `gomodguard_v2`.
|
|
||||||
|
|
||||||
- install the Docker build stage's prerequisites by running `script/bootstrap`
|
- install the Docker build stage's prerequisites by running `script/bootstrap`
|
||||||
instead of `apk add --no-cache make` inline (2026-08-09, branch
|
instead of `apk add --no-cache make` inline (2026-08-09, branch
|
||||||
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
|
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42):
|
||||||
it, and the inline install left the build stage maintaining its own notion of
|
the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
|
||||||
the toolchain — exactly the divergence #24 exists to close, one layer down.
|
which ends in `go mod download`, so the separate call to it is gone.
|
||||||
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
|
`COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap
|
||||||
`script/bootstrap`, which ends in `go mod download`, so the separate
|
layer: it is the only edge making this stage depend on the lint stage, so
|
||||||
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
|
deleting it would end fail-fast linting silently. A new
|
||||||
and moves above the bootstrap layer. It is the only edge making this stage
|
`script/verify-linter-pin`, run in the build stage before bootstrap, fails the
|
||||||
depend on the lint stage, so deleting it as redundant would end fail-fast
|
build naming both versions unless that copied binary is the version
|
||||||
linting silently. Letting bootstrap install its own linter here would have
|
`script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip.
|
||||||
reintroduced the second toolchain and paid for a from-source build of it. What
|
`$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. Everything
|
||||||
makes the two stages provably one toolchain rather than two that happen to
|
added sits above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still
|
||||||
agree is a new `script/verify-linter-pin`, run in the build stage on the
|
precede `make check`. Verified: the guard fails the build with both versions
|
||||||
binary that arrives from the lint stage, before bootstrap: it fails the build
|
named when the lint stage's linter is faked to another version, and passes an
|
||||||
naming both versions unless that binary is the version `script/bootstrap`
|
unmodified build; bootstrap runs clean under Alpine's `sh` and `apk`, finding
|
||||||
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
|
the copied linter already at the pin; a second build served the bootstrap and
|
||||||
pin from source and then verifies whatever `PATH` resolves, so drift
|
dependency layers `CACHED` while both gates ran with a fresh epoch; a planted
|
||||||
self-heals silently and a lint stage image bumped on its own would lint at the
|
`unused` finding failed the build at the lint gate in 48.9s with the build
|
||||||
new version while `make check` ran at the old one, green. The linter version
|
stage's `make check` never starting; and the suite run in the image as
|
||||||
is pinned in two independent places (the lint stage image digest and
|
`--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the drop to the
|
||||||
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
|
unprivileged user is still needed. That last check needs the Go test cache
|
||||||
half-applied bump is now a build failure. The pin is read out of
|
off: as root it first reported `ok ... (cached)`, reusing the build-time
|
||||||
`script/bootstrap`, which stays the single source of truth; a pin that cannot
|
result. Build times on a noisy shared host: 2m13s on an unchanged tree, 2m17s
|
||||||
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
|
and 4m29s after a source change, 5m14s cold, which breaches the policy
|
||||||
only inputs are the copied binary and `script/`, so Docker invalidates the
|
ceiling; `chown -R builder:builder /src /home/builder` walks the module cache
|
||||||
layer exactly when a cached result would stop being true, and it is documented
|
and alone varied from 77s to 210s across those builds, and `main` measured
|
||||||
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
|
5m03s cold with a 209s `chown`. Filed as
|
||||||
that is where bootstrap's `go install` lands and bootstrap verifies its
|
https://git.eeqj.de/sneak/sfdupes/issues/43
|
||||||
installs against what `PATH` resolves — nothing in the image is shadowed by
|
|
||||||
it, the directory does not exist until bootstrap runs. Everything added sits
|
|
||||||
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
|
|
||||||
`make check`. Verified: the guard fails the build with both versions named
|
|
||||||
when the lint stage's linter is faked to a different version, and an
|
|
||||||
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
|
|
||||||
its `apk` branch, installing `git` and `make` and finding the copied linter
|
|
||||||
already at the pin; a second build served the bootstrap and dependency layers
|
|
||||||
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
|
|
||||||
failed the build at the lint gate in 48.9s with the build stage's `make check`
|
|
||||||
never starting; and the suite run in the image as `--user 0:0` fails
|
|
||||||
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
|
|
||||||
still load-bearing. That last check needs the Go test cache disabled — the
|
|
||||||
first attempt reported `ok ... (cached)` as root, reusing the result the
|
|
||||||
build-time run had left in the shared cache, which would have read as a pass.
|
|
||||||
Build wall time, on a shared host running many concurrent builds and so noisy:
|
|
||||||
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
|
|
||||||
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
|
|
||||||
because of this change — `chown -R builder:builder /src /home/builder` walks
|
|
||||||
the module cache and re-runs on every source change, and it alone varied
|
|
||||||
between 77s and 210s across those four builds, which is also the whole spread
|
|
||||||
in the totals. The same cold measurement against `main` is 5m03s with a 209s
|
|
||||||
`chown`. Filed as #43
|
|
||||||
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
|
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
|
||||||
`script/docker` cannot report a green they did not earn (2026-08-09, branch
|
`script/docker` cannot report a green they did not earn (2026-08-09, branch
|
||||||
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
|
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
|
||||||
invocations with no cache control, and the `Dockerfile` copies the tree before
|
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
|
||||||
running its gates, so on an unchanged tree Docker served those layers from
|
served them from cache and the build exited 0 having run nothing. Both scripts
|
||||||
cache and the build exited 0 having executed nothing. That is not hypothetical
|
now pass `--build-arg CHECK_EPOCH="$(date +%s)"`. `ARG` is per stage and the
|
||||||
here — every merge this repo has done is a non-fast-forward merge of an
|
gates span two stages, so it is declared in both; BuildKit hashes the expanded
|
||||||
undiverged branch, so each merge commit's tree is byte-identical to the branch
|
command, so each gate `RUN` echoes the epoch, which also logs it as evidence
|
||||||
head's and each merge CI run was almost certainly a full cache hit; and PR
|
the layer ran. It sits below the dependency layers so they stay cached.
|
||||||
#31's reviewer found `make docker` returning success as a 17-layer cache hit,
|
Verified under `BUILDKIT_PROGRESS=plain`, each script run twice back to back
|
||||||
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
|
on an unchanged tree: all three gates ran on all four runs with a fresh epoch
|
||||||
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
|
(`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s then 53.4s), and
|
||||||
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
|
thirteen steps were still served `CACHED`. With a planted `unused` finding the
|
||||||
across two — `make fmt-check` and `make lint` in the lint stage, `make check`
|
build failed at `make lint` in 36.1s and the build-stage `make check` never
|
||||||
in the build stage — so a single declaration would have left one stage
|
started. Run as root, the same image fails `TestScanHardlinkRunFailsTogether`,
|
||||||
silently cacheable; it is declared in both. And BuildKit hashes the expanded
|
because root reads through the `chmod(0)` the test relies on, so the build
|
||||||
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
|
stage must drop to the unprivileged `builder` user. Local fix only;
|
||||||
nothing: each gate `RUN` echoes the epoch, which also puts the value in the
|
propagating it to the canonical templates is
|
||||||
build log as evidence the layer really ran. Placement is below the dependency
|
https://git.eeqj.de/sneak/prompts/issues/26
|
||||||
layers on purpose — a build that goes cold every time would be a different
|
|
||||||
bug, not a fix. Verified by running each script twice back to back on an
|
|
||||||
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
|
|
||||||
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
|
|
||||||
61.1s; `script/docker` 61.1s then 53.4s), and thirteen steps were still served
|
|
||||||
`CACHED` in the steady state — the lint stage's `WORKDIR /src`, both
|
|
||||||
`go mod download`s, `apk add`, `adduser`, the `chown`, every `go.mod`/`go.sum`
|
|
||||||
and source copy, the linter copy out of the lint stage, and the binary copy
|
|
||||||
into the runtime stage. The lint stage still gates the build stage: with a
|
|
||||||
deliberate `unused` finding planted in the tree, the build failed at
|
|
||||||
`make lint` in 36.1s and the build-stage `make check` never started. The build
|
|
||||||
stage also still drops to the unprivileged `builder` user before `make check`,
|
|
||||||
which the suite depends on rather than merely prefers: forcing the same image
|
|
||||||
to run the tests as root fails `TestScanHardlinkRunFailsTogether`, because
|
|
||||||
root reads straight through the `chmod(0)` the test uses to prove hard links
|
|
||||||
are read once. This is the local fix only; propagating it to the canonical
|
|
||||||
templates is `prompts` #26
|
|
||||||
- check the installed golangci-lint version in `script/bootstrap` instead of
|
- check the installed golangci-lint version in `script/bootstrap` instead of
|
||||||
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
|
only its presence (2026-08-09, branch `bootstrap-version-check`, closes
|
||||||
`missing golangci-lint` meant any linter already on `PATH` satisfied the
|
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
|
||||||
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
|
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
|
||||||
on every host that already had one — this host ran v2.10.1 against a v2.12.2
|
any installed version that is not the pin — older, newer, absent or
|
||||||
pin, `make check` went green, and `make docker` then rejected the same commit
|
unparseable — is reinstalled. `go install` writes into `GOBIN` (or
|
||||||
with findings the local gate never saw. The version now lives in one place,
|
`GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
|
||||||
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
|
bootstrap re-reads the effective version after installing and, on a mismatch,
|
||||||
bump cannot half-apply; a `golangci_lint_version` helper parses
|
prints both paths and both versions and exits non-zero; it does not reorder
|
||||||
`golangci-lint --version` (taking the field after the word `version` and
|
`PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
|
||||||
tolerating an optional leading `v`, which the module ref carries and the
|
bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
|
||||||
binary's output does not), and any version that is not the pin — older, newer,
|
presence-only checks. Verified by bootstrapping this host from v2.10.1 to
|
||||||
absent or unparseable — is reinstalled. The install is then verified against
|
v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
|
||||||
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
|
covering a thirteen-input version-parse matrix, a shadowed install that must
|
||||||
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
|
exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
|
||||||
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
|
binary that must hit the timeout; `make check` and `make lint` are clean at
|
||||||
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
|
v2.12.2, so v2.10.1 was not hiding any findings on `main`
|
||||||
the install and leave the local gate disagreeing with CI under an affirmative
|
|
||||||
`bootstrap complete`. Bootstrap now re-reads the effective version after
|
|
||||||
installing and, on a mismatch, prints both paths and both versions to stderr
|
|
||||||
and exits non-zero instead of claiming success; it does not reorder anyone's
|
|
||||||
`PATH` or delete their binary. The `--version` call keeps its stderr
|
|
||||||
connected, so a present-but-broken binary says why rather than reinstalling
|
|
||||||
forever in silence, and is bounded by `timeout(1)` where that exists, so a
|
|
||||||
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
|
|
||||||
presence-only checks and now say why in a comment: they are host
|
|
||||||
package-manager tools the repo deliberately does not pin, with `go.mod`
|
|
||||||
governing the language version and the digest-pinned images covering
|
|
||||||
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
|
|
||||||
v2.12.2 and running it again to a no-op, plus stub runs of the real script
|
|
||||||
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
|
|
||||||
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
|
|
||||||
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
|
|
||||||
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
|
|
||||||
wedged binary that must hit the timeout; `make check` and `make lint` are
|
|
||||||
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
|
|
||||||
- unwind the hash worker pool on the error path (2026-08-09, branch
|
- unwind the hash worker pool on the error path (2026-08-09, branch
|
||||||
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
|
`hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
|
||||||
`recordRun` failed and abandon the pool — the feeder parked forever on a full
|
pool is now an owned, context-aware `hashPool`: every blocking send in the
|
||||||
`jobs` channel and every worker on a full `results` channel. That only stopped
|
feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
|
||||||
being invisible when #4 landed and `runScan` began unwinding instead of
|
out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
|
||||||
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
|
`results` until the last goroutine has exited — draining is what frees a
|
||||||
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
worker already parked on a send. `ctx` is threaded from `cmd.Context()`
|
||||||
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
|
through `runScan`, `syncScan`, both worker pools and the whole database layer,
|
||||||
and then drains `results` until the last goroutine has exited — draining is
|
as the first parameter everywhere. The walk pool gets the same treatment plus
|
||||||
what frees a worker already parked on a send. `ctx` is threaded from
|
a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
|
||||||
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
|
census, and every file it never reached looks vanished to the update phase.
|
||||||
database layer (it is the first parameter everywhere), so #5 can hand this
|
That phase's own `BeginTx` also fails on the cancelled context before deleting
|
||||||
path a signal and needs to add nothing else. The walk pool never leaked,
|
anything, but the guard is the barrier that still holds once an interrupted
|
||||||
because `walkPhase` always drains its events to close, but it has the same
|
scan may commit what it has. Tests drive `run(scan)` against a database whose
|
||||||
unbounded-send shape and #5 will give it an early return, so it gets the same
|
insert trigger aborts and assert that the scan fails instead of hanging and
|
||||||
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
|
that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
|
||||||
partial size census, and every file it never reached looks vanished to the
|
cancel a scan part-way through the walk, deterministically, by counting its
|
||||||
update phase. That phase's own `BeginTx` fails on the same cancelled context
|
own consultations of `ctx.Done()`, and assert that it stops at the guard
|
||||||
before deleting anything, so the guard is defence in depth rather than the
|
holding a partial census and a still-populated record index, with every record
|
||||||
only barrier — but it is the one that survives #5 deciding an interrupted scan
|
intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
|
||||||
may commit what it has. Tests drive `run(scan)` against a database whose
|
`feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
|
||||||
insert trigger aborts, and assert both that the scan fails instead of hanging
|
branches of both pools
|
||||||
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
|
|
||||||
second set cancels a scan part-way through the walk — deterministically, by
|
|
||||||
counting the scan's own consultations of `ctx.Done()` rather than racing a
|
|
||||||
timer — and asserts that it stops at the guard holding a partial census and a
|
|
||||||
still-populated record index, with every record intact. The remaining
|
|
||||||
cancellation branches of both pools are covered by direct tests of
|
|
||||||
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
|
||||||
and `hashPhase`
|
|
||||||
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
|
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
|
||||||
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
|
`db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
|
||||||
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
|
`fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
|
||||||
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
|
with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
|
||||||
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
`runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
|
||||||
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
|
errors instead. The single exit point is `run` in `main.go`: it maps a
|
||||||
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
|
`fatalError` (anything a subcommand returned) to exit 1 and cobra's own
|
||||||
runtime failure from being reported as a usage error or printing the usage
|
argument and flag errors to exit 2, which keeps a runtime failure from being
|
||||||
text. New `main_test.go` drives the CLI in-process and asserts the exit codes
|
reported as a usage error or printing the usage text. New `main_test.go`
|
||||||
from README §Error handling plus the stdout/stderr split, including that a
|
drives the CLI in-process and asserts the exit codes from README §Error
|
||||||
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
|
handling plus the stdout/stderr split, including that a fatal error raised
|
||||||
behind for `scan`, `report` or `trees`
|
after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
|
||||||
|
`report` or `trees`
|
||||||
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
|
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
|
||||||
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
|
`golangci-v2.12.2`, merged as `38a01bd`, closes
|
||||||
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
|
https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
|
||||||
and replaced `.golangci.yml` with the canonical file — the linter settings
|
`Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
|
||||||
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
|
||||||
`linters.settings` per the v2 schema, so they are actually applied; no new
|
`funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
|
||||||
lint findings surfaced
|
the v2 schema, so they are actually applied; no new lint findings surfaced
|
||||||
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
|
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
|
||||||
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
|
like the other managed repos (2026-07-26, commit `3abeacf`, closes
|
||||||
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
|
https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
|
||||||
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
||||||
`install-precommit`) and every Makefile target is now a thin shim over them,
|
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
|
||||||
matching the other managed repos
|
and every Makefile target is now a thin shim over them
|
||||||
- make the binary the default Make target (2026-07-24, branch
|
- make the binary the default Make target (2026-07-24, branch
|
||||||
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
|
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
|
||||||
`check` plus `build`); `make build` remains as an alias
|
`check` plus `build`); `make build` remains as an alias
|
||||||
@@ -428,8 +325,7 @@
|
|||||||
- announce each operand on stderr before its passes (2026-07-24, branch
|
- announce each operand on stderr before its passes (2026-07-24, branch
|
||||||
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
|
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
|
||||||
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
|
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
|
||||||
whole run's — an operator watching operand 3 of 14 hash 300k files concluded
|
whole run's
|
||||||
20M files were being skipped
|
|
||||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
|
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
|
||||||
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
|
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
|
||||||
4h on a ZFS server); it is now a per-directory worker-pool traversal that
|
4h on a ZFS server); it is now a per-directory worker-pool traversal that
|
||||||
|
|||||||
+9
-21
@@ -1,14 +1,9 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/bootstrap: install all dependencies needed to build and develop
|
# script/bootstrap: install all dependencies needed to build and develop
|
||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent; assumes nothing is present (not git, make, or
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# go). Base tooling comes from nix, apt, brew, or apk (detected in that
|
||||||
# or apk (detected in that order); assumes nothing is present (not git,
|
# order). golangci-lint and prettier are never installed: they run via
|
||||||
# make, or go). Neither the linter nor the Markdown formatter is
|
# docker only (script/lint, script/fmt, script/fmt-check).
|
||||||
# installed: golangci-lint (script/lint) and prettier (script/fmt,
|
|
||||||
# script/fmt-check) run via docker only, pinned by hash, so their only
|
|
||||||
# prerequisite is a working docker — which is warned about, not
|
|
||||||
# installed, because everything except linting, formatting and
|
|
||||||
# make test-race works without it.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
@@ -63,22 +58,15 @@ missing() {
|
|||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
|
|
||||||
# System tooling, deliberately unpinned: these come from the host
|
# Deliberately unpinned, so presence is the whole check: go.mod
|
||||||
# package manager and whatever version it ships is what the host
|
# governs the Go version, and reproducible builds run in the
|
||||||
# gets, so a presence check is the right check. The repo pins no
|
# digest-pinned Docker images.
|
||||||
# system toolchain versions — the Go language version is governed by
|
|
||||||
# go.mod, and builds that must be reproducible run in the Docker
|
|
||||||
# image, whose base images are pinned by digest.
|
|
||||||
if missing git; then pkg_install git git git git; fi
|
if missing git; then pkg_install git git git git; fi
|
||||||
if missing make; then pkg_install gnumake make make make; fi
|
if missing make; then pkg_install gnumake make make make; fi
|
||||||
if missing go; then pkg_install go golang go go; fi
|
if missing go; then pkg_install go golang go go; fi
|
||||||
|
|
||||||
# Linting and Markdown formatting run via docker only, so docker is
|
# Warn, do not fail: only the targets named below, and the
|
||||||
# their prerequisite rather than something bootstrap installs. Warn,
|
# pre-commit hook, need docker.
|
||||||
# do not fail: everything except `make lint`, `make fmt`,
|
|
||||||
# `make fmt-check` and `make test-race` — and, through them,
|
|
||||||
# `make check`, `make docker` and the pre-commit hook — works
|
|
||||||
# without it.
|
|
||||||
if missing docker; then
|
if missing docker; then
|
||||||
echo "bootstrap: WARNING: docker not found; make lint, make fmt," >&2
|
echo "bootstrap: WARNING: docker not found; make lint, make fmt," >&2
|
||||||
echo "bootstrap: make fmt-check, make check, make docker and" >&2
|
echo "bootstrap: make fmt-check, make check, make docker and" >&2
|
||||||
|
|||||||
+6
-24
@@ -1,30 +1,12 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/cibuild: run the CI build. The Gitea workflow runs this on
|
# script/cibuild: run the CI build. The Gitea workflow runs this on
|
||||||
# push.
|
# push. The Dockerfile runs every gate make check runs, as build steps,
|
||||||
|
# so a successful build means the repo is green.
|
||||||
#
|
#
|
||||||
# The Dockerfile runs the gates individually as build steps, not the
|
# Without a fresh CHECK_EPOCH, a rebuild of an unchanged checkout serves
|
||||||
# make check aggregate: the lint stage runs the gofmt check,
|
# the gate layers from cache and passes having run none of them. The
|
||||||
# script/verify-lint-image-pin, golangci-lint config verify and
|
# process id goes in with the epoch so two runs started in the same
|
||||||
# golangci-lint run; the markdown stage runs the prettier check; the
|
# second still differ.
|
||||||
# build stage, dropped to an unprivileged user, runs make test. None of
|
|
||||||
# make lint, make fmt-check or make check appears, because each runs
|
|
||||||
# docker, and docker cannot run inside a docker build. Nothing is
|
|
||||||
# skipped by that — the linter, gofmt and prettier are invoked directly
|
|
||||||
# in their stages, and the build stage's COPY --from lines make those
|
|
||||||
# stages prerequisites, so BuildKit must finish them first. Between the
|
|
||||||
# three stages everything make check would run has run, which is why a
|
|
||||||
# successful build here implies the repo is green.
|
|
||||||
#
|
|
||||||
# That implication holds only because of CHECK_EPOCH. A COPY layer is
|
|
||||||
# invalidated only by changed content, and a rebuild of an unchanged
|
|
||||||
# checkout sends the same content, so without a fresh value here Docker
|
|
||||||
# serves the gate layers from cache and the build reports a green it
|
|
||||||
# never earned. Passing the current epoch invalidates the gate
|
|
||||||
# layers on every run while leaving the pinned base images and
|
|
||||||
# go mod download cached; see the Dockerfile for the placement. The
|
|
||||||
# process id goes in with the epoch so that two runs started in the
|
|
||||||
# same second still get different values, the same form script/lint
|
|
||||||
# uses.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|||||||
+4
-9
@@ -1,14 +1,9 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/docker: build the Docker image tagged with the project name.
|
# script/docker: build the Docker image tagged with the project name.
|
||||||
# The tag comes from script/projectname.
|
# The tag comes from script/projectname. CHECK_EPOCH is passed for the
|
||||||
#
|
# same reason script/cibuild passes it: without a fresh value an
|
||||||
# CHECK_EPOCH is passed for the same reason script/cibuild passes it:
|
# unchanged tree is served from cache and this exits 0 having run no
|
||||||
# without a fresh value Docker serves the gate layers from cache on an
|
# gate.
|
||||||
# unchanged tree and this exits 0 having run none of the lint stage's
|
|
||||||
# gates, the markdown stage's prettier gate or the builder stage's test
|
|
||||||
# gate. This is the set of gates a developer or reviewer runs by hand,
|
|
||||||
# so a cached pass here is the most misleading result the repo can
|
|
||||||
# produce. Dependency layers sit above the ARG and stay cached.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|||||||
+10
-16
@@ -1,23 +1,17 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/lint: run the linter. golangci-lint is never installed on a
|
# script/lint: run the linter. golangci-lint is never installed on a
|
||||||
# host: it runs via docker only, one way, everywhere — this builds
|
# host: this builds Dockerfile.lint, which copies the repo into the
|
||||||
# Dockerfile.lint, which COPYs the repo into the digest-pinned
|
# digest-pinned golangci-lint image and lints as a build step, so a
|
||||||
# golangci-lint image and lints as a build step, so a successful build
|
# successful build is a clean lint. A cold cache needs the network to
|
||||||
# is a clean lint. The only prerequisite is a working docker. The gate
|
# pull the image and for `go mod download`; once warm this runs offline
|
||||||
# steps make no network calls of their own, but Dockerfile.lint runs
|
# until go.mod or go.sum changes.
|
||||||
# `go mod download` above them, so a cold cache does reach the network
|
|
||||||
# (as does pulling the pinned image); that layer stays cached, and once
|
|
||||||
# it is warm this runs offline until go.mod or go.sum changes.
|
|
||||||
#
|
#
|
||||||
# CHECK_EPOCH is what makes the result mean anything. Without it docker
|
# Without a fresh CHECK_EPOCH docker serves the gate layers from cache
|
||||||
# serves the gate layers from cache on an unchanged tree and this exits
|
# on an unchanged tree and this exits 0 having run no linter. The PID is
|
||||||
# 0 in well under a second having run no linter. The PID is in the value
|
# in the value because two lint runs land inside the same second easily.
|
||||||
# as well as the epoch because two lint runs land inside the same second
|
|
||||||
# easily, and `date +%s` alone would cache the second one.
|
|
||||||
#
|
#
|
||||||
# The result is the build's exit status and the image is never used, so
|
# The image is never used, so --output=type=cacheonly writes none;
|
||||||
# --output=type=cacheonly writes none. Without it every run spends
|
# without it every run leaves an untagged image behind.
|
||||||
# seconds exporting an image and leaves it behind untagged.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|||||||
@@ -2,23 +2,16 @@
|
|||||||
# script/verify-lint-image-pin: fail unless the golangci-lint image
|
# script/verify-lint-image-pin: fail unless the golangci-lint image
|
||||||
# referenced by Dockerfile.lint and the one referenced by the main
|
# referenced by Dockerfile.lint and the one referenced by the main
|
||||||
# Dockerfile's lint stage are the same image at the same digest. Our own
|
# Dockerfile's lint stage are the same image at the same digest. Our own
|
||||||
# extension to scripts-to-rule-them-all, not one of its entrypoints.
|
# extension to scripts-to-rule-them-all, not one of its entrypoints; run
|
||||||
|
# as a gate in both files. Nothing else keeps the two pins in sync, and
|
||||||
|
# a bump applied to one alone would lint the same tree against different
|
||||||
|
# rulesets, both green.
|
||||||
#
|
#
|
||||||
# The linter version is pinned in two independent files. That is the
|
# Do not hardcode the expected digest here: that is a third copy to keep
|
||||||
# shape #42 turned into a build failure rather than tolerate: nothing
|
# in sync.
|
||||||
# else keeps the two in sync, and a bump applied to one file alone would
|
|
||||||
# leave `make lint` and the fail-fast lint stage of `make docker`
|
|
||||||
# linting the same tree against different rulesets, both green. This is
|
|
||||||
# the single guard that stops it, run as a gate in both files.
|
|
||||||
#
|
#
|
||||||
# It deliberately restates neither pin. A hardcoded expected digest here
|
# A reference that cannot be read is a hard failure, not a skip: two
|
||||||
# would be a third copy — one more thing to bump, and the same drift one
|
# empty strings compare equal.
|
||||||
# file further out. It compares the two files to each other and knows
|
|
||||||
# nothing about which version is correct.
|
|
||||||
#
|
|
||||||
# A reference that cannot be read is a hard failure, not a skip: a
|
|
||||||
# comparison of two empty strings succeeds, which would turn this guard
|
|
||||||
# into exactly the unearned green it exists to prevent.
|
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|||||||
Reference in New Issue
Block a user