Format Markdown with prettier in make fmt and make fmt-check (closes #19)
check / check (push) Failing after 3s
check / check (push) Failing after 3s
script/fmt and script/fmt-check run prettier over every Markdown file again, next to gofmt. prettier is pinned by hash through package.json and yarn.lock, copied from the prompts repo with .prettierrc and .prettierignore, and is never installed on a host: a new prettier stage of the Dockerfile installs it into a digest-pinned node image, and both scripts build that stage and run it with the repository mounted. CI checks the Markdown in a markdown stage that the build stage waits on. Because make fmt-check now runs docker, the Dockerfile runs gofmt directly in its lint stage instead. All Markdown is reformatted. Model: opus-5-5
This commit was merged in pull request #84.
This commit is contained in:
@@ -0,0 +1,2 @@
|
|||||||
|
node_modules/
|
||||||
|
yarn.lock
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"tabWidth": 4,
|
||||||
|
"proseWrap": "always"
|
||||||
|
}
|
||||||
+54
-21
@@ -12,9 +12,9 @@ COPY . .
|
|||||||
# build would exit 0 having run nothing. script/cibuild and
|
# build would exit 0 having run nothing. script/cibuild and
|
||||||
# script/docker pass a fresh CHECK_EPOCH on every invocation.
|
# script/docker pass a fresh CHECK_EPOCH on every invocation.
|
||||||
#
|
#
|
||||||
# Two properties this depends on. ARG is per-stage, so the build stage
|
# Two properties this depends on. ARG is per-stage, so the markdown and
|
||||||
# below declares it again; one declaration here would leave that
|
# build stages below declare it again; one declaration here would leave
|
||||||
# stage's gate cacheable. And each gate RUN must reference the value,
|
# their gates cacheable. And each gate RUN must reference the value,
|
||||||
# because BuildKit hashes the expanded command: a declared but
|
# because BuildKit hashes the expanded command: a declared but
|
||||||
# unreferenced ARG invalidates nothing.
|
# unreferenced ARG invalidates nothing.
|
||||||
#
|
#
|
||||||
@@ -27,9 +27,16 @@ ARG CHECK_EPOCH
|
|||||||
# target now runs `docker build -f Dockerfile.lint`, and a docker build
|
# target now runs `docker build -f Dockerfile.lint`, and a docker build
|
||||||
# cannot run a docker build: routing the gate through make would mean
|
# cannot run a docker build: routing the gate through make would mean
|
||||||
# nesting docker inside this image. Same reason `make check` is gone
|
# nesting docker inside this image. Same reason `make check` is gone
|
||||||
# from the build stage below. `make fmt-check` stays as it is — it is a
|
# from the build stage below, and `make fmt-check` from both stages: it
|
||||||
# gate, not the aggregate, and it shells out to nothing.
|
# runs prettier through docker too. Its gofmt half is the step below,
|
||||||
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
|
# its Markdown half the markdown stage further down. gofmt's output is
|
||||||
|
# assigned to a variable first so that its own exit status, as when it
|
||||||
|
# cannot parse a file, still fails the step.
|
||||||
|
RUN echo "gate gofmt, epoch ${CHECK_EPOCH}" && \
|
||||||
|
files="$(gofmt -s -l .)" && \
|
||||||
|
if [ -n "$files" ]; then \
|
||||||
|
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
|
||||||
|
fi
|
||||||
|
|
||||||
# The FROM above and the one in Dockerfile.lint pin the same linter
|
# The FROM above and the one in Dockerfile.lint pin the same linter
|
||||||
# twice, and nothing else keeps them in sync; this fails the build when
|
# twice, and nothing else keeps them in sync; this fails the build when
|
||||||
@@ -46,6 +53,30 @@ RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
|
|||||||
RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
|
RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
|
||||||
golangci-lint run --config .golangci.yml ./...
|
golangci-lint run --config .golangci.yml ./...
|
||||||
|
|
||||||
|
# Prettier stage: the prettier that formats this repository's Markdown,
|
||||||
|
# never installed on a host. script/fmt and script/fmt-check build this
|
||||||
|
# stage alone and run it with the repository mounted on /src. prettier
|
||||||
|
# is installed in /tools so that the repository, mounted or copied onto
|
||||||
|
# /src, cannot hide it.
|
||||||
|
# node:22-alpine, 2026-02-22
|
||||||
|
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS prettier
|
||||||
|
WORKDIR /tools
|
||||||
|
# yarn.lock pins prettier by hash, and --frozen-lockfile fails rather
|
||||||
|
# than install anything yarn.lock does not name.
|
||||||
|
COPY package.json yarn.lock ./
|
||||||
|
RUN yarn install --frozen-lockfile
|
||||||
|
ENV PATH=/tools/node_modules/.bin:$PATH
|
||||||
|
WORKDIR /src
|
||||||
|
|
||||||
|
# Markdown stage: the Markdown half of `make fmt-check`, as a gate.
|
||||||
|
FROM prettier AS markdown
|
||||||
|
COPY . .
|
||||||
|
# Second per-stage declaration of the gate cache-buster; see the lint
|
||||||
|
# stage above.
|
||||||
|
ARG CHECK_EPOCH
|
||||||
|
RUN echo "gate prettier, epoch ${CHECK_EPOCH}" && \
|
||||||
|
prettier --check '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
|
|
||||||
# Build stage
|
# Build stage
|
||||||
# golang:1.25-alpine, 2026-07-23
|
# golang:1.25-alpine, 2026-07-23
|
||||||
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
|
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
|
||||||
@@ -70,15 +101,17 @@ ENV GOCACHE=/home/builder/.cache/go-build
|
|||||||
|
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
|
|
||||||
# No-op file copy whose only purpose is the build-graph edge: it is what
|
# No-op file copies whose only purpose is the build-graph edge: they are
|
||||||
# makes this stage depend on the lint stage, and so what forces BuildKit
|
# what make this stage depend on the lint and markdown stages, and so
|
||||||
# to finish fmt-check, the pin guard and lint before compilation and
|
# what forces BuildKit to finish gofmt, the pin guard, lint and prettier
|
||||||
# tests start. Remove it and the fail-fast design dies silently — the
|
# before compilation and tests start. Remove one and the fail-fast
|
||||||
# build stops gating on lint and still exits 0. It replaces a copy of
|
# design dies silently — the build stops gating on that stage and still
|
||||||
# the linter binary itself, which is no longer wanted here: nothing in
|
# exits 0. The first replaces a copy of the linter binary itself, which
|
||||||
# this stage runs the linter, because `make lint` is now a docker build
|
# is no longer wanted here: nothing in this stage runs the linter,
|
||||||
# and a docker build cannot run inside one.
|
# because `make lint` is now a docker build and a docker build cannot
|
||||||
|
# run inside one.
|
||||||
COPY --from=lint /src/go.sum /dev/null
|
COPY --from=lint /src/go.sum /dev/null
|
||||||
|
COPY --from=markdown /src/go.sum /dev/null
|
||||||
|
|
||||||
# Install development prerequisites the same way a developer does,
|
# Install development prerequisites the same way a developer does,
|
||||||
# rather than duplicating the installs inline. Only script/ and the
|
# rather than duplicating the installs inline. Only script/ and the
|
||||||
@@ -112,19 +145,19 @@ USER builder
|
|||||||
# permission-denied test paths are exercised legitimately (root would
|
# permission-denied test paths are exercised legitimately (root would
|
||||||
# bypass the chmod(0) the tests rely on).
|
# bypass the chmod(0) the tests rely on).
|
||||||
#
|
#
|
||||||
# The gates are the individual targets, not `make check`: that aggregate
|
# The gate is `make test`, not `make check`: that aggregate runs
|
||||||
# runs `script/lint`, which is now a docker build, and nothing inside an
|
# `script/lint` and `script/fmt-check`, which both run docker, and
|
||||||
# image build may shell out to docker. Lint is not skipped by this — it
|
# nothing inside an image build may shell out to docker. Lint and the
|
||||||
# ran in the lint stage above, which this stage's COPY --from makes a
|
# format checks are not skipped by this — they ran in the lint and
|
||||||
# prerequisite. `make`, not the scripts directly, because the Makefile's
|
# markdown stages above, which this stage's COPY --from lines make
|
||||||
|
# prerequisites. `make`, not the script directly, because the Makefile's
|
||||||
# `export CGO_ENABLED = 0` applies only to what it invokes.
|
# `export CGO_ENABLED = 0` applies only to what it invokes.
|
||||||
#
|
#
|
||||||
# Second per-stage declaration of the gate cache-buster; see the lint
|
# Third per-stage declaration of the gate cache-buster; see the lint
|
||||||
# stage above for why one is not enough. It is placed after USER so the
|
# stage above for why one is not enough. It is placed after USER so the
|
||||||
# drop to the unprivileged user still happens before the checks run.
|
# drop to the unprivileged user still happens before the checks run.
|
||||||
ARG CHECK_EPOCH
|
ARG CHECK_EPOCH
|
||||||
RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
|
RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
|
||||||
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
|
|
||||||
|
|
||||||
# The version stamped into the binary: the VERSION build argument when
|
# The version stamped into the binary: the VERSION build argument when
|
||||||
# one is given, otherwise `git describe --tags --always` of the .git in
|
# one is given, otherwise `git describe --tags --always` of the .git in
|
||||||
|
|||||||
@@ -1,60 +1,62 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
- take an issue from the `1.0.0` milestone on the tracker; work not
|
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
|
||||||
yet on the tracker gets filed as an issue first
|
tracker gets filed as an issue first
|
||||||
- branch (from `main`)
|
- branch (from `main`)
|
||||||
- do the work, with tests, in small focused commits
|
- do the work, with tests, in small focused commits
|
||||||
- record it at the top of Completed Steps (`TODO.md` changes in the
|
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
|
||||||
same commit as the work)
|
as the work)
|
||||||
- push the branch and open a PR whose title ends with
|
- push the branch and open a PR whose title ends with ` (closes #N)`
|
||||||
` (closes #N)`
|
- an independent review gates the merge; every finding is addressed or
|
||||||
- an independent review gates the merge; every finding is addressed
|
explicitly rebutted on the PR
|
||||||
or explicitly rebutted on the PR
|
|
||||||
- merge to `main` once the review passes
|
- merge to `main` once the review passes
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
- pre-1.0
|
- pre-1.0
|
||||||
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
|
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
|
||||||
open issues under the `1.0.0` milestone are what remains before
|
under the `1.0.0` milestone are what remains before the tag, and this file
|
||||||
the tag, and this file records history and process, not the queue
|
records history and process, not the queue
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
- take the next issue from the `1.0.0` milestone on the tracker:
|
- take the next issue from the `1.0.0` milestone on the tracker:
|
||||||
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
|
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
|
||||||
the source of truth for what is left before 1.0.0. Individual
|
of truth for what is left before 1.0.0. Individual issues are deliberately not
|
||||||
issues are deliberately not restated here; a copy in this file
|
restated here; a copy in this file drifts out of date the moment the tracker
|
||||||
drifts out of date the moment the tracker moves
|
moves
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- `script/lint` writes no image, so a run no longer leaves an untagged
|
- `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and
|
||||||
one behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
CI checks it; all Markdown reformatted (2026-10-04,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||||
|
|
||||||
- tests cover a missing database, `scan` keeping stdout empty, its skip
|
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||||
warning, the `report` and `trees` summary lines, and every subcommand
|
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
||||||
going through `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
|
||||||
|
- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
|
||||||
|
the `report` and `trees` summary lines, and every subcommand going through
|
||||||
|
`runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||||
|
|
||||||
- `.golangci.yml` replaced with the current canonical copy, which uses
|
- `.golangci.yml` replaced with the current canonical copy, which uses
|
||||||
`gomodguard_v2`, so lint no longer prints a deprecation warning
|
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/26)
|
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||||
|
|
||||||
- a test fails when either `hashWorker` cancellation check in `scan.go`
|
- a test fails when either `hashWorker` cancellation check in `scan.go` is
|
||||||
is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
|
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||||
|
|
||||||
- a database path holding `?`, `#` or `%` opens exactly the file it names
|
- a database path holding `?`, `#` or `%` opens exactly the file it names
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
||||||
|
|
||||||
- `scan` rejects `--workers` below 1 as a usage error instead of
|
- `scan` rejects `--workers` below 1 as a usage error instead of running
|
||||||
running single-threaded (2026-10-04,
|
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/10)
|
|
||||||
|
|
||||||
- a test fails when either walk cancellation check in `scan.go` is
|
- a test fails when either walk cancellation check in `scan.go` is removed
|
||||||
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||||
|
|
||||||
- test that `scan` refuses a database with another schema version
|
- test that `scan` refuses a database with another schema version (2026-10-04,
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||||
|
|
||||||
- correct four inaccurate comments in `cancel_test.go` and rename
|
- correct four inaccurate comments in `cancel_test.go` and rename
|
||||||
`walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
|
`walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
|
||||||
@@ -67,479 +69,428 @@
|
|||||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||||
|
|
||||||
- README documents install, Docker, a daily cron scan and how to read
|
- README documents install, Docker, a daily cron scan and how to read and check
|
||||||
and check the reports (2026-10-04,
|
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/54)
|
|
||||||
|
|
||||||
- the `Dockerfile` build stage keeps the Go module cache out of `builder`'s
|
- the `Dockerfile` build stage keeps the Go module cache out of `builder`'s home
|
||||||
home and copies the sources with `--chown`, so no `chown -R` walks them
|
and copies the sources with `--chown`, so no `chown -R` walks them
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)
|
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)
|
||||||
|
|
||||||
- `--version` prints `sfdupes VERSION` to stdout; README documents it and
|
- `--version` prints `sfdupes VERSION` to stdout; README documents it and
|
||||||
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
|
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
|
||||||
|
|
||||||
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has
|
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
|
||||||
hashed, deletes nothing more, exits 1 (2026-10-04,
|
deletes nothing more, exits 1 (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/5)
|
https://git.eeqj.de/sneak/sfdupes/issues/5)
|
||||||
|
|
||||||
- `report` and `trees` stream the records instead of holding them all in
|
- `report` and `trees` stream the records instead of holding them all in memory;
|
||||||
memory; the schema gains the `files_signature` index (2026-10-04,
|
the schema gains the `files_signature` index (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
||||||
|
|
||||||
- progress prints at once on a non-terminal, uses a real terminal test,
|
- progress prints at once on a non-terminal, uses a real terminal test, and
|
||||||
and prints warnings through a spinner instead of racing its redraw
|
prints warnings through a spinner instead of racing its redraw (2026-10-03,
|
||||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/13)
|
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||||
|
|
||||||
- warn about and skip symlink, socket, FIFO, device and `.zfs`
|
- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
|
||||||
operands, keeping the records beneath them (2026-10-03,
|
the records beneath them (2026-10-03,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/9)
|
https://git.eeqj.de/sneak/sfdupes/issues/9)
|
||||||
|
|
||||||
- `scan` holds a lock on a lock file beside the database for its whole run,
|
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
||||||
so a second `scan` fails at once with exit 1 (2026-10-03,
|
second `scan` fails at once with exit 1 (2026-10-03,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/53)
|
https://git.eeqj.de/sneak/sfdupes/issues/53)
|
||||||
|
|
||||||
- test stdout write failures in `report` and `trees`; README states that
|
- test stdout write failures in `report` and `trees`; README states that
|
||||||
`| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
|
`| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
|
||||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
|
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
|
||||||
|
|
||||||
- `report` and `trees` open the database read-only, and `scan` leaves it
|
- `report` and `trees` open the database read-only, and `scan` leaves it out of
|
||||||
out of WAL mode, so reading needs only read access (2026-10-03, closes
|
WAL mode, so reading needs only read access (2026-10-03, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
||||||
|
|
||||||
- escape tabs, newlines, carriage returns and backslashes in report,
|
- escape tabs, newlines, carriage returns and backslashes in report, trees and
|
||||||
trees and warning paths; the root directory's path is `/`
|
warning paths; the root directory's path is `/` (2026-10-03,
|
||||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/7)
|
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||||
|
|
||||||
- stamp the git tag or short commit in a plain `docker build .`
|
- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
|
||||||
instead of `dev` (2026-10-02, branch `next`, closes
|
(2026-10-02, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` now
|
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` now sends
|
||||||
sends `.git`, without `.git/config`, and the `Dockerfile` build
|
`.git`, without `.git/config`, and the `Dockerfile` build stage takes the
|
||||||
stage takes the `VERSION` build argument when one is given,
|
`VERSION` build argument when one is given, otherwise
|
||||||
otherwise `git describe --tags --always` of that `.git`. The build
|
`git describe --tags --always` of that `.git`. The build fails if the context
|
||||||
fails if the context carries `.git` and the version still comes out
|
carries `.git` and the version still comes out empty, `dev` or `unknown`. The
|
||||||
empty, `dev` or `unknown`. The CI checkout step fetches the full
|
CI checkout step fetches the full history (`fetch-depth: 0`) so CI sees the
|
||||||
history (`fetch-depth: 0`) so CI sees the tag and stamps the same
|
tag and stamps the same value as `make build`.
|
||||||
value as `make build`.
|
|
||||||
|
|
||||||
- replace the 1 KiB end-window sampling with the head/tail plus
|
- replace the 1 KiB end-window sampling with the head/tail plus content-hash
|
||||||
content-hash ladder (2026-09-22, branch `next`, closes
|
ladder (2026-09-22, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
|
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is hashed in
|
||||||
hashed in full and compared directly, with no end-window step — its
|
full and compared directly, with no end-window step — its `head`, `tail`, and
|
||||||
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
`content` all hold the whole-file hash. A file at 10 MiB or above gets only
|
||||||
10 MiB or above gets only the 64 KiB `head` and `tail` in the hash
|
the 64 KiB `head` and `tail` in the hash phase; a new content phase, after the
|
||||||
phase; a new content phase, after the update phase, reads it for its
|
update phase, reads it for its `content` hash — the whole file below 50 MiB,
|
||||||
`content` hash — the whole file below 50 MiB, gigabyte-spaced 1 MiB
|
gigabyte-spaced 1 MiB samples at or above — only when its size, `head`, and
|
||||||
samples at or above — only when its size, `head`, and `tail` match
|
`tail` match another record's, from the same scan or stored by an earlier one,
|
||||||
another record's, from the same scan or stored by an earlier one, so
|
so a stored file gains its content hash when it gains a match. A file that is
|
||||||
a stored file gains its content hash when it gains a match. A file
|
gone or has changed since its record was written is not read. The `content`
|
||||||
that is gone or has changed since its record was written is not
|
column is part of the version 1 schema. `report` and `trees` group by the
|
||||||
read. The `content` column is part of the version 1 schema. `report`
|
extended signature and leave out any record without a `content` hash, so the
|
||||||
and `trees` group by the extended signature and leave out any record
|
ladder is applied across the whole database. README "Duplicate detection"
|
||||||
without a `content` hash, so the ladder is applied across the whole
|
documents every rung including the probabilistic large-file path.
|
||||||
database. README "Duplicate detection" documents every rung including
|
|
||||||
the probabilistic large-file path.
|
|
||||||
|
|
||||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
|
||||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
`.dockerignore` (2026-09-21, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||||
|
|
||||||
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||||
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
|
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
|
||||||
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the
|
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
|
||||||
redundant tag, so both pins are the policy `# image:vX.Y.Z,
|
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
|
||||||
YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest
|
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
|
||||||
unchanged. `script/verify-lint-image-pin` parses those `FROM` lines
|
parses those `FROM` lines and still matches the tagless form; its advice line
|
||||||
and still matches the tagless form; its advice line lost the now
|
lost the now meaningless "tag and digest". With no tag in either reference, a
|
||||||
meaningless "tag and digest". With no tag in either reference, a
|
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
|
||||||
tag-only disagreement no longer exists — a one-sided tag is caught as
|
mismatch.
|
||||||
a plain mismatch.
|
|
||||||
|
|
||||||
- run all linting in Docker via `Dockerfile.lint` and `script/lint`
|
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
|
||||||
(2026-08-10, branch `next`, closes
|
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
|
owner ruling, the linter runs inside a container invoked through the `script/`
|
||||||
linter runs inside a container invoked through the `script/`
|
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
|
||||||
entrypoint and is never installed on a host. New root
|
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
|
||||||
`Dockerfile.lint` COPYs the repo into the digest-pinned
|
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
|
||||||
`golangci/golangci-lint:v2.12.2` image and runs
|
a successful build IS a clean lint; `script/lint` is reduced to building it.
|
||||||
`golangci-lint config verify` and `golangci-lint run` as build
|
`script/bootstrap` loses the `go install`, the pin constants, the version
|
||||||
steps, so a successful build IS a clean lint; `script/lint` is
|
parser and `verify_golangci_lint` outright rather than hardening them — with
|
||||||
reduced to building it. `script/bootstrap` loses the `go install`,
|
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
|
||||||
the pin constants, the version parser and `verify_golangci_lint`
|
motivated them has no subject — and now warns rather than fails when `docker`
|
||||||
outright rather than hardening them — with nothing linting on the
|
is absent. Two traps handled. A lint build on an unchanged tree returns
|
||||||
host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
|
success in well under a second having run no linter, which is
|
||||||
no subject — and now warns rather than fails when `docker` is absent.
|
|
||||||
Two traps handled. A lint build on an unchanged tree returns success
|
|
||||||
in well under a second having run no linter, which is
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/32 and
|
https://git.eeqj.de/sneak/sfdupes/issues/32 and
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
|
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
|
||||||
`Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
|
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
|
||||||
inside every gate `RUN` (BuildKit hashes the expanded command, not
|
the expanded command, not the declaration) and `script/lint` passes
|
||||||
the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
|
`"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
|
||||||
PID matters because two lint runs land inside the same second easily.
|
same second easily. And nothing inside an image build may shell out to docker,
|
||||||
And nothing inside an image build may shell out to docker, so the
|
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
|
||||||
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
|
|
||||||
instead of `make lint`, and its build stage runs `make test` and
|
instead of `make lint`, and its build stage runs `make test` and
|
||||||
`make fmt-check` instead of the `make check` aggregate (`make`, not
|
`make fmt-check` instead of the `make check` aggregate (`make`, not the
|
||||||
the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
|
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
|
||||||
only reaches what it invokes). `COPY --from=lint`
|
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
|
||||||
`/usr/bin/golangci-lint` is replaced by
|
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
|
||||||
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
|
forcing BuildKit to finish linting before the build stage starts, and dropping
|
||||||
only edge forcing BuildKit to finish linting before the build stage
|
it without replacing the edge would have ended fail-fast linting silently
|
||||||
starts, and dropping it without replacing the edge would have ended
|
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
|
||||||
fail-fast linting silently under a still-green build. That is
|
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
|
||||||
canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
|
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
|
||||||
`ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
|
along with its README entry, because both of its subjects ceased to exist in
|
||||||
that justified it. `script/verify-linter-pin` is retired, deleted
|
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
|
||||||
along with its README entry, because both of its subjects ceased to
|
in `script/bootstrap`, and there is now neither a binary crossing between
|
||||||
exist in the same change: it compared a linter binary against
|
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
|
||||||
`GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
|
it has moved — the linter is still pinned twice, now as the `FROM` line of
|
||||||
neither a binary crossing between stages nor a version pin in
|
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
|
||||||
bootstrap. The drift it guarded has not gone away, it has moved — the
|
nothing syncing them, which is exactly what
|
||||||
linter is still pinned twice, now as the `FROM` line of
|
|
||||||
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
|
|
||||||
with nothing syncing them, which is exactly what
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
|
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
|
||||||
replacement is one new `script/verify-lint-image-pin`,
|
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
|
||||||
run as a gate in both files, which compares the two references to
|
files, which compares the two references to each other and deliberately
|
||||||
each other and deliberately restates neither: a hardcoded expected
|
restates neither: a hardcoded expected digest would be a third copy and the
|
||||||
digest would be a third copy and the same drift one file further out.
|
same drift one file further out. `golangci-lint config verify` is included per
|
||||||
`golangci-lint config verify` is included per the ruling, and the
|
the ruling, and the concern about its unpinned live HTTPS schema fetch was
|
||||||
concern about its unpinned live HTTPS schema fetch was measured
|
measured rather than assumed — under `--network none` the pinned binary both
|
||||||
rather than assumed — under `--network none` the pinned binary both
|
passes a valid config and rejects an invalid one with the jsonschema error, so
|
||||||
passes a valid config and rejects an invalid one with the jsonschema
|
it validates from an embedded schema and makes no network call of its own. The
|
||||||
error, so it validates from an embedded schema and makes no network
|
README scopes that to the gate steps rather than to linting as a whole:
|
||||||
call of its own. The README scopes that to the gate steps rather
|
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
|
||||||
than to linting as a whole: `Dockerfile.lint` runs `go mod download`
|
needs the network and only a warm one lints offline. Verified: `make lint`
|
||||||
above them, so a cold cache still needs the network and only a warm
|
green with every `PATH` directory containing a `golangci-lint` removed
|
||||||
one lints offline. Verified: `make lint` green with every `PATH`
|
|
||||||
directory containing a `golangci-lint` removed
|
|
||||||
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
|
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
|
||||||
`command -v golangci-lint` empty); two consecutive `script/lint` runs
|
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
|
||||||
on an untouched tree both executed the linter, 27.7s and 28.7s in the
|
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
|
||||||
lint step under distinct epochs with the `COPY . .` layer `CACHED`
|
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
|
||||||
above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
|
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
|
||||||
weakened to shorten that. Negative control: a planted
|
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
|
||||||
`var unusedIssue46Sentinel = 1` failed `script/lint` with
|
`script/lint` with
|
||||||
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
|
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
|
||||||
failed `make docker` at `[lint 9/9]` with the build stage stopped at
|
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
|
||||||
`[builder 3/12]` — `COPY --from=lint`, `script/bootstrap`, the test
|
— `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
|
||||||
gate and `make build` all zero occurrences — then reverted clean. The
|
zero occurrences — then reverted clean. The drift guard fails on a tag-only
|
||||||
drift guard fails on a tag-only disagreement, on a digest-only
|
disagreement, on a digest-only disagreement, and on an unreadable reference,
|
||||||
disagreement, and on an unreadable reference, naming both sides.
|
naming both sides. `make docker` green in 5m35s with all six gates executing
|
||||||
`make docker` green in 5m35s with all six gates executing under one
|
under one epoch (lint 37.6s, test 25.2s reporting
|
||||||
epoch (lint 37.6s, test 25.2s reporting
|
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
|
||||||
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
|
non-root quirk still holds: in the builder image with the Go test cache off,
|
||||||
The non-root quirk still holds: in the builder image with the Go test
|
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
|
||||||
cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
|
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
|
||||||
(exit 1) where the unprivileged user passes (exit 0). Noted for
|
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
|
||||||
follow-up, not fixed here: `golangci-lint` warns that the
|
in favour of `gomodguard_v2`.
|
||||||
`gomodguard` linter is deprecated since v2.12.0 in favour of
|
|
||||||
`gomodguard_v2`.
|
|
||||||
|
|
||||||
- install the Docker build stage's prerequisites by running
|
- install the Docker build stage's prerequisites by running `script/bootstrap`
|
||||||
`script/bootstrap` instead of `apk add --no-cache make` inline
|
instead of `apk add --no-cache make` inline (2026-08-09, branch
|
||||||
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
|
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
|
||||||
`REPO_POLICIES.md:97` requires it, and the inline install left the
|
it, and the inline install left the build stage maintaining its own notion of
|
||||||
build stage maintaining its own notion of the toolchain — exactly
|
the toolchain — exactly the divergence #24 exists to close, one layer down.
|
||||||
the divergence #24 exists to close, one layer down. The stage now
|
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
|
||||||
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
|
`script/bootstrap`, which ends in `go mod download`, so the separate
|
||||||
which ends in `go mod download`, so the separate invocation of that
|
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
|
||||||
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
|
and moves above the bootstrap layer. It is the only edge making this stage
|
||||||
above the bootstrap layer. It is the only edge making this stage
|
depend on the lint stage, so deleting it as redundant would end fail-fast
|
||||||
depend on the lint stage, so deleting it as redundant would end
|
linting silently. Letting bootstrap install its own linter here would have
|
||||||
fail-fast linting silently. Letting bootstrap install its own linter
|
reintroduced the second toolchain and paid for a from-source build of it. What
|
||||||
here would have reintroduced the second toolchain and paid for a
|
makes the two stages provably one toolchain rather than two that happen to
|
||||||
from-source build of it. What makes the two stages provably one
|
agree is a new `script/verify-linter-pin`, run in the build stage on the
|
||||||
toolchain rather than two that happen to agree is a new
|
binary that arrives from the lint stage, before bootstrap: it fails the build
|
||||||
`script/verify-linter-pin`, run in the build stage on the binary
|
naming both versions unless that binary is the version `script/bootstrap`
|
||||||
that arrives from the lint stage, before bootstrap: it fails the
|
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
|
||||||
build naming both versions unless that binary is the version
|
pin from source and then verifies whatever `PATH` resolves, so drift
|
||||||
`script/bootstrap` pins. Bootstrap's own check could not serve that
|
self-heals silently and a lint stage image bumped on its own would lint at the
|
||||||
purpose — it reinstalls its pin from source and then verifies
|
new version while `make check` ran at the old one, green. The linter version
|
||||||
whatever `PATH` resolves, so drift self-heals silently and a lint
|
is pinned in two independent places (the lint stage image digest and
|
||||||
stage image bumped on its own would lint at the new version while
|
|
||||||
`make check` ran at the old one, green. The linter version is pinned
|
|
||||||
in two independent places (the lint stage image digest and
|
|
||||||
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
|
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
|
||||||
half-applied bump is now a build failure. The pin is read out of
|
half-applied bump is now a build failure. The pin is read out of
|
||||||
`script/bootstrap`, which stays the single source of truth; a pin
|
`script/bootstrap`, which stays the single source of truth; a pin that cannot
|
||||||
that cannot be read is a hard failure, not a skip. The check needs
|
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
|
||||||
no `CHECK_EPOCH`: its only inputs are the copied binary and
|
only inputs are the copied binary and `script/`, so Docker invalidates the
|
||||||
`script/`, so Docker invalidates the layer exactly when a cached
|
layer exactly when a cached result would stop being true, and it is documented
|
||||||
result would stop being true, and it is documented with the other
|
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
|
||||||
entrypoints in the README. `$GOPATH/bin` joins `PATH` because
|
that is where bootstrap's `go install` lands and bootstrap verifies its
|
||||||
that is where bootstrap's `go install` lands and bootstrap verifies
|
installs against what `PATH` resolves — nothing in the image is shadowed by
|
||||||
its installs against what `PATH` resolves — nothing in the image is
|
it, the directory does not exist until bootstrap runs. Everything added sits
|
||||||
shadowed by it, the directory does not exist until bootstrap runs.
|
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
|
||||||
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
|
`make check`. Verified: the guard fails the build with both versions named
|
||||||
`USER builder` still precede `make check`. Verified: the guard fails
|
when the lint stage's linter is faked to a different version, and an
|
||||||
the build with both versions named when the lint stage's linter is
|
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
|
||||||
faked to a different version, and an unmodified build still passes
|
its `apk` branch, installing `git` and `make` and finding the copied linter
|
||||||
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
|
already at the pin; a second build served the bootstrap and dependency layers
|
||||||
installing `git` and `make` and finding the copied
|
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
|
||||||
linter already at the pin; a second build served the bootstrap and
|
failed the build at the lint gate in 48.9s with the build stage's `make check`
|
||||||
dependency layers `CACHED` while both gates ran with a fresh epoch;
|
never starting; and the suite run in the image as `--user 0:0` fails
|
||||||
a planted `unused` finding failed the build at the lint gate in
|
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
|
||||||
48.9s with the build stage's `make check` never starting; and the
|
still load-bearing. That last check needs the Go test cache disabled — the
|
||||||
suite run in the image as `--user 0:0` fails
|
first attempt reported `ok ... (cached)` as root, reusing the result the
|
||||||
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
|
build-time run had left in the shared cache, which would have read as a pass.
|
||||||
user is still load-bearing. That last check needs the Go test cache
|
Build wall time, on a shared host running many concurrent builds and so noisy:
|
||||||
disabled — the first attempt reported `ok ... (cached)` as root,
|
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
|
||||||
reusing the result the build-time run had left in the shared cache,
|
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
|
||||||
which would have read as a pass. Build wall time, on a shared host
|
because of this change — `chown -R builder:builder /src /home/builder` walks
|
||||||
running many concurrent builds and so noisy: 2m13s on an unchanged
|
the module cache and re-runs on every source change, and it alone varied
|
||||||
tree, 2m17s and 4m29s for two builds after a source change, 5m14s
|
between 77s and 210s across those four builds, which is also the whole spread
|
||||||
cold. Only the cold one breaches the policy ceiling, and not because
|
in the totals. The same cold measurement against `main` is 5m03s with a 209s
|
||||||
of this change — `chown -R builder:builder /src /home/builder` walks
|
`chown`. Filed as #43
|
||||||
the module cache and re-runs on every source change, and it alone
|
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
|
||||||
varied between 77s and 210s across those four builds, which is also
|
`script/docker` cannot report a green they did not earn (2026-08-09, branch
|
||||||
the whole spread in the totals. The same cold measurement against
|
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
|
||||||
`main` is 5m03s with a 209s `chown`. Filed as #43
|
invocations with no cache control, and the `Dockerfile` copies the tree before
|
||||||
- bust the Docker layer cache for the gate steps, so `script/cibuild`
|
running its gates, so on an unchanged tree Docker served those layers from
|
||||||
and `script/docker` cannot report a green they did not earn
|
cache and the build exited 0 having executed nothing. That is not hypothetical
|
||||||
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
|
here — every merge this repo has done is a non-fast-forward merge of an
|
||||||
were bare `docker build` invocations with no cache control, and the
|
undiverged branch, so each merge commit's tree is byte-identical to the branch
|
||||||
`Dockerfile` copies the tree before running its gates, so on an
|
head's and each merge CI run was almost certainly a full cache hit; and PR
|
||||||
unchanged tree Docker served those layers from cache and the build
|
#31's reviewer found `make docker` returning success as a 17-layer cache hit,
|
||||||
exited 0 having executed nothing. That is not hypothetical here —
|
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
|
||||||
every merge this repo has done is a non-fast-forward merge of an
|
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
|
||||||
undiverged branch, so each merge commit's tree is byte-identical to
|
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
|
||||||
the branch head's and each merge CI run was almost certainly a full
|
across two — `make fmt-check` and `make lint` in the lint stage, `make check`
|
||||||
cache hit; and PR #31's reviewer found `make docker` returning
|
in the build stage — so a single declaration would have left one stage
|
||||||
success as a 17-layer cache hit, catching it only by being
|
silently cacheable; it is declared in both. And BuildKit hashes the expanded
|
||||||
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
|
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
|
||||||
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
|
nothing: each gate `RUN` echoes the epoch, which also puts the value in the
|
||||||
it. `ARG` is scoped per stage and this `Dockerfile` has three gates
|
build log as evidence the layer really ran. Placement is below the dependency
|
||||||
across two — `make fmt-check` and `make lint` in the lint stage,
|
layers on purpose — a build that goes cold every time would be a different
|
||||||
`make check` in the build stage — so a single declaration would have
|
bug, not a fix. Verified by running each script twice back to back on an
|
||||||
left one stage silently cacheable; it is declared in both. And
|
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
|
||||||
BuildKit hashes the expanded command, not the declaration, so a
|
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
|
||||||
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
|
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
|
||||||
echoes the epoch, which also puts the value in the build log as
|
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
|
||||||
evidence the layer really ran. Placement is below the dependency
|
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
|
||||||
layers on purpose — a build that goes cold every time would be a
|
the lint stage, and the binary copy into the runtime stage. The lint stage
|
||||||
different bug, not a fix. Verified by running each script twice back
|
still gates the build stage: with a deliberate `unused` finding planted in the
|
||||||
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
|
tree, the build failed at `make lint` in 36.1s and the build-stage
|
||||||
three gates executed on all four runs, each with a fresh epoch in
|
`make check` never started. The build stage also still drops to the
|
||||||
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
|
unprivileged `builder` user before `make check`, which the suite depends on
|
||||||
then 53.4s), and twelve steps were still served `CACHED` in the
|
rather than merely prefers: forcing the same image to run the tests as root
|
||||||
steady state — both `go mod download`s, `apk add`, `adduser`, the
|
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
|
||||||
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy
|
the `chmod(0)` the test uses to prove hard links are read once. This is the
|
||||||
out of the lint stage, and the binary copy into the runtime stage.
|
local fix only; propagating it to the canonical templates is `prompts` #26
|
||||||
The lint stage still gates the build stage: with a deliberate
|
- check the installed golangci-lint version in `script/bootstrap` instead of
|
||||||
`unused` finding planted in the tree, the build failed at
|
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
|
||||||
`make lint` in 36.1s and the build-stage `make check` never started.
|
`missing golangci-lint` meant any linter already on `PATH` satisfied the
|
||||||
The build stage also still drops to the unprivileged `builder` user
|
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
|
||||||
before `make check`, which the suite depends on rather than merely
|
on every host that already had one — this host ran v2.10.1 against a v2.12.2
|
||||||
prefers: forcing the same image to run the tests as root fails
|
pin, `make check` went green, and `make docker` then rejected the same commit
|
||||||
`TestScanHardlinkRunFailsTogether`, because root reads straight
|
with findings the local gate never saw. The version now lives in one place,
|
||||||
through the `chmod(0)` the test uses to prove hard links are read
|
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
|
||||||
once. This is the local fix only; propagating it to the canonical
|
bump cannot half-apply; a `golangci_lint_version` helper parses
|
||||||
templates is `prompts` #26
|
`golangci-lint --version` (taking the field after the word `version` and
|
||||||
- check the installed golangci-lint version in `script/bootstrap`
|
tolerating an optional leading `v`, which the module ref carries and the
|
||||||
instead of only its presence (2026-08-09, branch
|
binary's output does not), and any version that is not the pin — older, newer,
|
||||||
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant
|
absent or unparseable — is reinstalled. The install is then verified against
|
||||||
any linter already on `PATH` satisfied the check, so the pin was never
|
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
|
||||||
consulted and the v2.12.2 bump from #3 was inert on every host that
|
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
|
||||||
already had one — this host ran v2.10.1 against a v2.12.2 pin,
|
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
|
||||||
`make check` went green, and `make docker` then rejected the same
|
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
|
||||||
commit with findings the local gate never saw. The version now lives
|
the install and leave the local gate disagreeing with CI under an affirmative
|
||||||
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
|
`bootstrap complete`. Bootstrap now re-reads the effective version after
|
||||||
ref derived from it so a bump cannot half-apply; a
|
installing and, on a mismatch, prints both paths and both versions to stderr
|
||||||
`golangci_lint_version` helper parses `golangci-lint --version`
|
and exits non-zero instead of claiming success; it does not reorder anyone's
|
||||||
(taking the field after the word `version` and tolerating an optional
|
|
||||||
leading `v`, which the module ref carries and the binary's output does
|
|
||||||
not), and any version that is not the pin — older, newer, absent or
|
|
||||||
unparseable — is reinstalled. The install is then verified against the
|
|
||||||
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
|
|
||||||
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
|
|
||||||
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
|
|
||||||
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
|
|
||||||
stage makes — would swallow the install and leave the local gate
|
|
||||||
disagreeing with CI under an affirmative `bootstrap complete`.
|
|
||||||
Bootstrap now re-reads the effective version after installing and, on
|
|
||||||
a mismatch, prints both paths and both versions to stderr and exits
|
|
||||||
non-zero instead of claiming success; it does not reorder anyone's
|
|
||||||
`PATH` or delete their binary. The `--version` call keeps its stderr
|
`PATH` or delete their binary. The `--version` call keeps its stderr
|
||||||
connected, so a present-but-broken binary says why rather than
|
connected, so a present-but-broken binary says why rather than reinstalling
|
||||||
reinstalling forever in silence, and is bounded by `timeout(1)` where
|
forever in silence, and is bounded by `timeout(1)` where that exists, so a
|
||||||
that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
|
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
|
||||||
and `go` keep their presence-only checks and now say why in a
|
presence-only checks and now say why in a comment: they are host
|
||||||
comment: they are host package-manager tools the repo deliberately
|
package-manager tools the repo deliberately does not pin, with `go.mod`
|
||||||
does not pin, with `go.mod` governing the language version and the
|
governing the language version and the digest-pinned images covering
|
||||||
digest-pinned images covering reproducible builds. Verified on this
|
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
|
||||||
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
|
v2.12.2 and running it again to a no-op, plus stub runs of the real script
|
||||||
a no-op, plus stub runs of the real script under `dash` covering a
|
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
|
||||||
thirteen-input parse matrix (absent, older, newer, host-style,
|
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
|
||||||
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
|
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
|
||||||
binary, `(devel)`, trailing `version`), a shadowed install that must
|
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
|
||||||
exit non-zero, an install destination not on `PATH` at all, `GOBIN`
|
wedged binary that must hit the timeout; `make check` and `make lint` are
|
||||||
set, and a wedged binary that must hit the timeout; `make check` and
|
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
|
||||||
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
|
|
||||||
findings on `main`
|
|
||||||
- unwind the hash worker pool on the error path (2026-08-09, branch
|
- unwind the hash worker pool on the error path (2026-08-09, branch
|
||||||
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
|
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
|
||||||
moment `recordRun` failed and abandon the pool — the feeder parked
|
`recordRun` failed and abandon the pool — the feeder parked forever on a full
|
||||||
forever on a full `jobs` channel and every worker on a full
|
`jobs` channel and every worker on a full `results` channel. That only stopped
|
||||||
`results` channel. That only stopped being invisible when #4 landed
|
being invisible when #4 landed and `runScan` began unwinding instead of
|
||||||
and `runScan` began unwinding instead of calling `os.Exit`. The
|
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
|
||||||
pool is now an owned, context-aware `hashPool`: every blocking send
|
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
||||||
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
|
||||||
closed on every path out, and `hashPhase` defers `pool.stop()`,
|
and then drains `results` until the last goroutine has exited — draining is
|
||||||
which cancels and then drains `results` until the last goroutine
|
what frees a worker already parked on a send. `ctx` is threaded from
|
||||||
has exited — draining is what frees a worker already parked on a
|
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
|
||||||
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
|
database layer (it is the first parameter everywhere), so #5 can hand this
|
||||||
`syncScan`, both worker pools and the whole database layer (it is
|
path a signal and needs to add nothing else. The walk pool never leaked,
|
||||||
the first parameter everywhere), so #5 can hand this path a signal
|
because `walkPhase` always drains its events to close, but it has the same
|
||||||
and needs to add nothing else. The walk pool never leaked, because
|
unbounded-send shape and #5 will give it an early return, so it gets the same
|
||||||
`walkPhase` always drains its events to close, but it has the same
|
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
|
||||||
unbounded-send shape and #5 will give it an early return, so it
|
partial size census, and every file it never reached looks vanished to the
|
||||||
gets the same treatment plus a `ctx.Err()` guard after the walk: a
|
update phase. That phase's own `BeginTx` fails on the same cancelled context
|
||||||
cancelled walk yields a partial size census, and every file it never
|
before deleting anything, so the guard is defence in depth rather than the
|
||||||
reached looks vanished to the update phase. That phase's own
|
only barrier — but it is the one that survives #5 deciding an interrupted scan
|
||||||
`BeginTx` fails on the same cancelled context before deleting
|
may commit what it has. Tests drive `run(scan)` against a database whose
|
||||||
anything, so the guard is defence in depth rather than the only
|
insert trigger aborts, and assert both that the scan fails instead of hanging
|
||||||
barrier — but it is the one that survives #5 deciding an interrupted
|
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
|
||||||
scan may commit what it has. Tests drive `run(scan)` against a
|
second set cancels a scan part-way through the walk — deterministically, by
|
||||||
database whose insert trigger aborts, and assert both that the scan
|
counting the scan's own consultations of `ctx.Done()` rather than racing a
|
||||||
fails instead of hanging and that `runtime.NumGoroutine()` polls
|
timer — and asserts that it stops at the guard holding a partial census and a
|
||||||
back to its pre-scan baseline; a second set cancels a scan part-way
|
still-populated record index, with every record intact. The remaining
|
||||||
through the walk — deterministically, by counting the scan's own
|
cancellation branches of both pools are covered by direct tests of
|
||||||
consultations of `ctx.Done()` rather than racing a timer — and
|
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||||
asserts that it stops at the guard holding a partial census and a
|
and `hashPhase`
|
||||||
still-populated record index, with every record intact. The
|
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
|
||||||
remaining cancellation branches of both pools are covered by direct
|
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
|
||||||
tests of `sendEvent`, the walk workers, `dispatchDirs`,
|
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
|
||||||
`feedHashJobs`, `hashWorker` and `hashPhase`
|
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
|
||||||
- guarantee the database is closed on every fatal exit path
|
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||||
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
|
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
|
||||||
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
|
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
|
||||||
it the SQLite WAL checkpoint — now actually runs when a subcommand
|
runtime failure from being reported as a usage error or printing the usage
|
||||||
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
|
text. New `main_test.go` drives the CLI in-process and asserts the exit codes
|
||||||
`resolveRoots` return errors instead. The single exit point is `run`
|
from README §Error handling plus the stdout/stderr split, including that a
|
||||||
in `main.go`: it maps a `fatalError` (anything a subcommand
|
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
|
||||||
returned) to exit 1 and cobra's own argument and flag errors to exit
|
behind for `scan`, `report` or `trees`
|
||||||
2, which keeps a runtime failure from being reported as a usage
|
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
|
||||||
error or printing the usage text. New `main_test.go` drives the CLI
|
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
|
||||||
in-process and asserts the exit codes from README §Error handling
|
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
|
||||||
plus the stdout/stderr split, including that a fatal error raised
|
and replaced `.golangci.yml` with the canonical file — the linter settings
|
||||||
after the database is open leaves no `-wal`/`-shm` sidecar behind
|
|
||||||
for `scan`, `report` or `trees`
|
|
||||||
- update golangci-lint to v2.12.2 with the canonical config
|
|
||||||
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
|
|
||||||
closes #3): bumped the pinned linter in the `Dockerfile` lint
|
|
||||||
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
|
|
||||||
`.golangci.yml` with the canonical file — the linter settings
|
|
||||||
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
||||||
`linters.settings` per the v2 schema, so they are actually
|
`linters.settings` per the v2 schema, so they are actually applied; no new
|
||||||
applied; no new lint findings surfaced
|
lint findings surfaced
|
||||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
|
||||||
entrypoints like the other managed repos (2026-07-26, commit
|
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
|
||||||
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
|
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
|
||||||
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
||||||
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
`install-precommit`) and every Makefile target is now a thin shim over them,
|
||||||
`install-precommit`) and every Makefile target is now a thin shim
|
matching the other managed repos
|
||||||
over them, matching the other managed repos
|
|
||||||
- make the binary the default Make target (2026-07-24, branch
|
- make the binary the default Make target (2026-07-24, branch
|
||||||
`make-default-target`): plain `make` now builds `sfdupes`
|
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
|
||||||
(previously it ran `check` plus `build`); `make build` remains as
|
`check` plus `build`); `make build` remains as an alias
|
||||||
an alias
|
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
|
||||||
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
|
`scan-wide-phases`): all operands seed the shared walk pool and every pass
|
||||||
branch `scan-wide-phases`): all operands seed the shared walk pool
|
runs once over the whole scan, so totals and ETAs are scan-global; the
|
||||||
and every pass runs once over the whole scan, so totals and ETAs
|
per-operand walk/hash/update cycles and their stderr announcements are gone;
|
||||||
are scan-global; the per-operand walk/hash/update cycles and their
|
the update pass commits in batched transactions — the filesystem is
|
||||||
stderr announcements are gone; the update pass commits in batched
|
authoritative and the database an eventually-consistent reflection, so
|
||||||
transactions — the filesystem is authoritative and the database an
|
scan-level atomicity is not required
|
||||||
eventually-consistent reflection, so scan-level atomicity is not
|
|
||||||
required
|
|
||||||
- split the stat pass back out of the walk (2026-07-24, branch
|
- split the stat pass back out of the walk (2026-07-24, branch
|
||||||
`parallel-phases`): phases are strictly sequential again — walk,
|
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
|
||||||
stat, hash, update per operand — with parallelism only inside each
|
update per operand — with parallelism only inside each phase; the walk
|
||||||
phase; the walk enumerates paths with per-directory workers and the
|
enumerates paths with per-directory workers and the stat pass lstats them with
|
||||||
stat pass lstats them with per-file workers, restoring the exact
|
per-file workers, restoring the exact total/ETA stat bar
|
||||||
total/ETA stat bar
|
- announce each operand on stderr before its passes (2026-07-24, branch
|
||||||
- announce each operand on stderr before its passes (2026-07-24,
|
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
|
||||||
branch `scan-operand-progress`): with per-operand walk/hash/update
|
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
|
||||||
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
|
whole run's — an operator watching operand 3 of 14 hash 300k files concluded
|
||||||
that looked like the whole run's — an operator watching operand 3 of
|
20M files were being skipped
|
||||||
14 hash 300k files concluded 20M files were being skipped
|
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
|
||||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
|
||||||
was a single goroutine and took hours at ~20M files on a busy pool
|
4h on a ZFS server); it is now a per-directory worker-pool traversal that
|
||||||
(observed: 22M files in 4h on a ZFS server); it is now a
|
records size/mtime during the walk (folding away the separate stat pass,
|
||||||
per-directory worker-pool traversal that records size/mtime during
|
halving metadata I/O), and each `PATH` operand commits in its own transaction
|
||||||
the walk (folding away the separate stat pass, halving metadata
|
so an interrupted scan keeps completed operands
|
||||||
I/O), and each `PATH` operand commits in its own transaction so an
|
|
||||||
interrupted scan keeps completed operands
|
|
||||||
|
|
||||||
- persistent scan database (2026-07-24, branch `persistent-database`):
|
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
|
||||||
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
|
||||||
Go, cgo stays disabled) keyed by absolute path that survives between
|
disabled) keyed by absolute path that survives between runs — a rescan hashes
|
||||||
runs — a rescan hashes only new or changed files (by mtime/size),
|
only new or changed files (by mtime/size), deletes records for files vanished
|
||||||
deletes records for files vanished from under the scanned operands,
|
from under the scanned operands, and leaves records outside them untouched, so
|
||||||
and leaves records outside them untouched, so `scan` can be cronned
|
`scan` can be cronned daily; `report` and `trees` read the database (no
|
||||||
daily; `report` and `trees` read the database (no positional
|
positional arguments) instead of a scan stream. Database at
|
||||||
arguments) instead of a scan stream. Database at
|
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
|
||||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
journaling plus a single-transaction update keep a report run during a scan
|
||||||
WAL journaling plus a single-transaction update keep a report run
|
safe
|
||||||
during a scan safe
|
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
|
||||||
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
and push `main` plus tags (2026-07-23)
|
||||||
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
|
||||||
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
||||||
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
|
||||||
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
|
||||||
at filesystem boundaries, which are crossed by default
|
boundaries, which are crossed by default
|
||||||
- bring the repo into full policy compliance (2026-07-23, branch
|
- bring the repo into full policy compliance (2026-07-23, branch
|
||||||
`repo-policy-compliance`; checklist below)
|
`repo-policy-compliance`; checklist below)
|
||||||
- `git init` with README-only first commit; code baseline committed on
|
- `git init` with README-only first commit; code baseline committed on `main`
|
||||||
`main` (2026-07-22)
|
(2026-07-22)
|
||||||
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- possible later features (explicitly out of scope per README):
|
- possible later features (explicitly out of scope per README): full-content
|
||||||
full-content verification of candidates, removal-script helpers
|
verification of candidates, removal-script helpers
|
||||||
|
|
||||||
# Repo Policy Compliance
|
# Repo Policy Compliance
|
||||||
|
|
||||||
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
|
||||||
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
|
||||||
standalone formatting commit is needed.
|
formatting commit is needed.
|
||||||
|
|
||||||
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
|
||||||
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
|
||||||
artifacts plus secrets patterns
|
|
||||||
- [x] `.editorconfig` missing
|
- [x] `.editorconfig` missing
|
||||||
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
- [x] `LICENSE` missing and README has no License section (MIT assumed from
|
||||||
from house convention — user to confirm)
|
house convention — user to confirm)
|
||||||
- [x] `REPO_POLICIES.md` missing from repo root
|
- [x] `REPO_POLICIES.md` missing from repo root
|
||||||
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
|
||||||
pass `make lint` (150 findings fixed; `make lint` is clean)
|
`make lint` (150 findings fixed; `make lint` is clean)
|
||||||
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
|
||||||
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
`docker`, `hooks`; `check` currently depends on `build`, which writes the
|
||||||
`build`, which writes the binary (`make check` must not modify
|
binary (`make check` must not modify files)
|
||||||
files)
|
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
|
||||||
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
with a 30-second timeout and the conditional `-v` rerun pattern (suite
|
||||||
real tests with a 30-second timeout and the conditional `-v`
|
covers parsing, grouping, digests, suppression, hashing, and the scan
|
||||||
rerun pattern (suite covers parsing, grouping, digests,
|
pipeline; 64% coverage)
|
||||||
suppression, hashing, and the scan pipeline; 64% coverage)
|
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
|
||||||
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
lint stage, build stage running `make check`
|
||||||
fail-fast lint stage, build stage running `make check`
|
|
||||||
- [x] `.dockerignore` missing
|
- [x] `.dockerignore` missing
|
||||||
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
|
||||||
checkout action pinned by commit SHA)
|
action pinned by commit SHA)
|
||||||
- [x] README lacks required sections: Description first line
|
- [x] README lacks required sections: Description first line
|
||||||
(name/purpose/category/license/author), Getting Started,
|
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
|
||||||
Rationale, TODO, License, Author
|
License, Author
|
||||||
- [x] README non-goal "no git repository setup and no CI" is stale now
|
- [x] README non-goal "no git repository setup and no CI" is stale now that the
|
||||||
that the repo is under git with CI
|
repo is under git with CI
|
||||||
- [x] pre-commit hook not installed (`make hooks` once the target
|
- [x] pre-commit hook not installed (`make hooks` once the target exists)
|
||||||
exists)
|
|
||||||
|
|
||||||
Accepted divergences (no action):
|
Accepted divergences (no action):
|
||||||
|
|
||||||
- flat single-package layout with `.go` files in the repo root — fine
|
- flat single-package layout with `.go` files in the repo root — fine for a
|
||||||
for a small single-binary tool per the Go styleguide; the tracker
|
small single-binary tool per the Go styleguide; the tracker audit agrees
|
||||||
audit agrees
|
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
|
||||||
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
builds) and the race detector requires cgo
|
||||||
(pure-Go builds) and the race detector requires cgo
|
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
{
|
||||||
|
"devDependencies": {
|
||||||
|
"prettier": "3.8.1"
|
||||||
|
}
|
||||||
|
}
|
||||||
+13
-11
@@ -3,10 +3,12 @@
|
|||||||
# this repo. Idempotent: every install is guarded by a check so already
|
# this repo. Idempotent: every install is guarded by a check so already
|
||||||
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
# installed tools are skipped. Base tooling comes from nix, apt, brew,
|
||||||
# or apk (detected in that order); assumes nothing is present (not git,
|
# or apk (detected in that order); assumes nothing is present (not git,
|
||||||
# make, or go). The linter is NOT installed: golangci-lint runs via
|
# make, or go). Neither the linter nor the Markdown formatter is
|
||||||
# docker only (script/lint), pinned by image digest, so the only lint
|
# installed: golangci-lint (script/lint) and prettier (script/fmt,
|
||||||
|
# script/fmt-check) run via docker only, pinned by hash, so their only
|
||||||
# prerequisite is a working docker — which is warned about, not
|
# prerequisite is a working docker — which is warned about, not
|
||||||
# installed, because everything except linting works without it.
|
# installed, because everything except linting and formatting works
|
||||||
|
# without it.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
@@ -71,15 +73,15 @@ main() {
|
|||||||
if missing make; then pkg_install gnumake make make make; fi
|
if missing make; then pkg_install gnumake make make make; fi
|
||||||
if missing go; then pkg_install go golang go go; fi
|
if missing go; then pkg_install go golang go go; fi
|
||||||
|
|
||||||
# Linting runs via docker only (script/lint), so docker is a lint
|
# Linting and Markdown formatting run via docker only, so docker is
|
||||||
# prerequisite rather than something bootstrap installs. Warn, do
|
# their prerequisite rather than something bootstrap installs. Warn,
|
||||||
# not fail: everything except `make lint` — and, through it,
|
# do not fail: everything except `make lint`, `make fmt` and
|
||||||
# `make check`, `make docker` and the pre-commit hook — works
|
# `make fmt-check` — and, through them, `make check`, `make docker`
|
||||||
# without it.
|
# and the pre-commit hook — works without it.
|
||||||
if missing docker; then
|
if missing docker; then
|
||||||
echo "bootstrap: WARNING: docker not found; make lint, make check" >&2
|
echo "bootstrap: WARNING: docker not found; make lint, make fmt," >&2
|
||||||
echo "bootstrap: and make docker require it. Install docker to" >&2
|
echo "bootstrap: make fmt-check, make check and make docker" >&2
|
||||||
echo "bootstrap: run the linter." >&2
|
echo "bootstrap: require it." >&2
|
||||||
fi
|
fi
|
||||||
|
|
||||||
go mod download
|
go mod download
|
||||||
|
|||||||
+10
-10
@@ -3,17 +3,17 @@
|
|||||||
# push.
|
# push.
|
||||||
#
|
#
|
||||||
# The Dockerfile runs the gates individually as build steps, not the
|
# The Dockerfile runs the gates individually as build steps, not the
|
||||||
# make check aggregate: the lint stage runs make fmt-check,
|
# make check aggregate: the lint stage runs the gofmt check,
|
||||||
# script/verify-lint-image-pin, golangci-lint config verify and
|
# script/verify-lint-image-pin, golangci-lint config verify and
|
||||||
# golangci-lint run; the build stage, dropped to an unprivileged user,
|
# golangci-lint run; the markdown stage runs the prettier check; the
|
||||||
# runs make test and make fmt-check. Neither make lint nor make check
|
# build stage, dropped to an unprivileged user, runs make test. None of
|
||||||
# appears, because both reach script/lint, which is itself a docker
|
# make lint, make fmt-check or make check appears, because each runs
|
||||||
# build, and a docker build cannot run inside one. Lint is not skipped
|
# docker, and docker cannot run inside a docker build. Nothing is
|
||||||
# by that — the linter is invoked directly in the lint stage, and the
|
# skipped by that — the linter, gofmt and prettier are invoked directly
|
||||||
# build stage's COPY --from=lint makes that stage a prerequisite, so
|
# in their stages, and the build stage's COPY --from lines make those
|
||||||
# BuildKit must finish it first. Between the two stages everything
|
# stages prerequisites, so BuildKit must finish them first. Between the
|
||||||
# make check would run has run, which is why a successful build here
|
# three stages everything make check would run has run, which is why a
|
||||||
# implies the repo is green.
|
# successful build here implies the repo is green.
|
||||||
#
|
#
|
||||||
# That implication holds only because of CHECK_EPOCH. A COPY layer is
|
# That implication holds only because of CHECK_EPOCH. A COPY layer is
|
||||||
# invalidated only by changed content, and a rebuild of an unchanged
|
# invalidated only by changed content, and a rebuild of an unchanged
|
||||||
|
|||||||
+5
-5
@@ -4,11 +4,11 @@
|
|||||||
#
|
#
|
||||||
# CHECK_EPOCH is passed for the same reason script/cibuild passes it:
|
# CHECK_EPOCH is passed for the same reason script/cibuild passes it:
|
||||||
# without it Docker serves the Dockerfile's gate layers from cache on an
|
# without it Docker serves the Dockerfile's gate layers from cache on an
|
||||||
# unchanged tree and this exits 0 having run neither the lint stage's
|
# unchanged tree and this exits 0 having run none of the lint stage's
|
||||||
# gates nor the builder stage's test and fmt-check gates. This is the
|
# gates, the markdown stage's prettier gate or the builder stage's test
|
||||||
# set of gates a developer or reviewer runs by hand, so a cached pass
|
# gate. This is the set of gates a developer or reviewer runs by hand,
|
||||||
# here is the most misleading result the repo can produce. Dependency
|
# so a cached pass here is the most misleading result the repo can
|
||||||
# layers sit above the ARG and stay cached.
|
# produce. Dependency layers sit above the ARG and stay cached.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
|||||||
+12
-2
@@ -1,12 +1,22 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt: format all files (writes).
|
# script/fmt: format all files (writes): the Go sources with gofmt, the
|
||||||
|
# Markdown with prettier. prettier is never installed on the host: it
|
||||||
|
# runs from the Dockerfile's prettier stage with the repository mounted,
|
||||||
|
# as the calling user so the files it rewrites keep their owner. The tag
|
||||||
|
# makes each build replace the previous image instead of leaving another
|
||||||
|
# one behind.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
gofmt -s -w .
|
gofmt -s -w .
|
||||||
|
image="$("$SCRIPT_DIR/projectname")-prettier"
|
||||||
|
docker build -q --target prettier -t "$image" . >/dev/null
|
||||||
|
docker run --rm --user "$(id -u):$(id -g)" -v "$ROOT:/src" "$image" \
|
||||||
|
prettier --write '**/*.md' --tab-width 4 --prose-wrap always
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
+25
-4
@@ -1,18 +1,39 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
# script/fmt-check: check formatting (read-only). Same scope as
|
# script/fmt-check: check formatting (read-only). Same scope as
|
||||||
# script/fmt, but fails instead of writing.
|
# script/fmt, but fails instead of writing. gofmt and prettier both run
|
||||||
|
# every time and each reports its own failure, so the output says which
|
||||||
|
# one failed.
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||||
|
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
|
||||||
|
|
||||||
main() {
|
main() {
|
||||||
cd "$ROOT"
|
cd "$ROOT"
|
||||||
files="$(gofmt -s -l .)"
|
status=0
|
||||||
|
|
||||||
|
# Under set -e a bare assignment would end the script when gofmt
|
||||||
|
# fails (a Go file it cannot parse), and prettier would never run.
|
||||||
|
if ! files="$(gofmt -s -l .)"; then
|
||||||
|
echo "gofmt: failed; see its errors above" >&2
|
||||||
|
status=1
|
||||||
|
fi
|
||||||
if [ -n "$files" ]; then
|
if [ -n "$files" ]; then
|
||||||
echo "gofmt: files not formatted:" >&2
|
echo "gofmt: files not formatted:" >&2
|
||||||
echo "$files" >&2
|
echo "$files" >&2
|
||||||
exit 1
|
status=1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Same image as script/fmt; see there.
|
||||||
|
image="$("$SCRIPT_DIR/projectname")-prettier"
|
||||||
|
docker build -q --target prettier -t "$image" . >/dev/null
|
||||||
|
if ! docker run --rm -v "$ROOT:/src:ro" "$image" \
|
||||||
|
prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
|
||||||
|
echo "prettier: Markdown not formatted; run make fmt" >&2
|
||||||
|
status=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
exit "$status"
|
||||||
}
|
}
|
||||||
|
|
||||||
main "$@"
|
main "$@"
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
|
||||||
|
# yarn lockfile v1
|
||||||
|
|
||||||
|
|
||||||
|
prettier@3.8.1:
|
||||||
|
version "3.8.1"
|
||||||
|
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||||
|
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||||
Reference in New Issue
Block a user