The rate was the phase's average, so a burst after a long wait read as
over 1,000 files a second. It now covers the last 5 seconds, or the
time since the previous line off a terminal; the ETA still comes from
the average. Once no item has completed for 10 seconds, the rate gives
way to "stalled" and where the phase waits: disk, db, out (stdout) or
cpu, or two of them, as "stalled disk/db". A line too wide for the
terminal loses its bar first, then its rate or those words, so the
phase, progress, elapsed time and ETA stay whole. The content phase's
last partial batch is now written under a display of its own.
Deviation: the stall words hold no time and name where the phase waits, not what it does.
Judgement call: trees' group phase, which works in memory, says "cpu".
Model: opus-5-5
info prints the database's path and source, the sizes of the database
file and its -wal file, when the database file last changed, the schema
version and the record counts, one per line as a name, a tab and the
value. It opens the database as report does and shows an open and a
count phase on stderr. The schema version and the counts come from one
read-only transaction, the counts from one query.
seedDatabase now closes test databases out of WAL mode, as scan does;
otherwise info's own connection creates an empty -wal file. The open
phase's name and text became constants for the linter, and the scan
command moved to newScanCommand to keep newRootCommand short enough.
Judgement call: the lines are written after the last display ends, so they skip the display's stdout writer.
Model: opus-5-5
report now runs in two phases, open and read, under the progress
display scan and trees use, until the summary line. open covers a wait
on a busy database. read counts the records, then reads them in the
same read-only transaction in signature order, the files_signature
index read backwards, so SQLite sorts nothing and the read ends at
exactly its count. The duplicate groups are formed in Go from runs of
equal signatures; only those of the size being read are held, and they
are written, ordered by first path, before the next size. The rows go
through the display's stdout writer. stdout and the summary line are
unchanged.
Judgement call: writing is part of read, not a phase of its own, since each size's rows are written while the read goes on.
Model: opus-5-5
trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, until the summary line. open covers a
wait on a busy database. read counts the records and reads them in one
read-only transaction, so the two agree during a scan; group counts the
directories, and write the groups. The rows reach stdout through a
writer. When stdout and stderr are the same file, it clears the display
and holds its lock while writing, so no row shares a line with the
display on a shared terminal; otherwise it writes without the lock, so
a slow reader of stdout cannot stop the display. Each rate names what
its phase counts.
Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Model: opus-5-5
Every scan display names its phase and its place among the five, as
"[2/5] walk", with its progress, elapsed time and ETA. load counts the
records first; the walk estimates its total from the records under the
operands; the content phase adds up the sizes of the matching groups
and says it is searching until the first record returns. Each display
redraws every 100 ms on a terminal and every 5 seconds in plain mode.
On a terminal no line reaches the last column: a line too wide drops
its rate, then its bar. After the terminal is made narrower than the
widest line drawn, the display makes a new library object, whose erase
fits.
Judgement call: load and the content checks draw the bar too, once their totals are known.
Judgement call: the bar is 10 columns wide, not the width the line leaves.
Model: opus-5-5
scan, report and trees now print one line on stderr before anything
else, naming the database and whether its path is the default or comes
from SFDUPES_DATABASE; scan also says when the file is new. report and
trees print it before opening the database, so a missing-database error
follows it. A relative path gets the working directory put in front of
it, with its symlinks resolved as the kernel reports it, and is not
cleaned: a ".." after a symlink names another file than the cleaned
path does. README "Database" lists when scan writes each file.
Unverified: the line on a terminal has no test of its own.
Model: opus-5-5
Every vendored file, REPO_POLICIES.md and every model script is the
copy at sneak/prompts e3f79c4, with this repository's own entries kept
after the canonical content. Lint and test are Dockerfile phases that
write no image; make test runs the suite under the race detector as
nobody. Every build in script/ passes a new CACHEBUST value, which every
stage declares first, so every RUN step runs again; the steps that
compile keep Go's build cache, and both go test runs pass -count=1.
Dockerfile.lint, script/verify-lint-image-pin and make test-race are
gone, prettier runs on the host, and .claude/settings.json is deleted.
Deviation: the set comes from e3f79c4 on next rather than dd4027b.
Judgement call: the test phase mounts the cache at nobody's
/tmp/.cache/go-build, owned by nobody.
Model: opus-5-5
With -x, an operand under another operand is kept and walked on its
own when the walk of the deepest kept operand above it cannot reach
it: some directory from it (from its directory, for a regular file) up
to but not including that operand is on a different filesystem than
that operand. Below a kept operand, the rule keeping records under a
directory -x did not enter covers only directories its own walk did
not enter, so records of files gone from it are deleted.
The new tests stub the devices and share their scan steps with the
test for #99. Comments and
README sentences on operands dropped before the walk now also hold
with -x.
Model: opus-5-5
With -x the walk does not enter a directory on another filesystem than
its operand, but the update phase still deleted every record below it.
A separate scan of a filesystem mounted under the operand therefore lost
its records on every -x scan and re-hashed its whole tree.
The walk now sends an event naming each directory it declines this way,
and the walk phase treats the records below those directories like
records outside the operands: they are not deleted, and their sizes join
the size census. README "scan mode" and "Rules for the walk" state the
rule.
Model: opus-5-5
A regular file replaced by a FIFO after the walk statted it made the
hash phase's open wait forever for a writer, so the scan never ended
and kept the lock. The hash and content phases now open each file with
O_NONBLOCK and check with fstat that it is still a regular file;
anything else is a per-path error, warned about and skipped. The new
test replaces a walked file with a FIFO before the hash phase reads it.
Model: opus-5-5
The lock file was the configured database path with .lock appended, so
a scan reaching the database through a symlink to it locked a different
file and ran alongside a scan holding the lock through the real path.
Before it takes the lock, scan now creates the database file, empty, if
it is missing, following a symlink to its target as SQLite would, and
names the lock file from that file's path with symlinks resolved. A
scan started before the database exists therefore takes the same lock
file as every later scan through the same path. testDBPath resolves its
temp dir, so lock errors name the expected path where the temp dir is
itself reached through a symlink.
Model: opus-5-5
On a terminal, warnf clears the bar's line before it prints a warning.
When the warning is for the last item of a phase, the bar has already
drawn its finished frame and the progress library draws nothing more
for a finished bar, so the warning took the bar's place for good. warnf
now draws the finished bar's last frame again below the warning.
Output when stderr is not a terminal does not change.
Model: opus-5-5
scan recorded mtime in whole seconds, so a file rewritten in place at
the same size within the same second as its recorded mtime was classed
unchanged and kept its old hashes. The files table keeps mtime as whole
Unix seconds and gains mtime_nsec, the nanoseconds within that second.
scan holds the mtime as a time.Time and decides "newer" by comparing
Unix() and then Nanosecond(), so any time a filesystem can record
compares in the right order; After would misorder one too late for a
time.Time to hold without wrapping. The walk, a file given as an
operand, and the content phase's recheck all move over. PRAGMA
user_version stays 1, per the owner's ruling. README states what both
columns hold.
Model: opus-5-5
Completed Steps entries keep what landed, the traps, every disclosure
and every record that a check ran; the argument and history go, with
bare issue numbers turned into full links. Comment blocks in script/,
Dockerfile and Dockerfile.lint keep the trap and drop the defence of
past decisions. TODO.md Workflow now branches from next, targets next,
and leaves merging next to main to the owner. Only comments and
Markdown change.
Model: opus-5-5
script/test-race runs go test -race in a digest-pinned Debian golang
image that has gcc, since the detector needs cgo and the build keeps it
off. The checkout is mounted read-only and the container is removed
afterwards. The tests run as the calling user, or as nobody when that is
root, so the tests that make a file unreadable still see the read fail.
It is not part of make check. The detector found no races.
Model: opus-5-5
Each Dockerfile stage that runs gates now checks, right after its
ARG CHECK_EPOCH, that the value is not empty, and stops with a message
naming script/cibuild and script/docker. A plain `docker build .` can
no longer report a green from cached gate layers.
script/cibuild and script/docker now append the process id to the
epoch, the form script/lint already uses, so two runs started in the
same second still get different values.
README says both. TODO.md corrects the steady-state CACHED count
recorded for issue 32 from twelve to thirteen.
Model: opus-5-5
script/fmt and script/fmt-check run prettier over every Markdown file
again, next to gofmt. prettier is pinned by hash through package.json
and yarn.lock, copied from the prompts repo with .prettierrc and
.prettierignore, and is never installed on a host: a new prettier stage
of the Dockerfile installs it into a digest-pinned node image, and both
scripts build that stage and run it with the repository mounted. CI
checks the Markdown in a markdown stage that the build stage waits on.
Because make fmt-check now runs docker, the Dockerfile runs gofmt
directly in its lint stage instead. All Markdown is reformatted.
Model: opus-5-5
script/lint builds Dockerfile.lint only for the exit status, but every
run exported the result as an image: seconds spent exporting, and one
untagged image left behind each time. It now builds with
--output=type=cacheonly, so nothing is exported. CHECK_EPOCH still
changes on every run, so the gate steps still run each time; the build
cache is kept as before.
Model: opus-5-5
Most of the command-line contract was already tested through run. This
adds what was missing: report and trees with no database exit 1 with
the message telling the user to run scan; a scan that skips an
unreadable file prints its warning and counts the skip in its summary;
the report and trees summary lines are checked exactly. The scan tests
now capture the process's own stdout, which scan would write to
directly, so a stray stdout write in scan fails them. The fatal-path
test takes its subcommands from the command tree, so a new subcommand
wired without runE fails it. The nonexistent-operand test gets an
accurate name.
Model: opus-5-5
The shared lint config in the prompts repo moved from the deprecated
gomodguard linter to gomodguard_v2, with a module block list, and now
enables depguard to keep test-support packages out of non-test files.
This replaces the repo's copy with that file unchanged, so lint no
longer prints the gomodguard deprecation warning.
Model: opus-5-5
TestHashWorkerDropsQueuedRuns now passes hashWorker a hash function
that records being called, so a worker that hashes a run after the
scan is cancelled fails the test every time instead of only when it
then chose to send its result.
TestHashWorkerAbandonsBlockedSend cancels the scan from inside the
hash function and leaves the result channel unread, so the worker can
only return through the cancellation case beside its send. The scan
tests could not show this, because stop drains results and frees a
parked worker anyway.
Model: opus-5-5
openDB put the path into the connection string unescaped, so a ? or #
in it ended the file name and a % started an escape: scan could
silently fill a database under a shortened name. The path now goes
through net/url as a file: URI. An absolute path gets an empty host and
a relative path none, because SQLite reads what follows file:// up to
the next slash as a host name. The path is not cleaned, so it stays
exactly what the operator gave.
A test runs scan, report and trees against such a file name given as an
absolute path, as one starting with //, and as a relative path, and
checks that only that file and its lock file exist afterwards.
Model: opus-5-5
A --workers value of 0 or less used to be quietly raised to 1, so a
typo ran the whole scan on one worker with nothing on stderr to say
why. scan now refuses it before anything is scanned: one line on
stderr and exit 2, like the other usage errors. The clamp in runScan
is gone, and README states the rule and the default.
Model: opus-5-5
walkOneDir's check was hiding the worker's: a worker that walked a
queued directory on a cancelled scan still emitted nothing, because
walkOneDir stopped at its first entry. The worker test now queues a
missing directory, whose read fails and sends a warning before
walkOneDir's check is reached. A new test calls walkOneDir directly on
a cancelled scan and checks it returns no subdirectory to descend into.
Model: opus-5-5
The version-mismatch test only opened its database through
openReportDatabase, so nothing exercised the branch of initSchema that
stops scan on a database stamped with an unknown schema version. The
test now opens the same database through openScanDatabase too and
requires errSchemaVersion, and is renamed to match the unversioned-file
test beside it, which also covers both paths.
Model: opus-5-5
Fix the comments the re-review of
#6 found misdescribing their
tests; no test's behaviour changes.
State the property the walkClock tests rely on, that the index load's
Done cost does not grow with the record count, instead of a wrong fixed
figure. Describe poolUnwind so it is true of every use, and mark the
tests that have no bound. Record that hashWorker's results-send exit is
reachable through stop but has no test that fails without it. Rename
walkCancelInFlightDirs to walkCancelInFlightFiles, since it counts
files. Note at the top why the file departs from the
one-test-file-per-source-file convention.
Model: opus-4-8 (implementation); opus-5-5 (rework)
No test ever ran the part of subdirJob that refuses to descend across
a filesystem boundary under -x. The new tests call subdirJob directly
with a made-up device for the operand, so no second real filesystem is
needed. They cover refusal across a boundary, skipping the check when
the operand's device is unknown, the warning when the subdirectory
cannot be statted, and crossing by default without -x. When descent is
accepted, the whole returned job is compared, so losing the operand's
device on the way down fails a test. scan.go is unchanged.
Model: opus-4-8 (implementation); opus-5-5 (rebase)
A database at user_version 0 that already has a files table was made
by something else: scan used to run its CREATE TABLE on it and fail
with a raw SQLite error, and report and trees gave only a bare version
mismatch. All three now refuse such a database with the schema-version
error telling the operator to remove the file and rescan. scan creates
the table and index and sets the version in one transaction, so a first
scan stopped partway leaves an empty database the next scan sets up,
never a files table at version 0. A genuinely empty database is
unchanged.
Model: opus-4-8 (implementation); opus-5-5 (rebase)
Getting Started gains three parts: installing with go install, from a
clone or as the Docker image; a crontab line for a daily root scan,
with where its stderr and failures go; and how to read the two reports,
why a row is a candidate rather than proof, and how to check a pair
with cmp before removing anything. The usage block names --help, and
the text below it --workers and -x.
The install line uses @main, not @latest: @latest resolves to the
v0.0.1 tag, which predates the database.
Model: opus-5-5
The build stage handed /src and the whole Go module cache to the
unprivileged user with chown -R, in a layer that re-ran on every source
change. On this host that step took from about 80 s to over ten minutes,
depending on load.
The module cache now sits at /go/pkg/mod and stays root's: bootstrap
fills it as root. The sources are copied with --chown. A small chown,
cached with bootstrap, hands builder the /src directory itself, the
module cache's cache/download directory (where make build saves its
lookup of this module's own version), and the telemetry files root's go
commands left in its home. Tests and the build still run as builder.
Model: opus-5-5
Cobra's built-in version flag prints through the writer that carries
help and usage, which is stderr here. The root command now defines
its own -v/--version flag and prints one line, "sfdupes VERSION", to
stdout; a failed write is a fatal error (exit 1). Help and usage stay
on stderr. README documents --version and --help, their streams and
exit codes. Tests cover both flags and the failed write.
Model: opus-5-5
A first SIGINT or SIGTERM cancels the scan. It commits the hashed
records still in its batch, with a context that is not cancelled for
that one write, and starts no other write or deletion; deletions need
a complete walk, so records under paths an interrupted walk never
reached are kept. A batch whose commit failed is now kept for that
final commit instead of dropped. The progress display is finished (a
bar stopped short is no longer filled up), `scan: interrupted after N
files` goes to stderr, and the exit code is 1. A second signal ends the
process at once. A SIGINT inherited as ignored stays ignored.
Model: opus-5-5
report now has SQLite group the records and put the rows in report
order, helped by a new files_signature index on (size, head, tail,
content), and writes each row as it reads it. trees reads the records
in path order, where all the paths under a directory come together, so
it computes each directory's digest as soon as the stream leaves it and
keeps only its path, parent, digest and totals. Output is unchanged.
The tests that called the removed in-memory grouping functions now group
records stored in a database. New tests check that both commands give
the same output whatever order the records were inserted in, and that a
stdout failure partway through a long report is reported as one.
Model: opus-5-5
When stderr is not a terminal, each phase prints its zero-state line
as it starts instead of after its first item. stderrIsTTY uses
term.IsTerminal from golang.org/x/term, now a direct dependency, so
/dev/null is no longer taken for a terminal.
A spinner keeps the library's background redraw, so its count and
elapsed time stay current while a phase waits for its next item. A
warning printed during a spinner phase goes through the bar
(progressbar.Bprintln), which prints it before its next redraw instead
of racing it. Bars with a total have no background redraw and still
print warnings directly.
Model: opus-5-5
A symlink, socket, FIFO or device-node operand, or a directory operand
named .zfs, was silently ignored yet stayed in the scanned operands, so
the update phase deleted every record stored beneath it. Such an
operand now gets a one-line warning, counts as skipped, and is dropped
before overlapping operands are pruned and the database index is
loaded: another operand beneath it is still scanned, and the records
beneath it count as outside the scanned operands and are not deleted,
unless it lies under another operand. The exit status stays 0. An
operand that turns into one of these after that check is warned about
and skipped by the walk instead. README "scan mode" and "Rules for the
walk" say so.
Model: opus-5-5
scan takes an exclusive flock(2) on a lock file beside the database
(its path with .lock appended) before it walks anything or opens the
database, and holds it until it returns. A second scan against the
same database fails at once with a one-line error naming the lock
file and exits 1. report and trees never take the lock. The lock ends
with the process, so a fatal error or an interrupt releases it; the
file is never deleted. golang.org/x/sys becomes a direct dependency.
The README smoke test now keeps the database outside the scanned
tree, where its empty lock file would have joined the empty-file
group.
Model: opus-5-5
report and trees already checked every stdout write and the final
flush. run now takes the stdout it hands to them, so tests pass a
closed file or a failing writer instead of swapping os.Stdout: a
closed stdout exits 1 with a one-line diagnostic, and the writer's
error reaches the caller.
README "Error handling" now states the two cases that never reach
sfdupes as a failed write: a pipe reader that exits early ends the
process with SIGPIPE, as with cat; and stdout closed with >&- is
replaced by /dev/null by the Go runtime before main runs, so the run
succeeds.
Model: opus-5-5
report and trees now connect read-only (mode=ro, query_only, the same
busy timeout) and no longer set the journal mode, which is a write. A
read-only connection to a WAL database still needs its -wal and -shm
files, or write access to the directory to create them, so scan now
switches the database back to rollback-journal mode whenever it closes
it: between scans the file alone holds the database. If a report has
the database open at that moment the switch is refused; scan warns and
the database stays in WAL mode, with its -wal and -shm files, until the
next scan. README §Database states what readers need.
Model: opus-5-5
A path holding a tab or newline split a row of the report or trees
output. The path columns of both now write a backslash, tab, newline
and carriage return as \\, \t, \n and \r; every other byte is written
unchanged. Grouping and sorting still use the stored path. Warnings
on stderr are escaped the same way in warnf, so each stays one line.
In trees, the root directory's node now has the path "/" instead of
an empty string, and its children's paths start with a single slash.
README states the rule under "Report output format".
Model: opus-5-5
.dockerignore now sends .git, without .git/config, which can hold a
credential. The build stage takes the VERSION build argument when one
is given, otherwise git describe --tags --always of that .git, and
fails if the context carries .git and still yields no version. A plain
docker build . used to stamp dev. The CI checkout fetches full history
so CI sees the tag and stamps the same value as make build.
Model: opus-5-5
A file of 10 MiB or more now gets only its 64 KiB head and tail in the
hash phase, so its content is read only when it can be a duplicate. A
new content phase after the update phase finds every group of records,
anywhere in the database, that share size, head and tail and include
one without a content hash. It checks every member with lstat and, when
at least two pass, reads those without a content hash through the
existing worker pool; a stale file does not count as a match. report
and trees leave out records without a content hash. The README, help
text and TODO entry describe the gate; the schema stays at version 1.
Lint suppressed: gosec on the file open in hashContentOnly, as in
hashSignature, and on one chmod in a test.
Model: opus-5-5
sfdupes is pre-1.0, with no installed base and no databases anywhere,
so the schema is changed in place and its version stays 1.
schemaVersion goes back to 1; the six-column files table, content
included, is the version 1 schema. The check that stops on a database
with any other version stays. README.md and TODO.md no longer describe
a version 2 or rejecting and rescanning version 1 databases. The
main.go package comment still described 1024-byte end windows and said
full file contents are never read; it now describes the hashes the
code computes.
Model: opus-5-5
files.dat was the scan format before the SQLite database; nothing has
produced it since. Drop the stale references from the Makefile clean
target, .gitignore and .dockerignore. make clean still removes the
binary and .gitignore still covers the database files. The only
remaining mention is the historical entry in TODO.md.
Model: opus-4-8 (implementation); fable-5-1 (merge)
The `(Debian-based)` parenthetical broke the required
`# image:vX.Y.Z, YYYY-MM-DD` form and asserted a base change that never
happened (v2.12.1 was Debian too); the tag before the digest left three
FROM lines in one file using two conventions. Digest unchanged, in both
Dockerfile and Dockerfile.lint. The golang and alpine pin comments
already matched the required form.
script/verify-lint-image-pin parses these two FROM lines to keep them
identical and still matches the tagless form; its advice line drops the
now-meaningless "tag and digest". With no tag in either reference a
tag-only disagreement cannot arise; a tag reintroduced on one side is
caught as a plain mismatch.
Per the owner ruling the linter runs in a container invoked through the
script/ entrypoint, never installed on a host. Dockerfile.lint COPYs
the repo into the digest-pinned golangci/golangci-lint:v2.12.2 image
and runs `golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build IS a clean lint. script/lint is reduced to
building it, and works with a remote docker daemon, where bind mounts
are impossible.
script/bootstrap loses the `go install`, the pin constants, the version
parser and verify_golangci_lint: with nothing linting on the host, the
$GOPATH/bin versus PATH shadowing they diagnosed has no subject. It
keeps the git/make/go presence checks and `go mod download`, and warns
rather than fails when docker is absent.
Traps for anyone changing this.
A lint build on an unchanged tree exits 0 in under a second having run
no linter -- #32 and #39 again. Caching is waived by ruling:
Dockerfile.lint carries ARG CHECK_EPOCH referenced inside every gate
RUN, because BuildKit hashes the expanded command and a declared but
unreferenced ARG invalidates nothing. script/lint passes
"$(date +%s)-$$"; the PID is there because two runs land in the same
second easily and a bare epoch would cache the second.
Nothing inside an image build may shell out to docker. The main
Dockerfile's lint stage therefore invokes golangci-lint directly rather
than `make lint`, and its build stage runs `make test` and
`make fmt-check` rather than the `make check` aggregate, which reaches
script/lint. Both stay `make` invocations rather than bare scripts
because the Makefile's `export CGO_ENABLED = 0` only reaches what it
invokes.
COPY --from=lint /usr/bin/golangci-lint becomes
COPY --from=lint /src/go.sum /dev/null. The copied binary was the only
edge forcing BuildKit to finish linting before the build stage starts;
dropping it without replacing the edge would have ended fail-fast
linting silently under a still-green build. That no-op copy is the
ordering edge canonical REPO_POLICIES.md prescribes. Nothing in the
build stage runs the linter now, so ENV PATH=/home/builder/go/bin:$PATH
goes with the `go install` that justified it.
script/verify-linter-pin is retired with its README entry: it compared
a linter binary against GOLANGCI_LINT_VERSION in script/bootstrap and
neither subject still exists. The drift moved rather than went away --
the linter is pinned twice, as the FROM line of Dockerfile.lint and the
FROM line of the Dockerfile lint stage, which is what #42 made a build
failure. script/verify-lint-image-pin compares those two references to
each other and restates neither pin; a hardcoded digest would be a
third copy and the same drift one file further out. It runs as a gate
in both files, and an unreadable reference is a hard failure rather
than a vacuous pass.
`golangci-lint config verify` is included per the ruling, and its
unpinned live HTTPS schema fetch was measured rather than assumed:
under --network none the pinned binary passes a valid config and
rejects an invalid one with the jsonschema error, so it validates
against a schema it embeds. That holds for the gate steps, none of
which makes a network call, but not for the build around them --
Dockerfile.lint runs `go mod download` above the gates, so a cold cache
needs the network and only a warm one lints offline, until go.mod or
go.sum changes.
Verified. `make lint` green with every PATH directory containing a
golangci-lint removed and `command -v golangci-lint` empty. Two
consecutive script/lint runs on an untouched tree both executed the
linter, 27.7s and 28.7s under distinct epochs with the COPY layer
CACHED above them. A planted unused variable failed script/lint with
that finding, and failed `make docker` at the lint stage with the build
stage stopped before its COPY --from=lint; reverted clean. The drift
guard fails on tag-only, digest-only and unreadable-reference cases,
naming both sides. `make check` green; `make docker` green in 5m35s
with all six gates executing and the test gate reporting real coverage
rather than a cached ok. In the builder image with the Go test cache
off, --user 0:0 still fails TestScanHardlinkRunFailsTogether where the
unprivileged user passes, so the non-root quirk is intact.
The build stage installed prerequisites inline instead of running
script/bootstrap, which canonical REPO_POLICIES forbids. It now copies
script/ and the manifests and runs bootstrap.
The first attempt claimed that reordering COPY --from=lint made the
build fail loudly on linter drift. It did the opposite: bootstrap
reinstalled to GOPATH/bin, which ENV PATH placed ahead of the copied
binary, so verification passed by construction and lint and build
stages could silently run different linters. Rather than soften the
claim, the guarantee is now implemented. script/verify-linter-pin runs
against the copied binary before bootstrap and fails, naming both
versions, unless it matches the pin read out of script/bootstrap.
Reviewed independently. The reviewer reproduced the negative control in
both directions, including one the PR did not claim - bumping the pin
alone also fails the build - and established that no version output or
mangled pin file yields a silent pass.
The reordered COPY --from=lint did not make the two stages provably one
toolchain, as the Dockerfile comment, the previous commit message and
TODO.md all claimed. script/bootstrap compares its pin against whatever
PATH resolves, and $GOPATH/bin sits ahead of /usr/local/bin, so any
drift was absorbed: bootstrap rebuilt the pinned version from source,
verified that, and the build went green with the lint stage having
linted at one version and make check having run at another. Bumping the
lint stage image without touching the pin was enough to produce it.
New script/verify-linter-pin fails, naming both versions, unless a given
golangci-lint binary is exactly the version script/bootstrap pins. The
build stage runs it on the binary copied out of the lint stage,
immediately after the copy and before bootstrap, so no reinstall can
satisfy it. The pin is read out of script/bootstrap, which stays its
single source of truth; a pin that cannot be read is a hard failure
rather than a skip. The check takes no CHECK_EPOCH because its only
inputs are the copied binary and script/, so Docker invalidates the
layer exactly when a cached result would stop being true.
The linter version is pinned independently in the lint stage's image
digest and in GOLANGCI_LINT_VERSION, with nothing keeping them in sync;
a half-applied bump is now a build failure instead of a silent split.
ENV PATH keeps $GOPATH/bin, but its comment no longer claims a reinstall
is the reason: bootstrap must be able to run and verify what it
installs, and nothing in this image is shadowed by the entry.
Verified: with the lint stage's linter faked to 2.11.0 after the gates
had really run, the build fails at verify-linter-pin naming 2.11.0 and
2.12.2, with bootstrap and the check gate never reached; an unmodified
make docker is green with all three gates run on a fresh epoch and real
test results. A planted unused finding still fails at the lint stage
with gate check absent from the log; the image still fails
TestScanHardlinkRunFailsTogether under --user 0:0 and passes as uid
1000, both with the Go test cache disabled; and a second build serves
bootstrap, the verify layer and the dependency layers CACHED while the
gates go cold.
Canonical REPO_POLICIES.md:97 requires Dockerfiles to install
development prerequisites by running script/bootstrap rather than
duplicating installs inline. The build stage did the opposite: an
inline `apk add --no-cache make` and its own `go mod download`, so it
maintained a second, independent notion of the toolchain — the
local-versus-CI divergence #24 exists to close, reintroduced one layer
down.
The stage now copies script/ plus go.mod/go.sum and runs
script/bootstrap, which ends in `go mod download`.
COPY --from=lint /usr/bin/golangci-lint is kept and moved above the
bootstrap layer. It is the only edge making this stage depend on the
lint stage, so removing it as redundant would silently stop the build
gating on lint. Copying it first also puts it on PATH before bootstrap
runs, so bootstrap's version check compares the lint stage's linter
against the pin on every build: the two stages are now provably one
toolchain rather than two that happen to agree, and bootstrap does not
pay for a from-source build of its own linter.
$GOPATH/bin joins PATH so that if the copied binary ever stops matching
the pin, bootstrap's reinstall lands somewhere PATH resolves instead of
failing its own verification.
All of it sits above ARG CHECK_EPOCH, and the chown and USER builder
still precede make check.
clawbot
sneak
clawbot