Per the owner ruling the linter runs in a container invoked through the
script/ entrypoint, never installed on a host. Dockerfile.lint COPYs
the repo into the digest-pinned golangci/golangci-lint:v2.12.2 image
and runs `golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build IS a clean lint. script/lint is reduced to
building it, and works with a remote docker daemon, where bind mounts
are impossible.
script/bootstrap loses the `go install`, the pin constants, the version
parser and verify_golangci_lint: with nothing linting on the host, the
$GOPATH/bin versus PATH shadowing they diagnosed has no subject. It
keeps the git/make/go presence checks and `go mod download`, and warns
rather than fails when docker is absent.
Traps for anyone changing this.
A lint build on an unchanged tree exits 0 in under a second having run
no linter -- #32 and #39 again. Caching is waived by ruling:
Dockerfile.lint carries ARG CHECK_EPOCH referenced inside every gate
RUN, because BuildKit hashes the expanded command and a declared but
unreferenced ARG invalidates nothing. script/lint passes
"$(date +%s)-$$"; the PID is there because two runs land in the same
second easily and a bare epoch would cache the second.
Nothing inside an image build may shell out to docker. The main
Dockerfile's lint stage therefore invokes golangci-lint directly rather
than `make lint`, and its build stage runs `make test` and
`make fmt-check` rather than the `make check` aggregate, which reaches
script/lint. Both stay `make` invocations rather than bare scripts
because the Makefile's `export CGO_ENABLED = 0` only reaches what it
invokes.
COPY --from=lint /usr/bin/golangci-lint becomes
COPY --from=lint /src/go.sum /dev/null. The copied binary was the only
edge forcing BuildKit to finish linting before the build stage starts;
dropping it without replacing the edge would have ended fail-fast
linting silently under a still-green build. That no-op copy is the
ordering edge canonical REPO_POLICIES.md prescribes. Nothing in the
build stage runs the linter now, so ENV PATH=/home/builder/go/bin:$PATH
goes with the `go install` that justified it.
script/verify-linter-pin is retired with its README entry: it compared
a linter binary against GOLANGCI_LINT_VERSION in script/bootstrap and
neither subject still exists. The drift moved rather than went away --
the linter is pinned twice, as the FROM line of Dockerfile.lint and the
FROM line of the Dockerfile lint stage, which is what #42 made a build
failure. script/verify-lint-image-pin compares those two references to
each other and restates neither pin; a hardcoded digest would be a
third copy and the same drift one file further out. It runs as a gate
in both files, and an unreadable reference is a hard failure rather
than a vacuous pass.
`golangci-lint config verify` is included per the ruling, and its
unpinned live HTTPS schema fetch was measured rather than assumed:
under --network none the pinned binary passes a valid config and
rejects an invalid one with the jsonschema error, so it validates
against a schema it embeds. That holds for the gate steps, none of
which makes a network call, but not for the build around them --
Dockerfile.lint runs `go mod download` above the gates, so a cold cache
needs the network and only a warm one lints offline, until go.mod or
go.sum changes.
Verified. `make lint` green with every PATH directory containing a
golangci-lint removed and `command -v golangci-lint` empty. Two
consecutive script/lint runs on an untouched tree both executed the
linter, 27.7s and 28.7s under distinct epochs with the COPY layer
CACHED above them. A planted unused variable failed script/lint with
that finding, and failed `make docker` at the lint stage with the build
stage stopped before its COPY --from=lint; reverted clean. The drift
guard fails on tag-only, digest-only and unreadable-reference cases,
naming both sides. `make check` green; `make docker` green in 5m35s
with all six gates executing and the test gate reporting real coverage
rather than a cached ok. In the builder image with the Go test cache
off, --user 0:0 still fails TestScanHardlinkRunFailsTogether where the
unprivileged user passes, so the non-root quirk is intact.
The reordered COPY --from=lint did not make the two stages provably one
toolchain, as the Dockerfile comment, the previous commit message and
TODO.md all claimed. script/bootstrap compares its pin against whatever
PATH resolves, and $GOPATH/bin sits ahead of /usr/local/bin, so any
drift was absorbed: bootstrap rebuilt the pinned version from source,
verified that, and the build went green with the lint stage having
linted at one version and make check having run at another. Bumping the
lint stage image without touching the pin was enough to produce it.
New script/verify-linter-pin fails, naming both versions, unless a given
golangci-lint binary is exactly the version script/bootstrap pins. The
build stage runs it on the binary copied out of the lint stage,
immediately after the copy and before bootstrap, so no reinstall can
satisfy it. The pin is read out of script/bootstrap, which stays its
single source of truth; a pin that cannot be read is a hard failure
rather than a skip. The check takes no CHECK_EPOCH because its only
inputs are the copied binary and script/, so Docker invalidates the
layer exactly when a cached result would stop being true.
The linter version is pinned independently in the lint stage's image
digest and in GOLANGCI_LINT_VERSION, with nothing keeping them in sync;
a half-applied bump is now a build failure instead of a silent split.
ENV PATH keeps $GOPATH/bin, but its comment no longer claims a reinstall
is the reason: bootstrap must be able to run and verify what it
installs, and nothing in this image is shadowed by the entry.
Verified: with the lint stage's linter faked to 2.11.0 after the gates
had really run, the build fails at verify-linter-pin naming 2.11.0 and
2.12.2, with bootstrap and the check gate never reached; an unmodified
make docker is green with all three gates run on a fresh epoch and real
test results. A planted unused finding still fails at the lint stage
with gate check absent from the log; the image still fails
TestScanHardlinkRunFailsTogether under --user 0:0 and passes as uid
1000, both with the Go test cache disabled; and a second build serves
bootstrap, the verify layer and the dependency layers CACHED while the
gates go cold.
Four independent README defects, one docs-only pass. README.md is the
only file touched: the issue's definition of done restricts the change
to the README, so TODO.md is deliberately not updated here.
Remove the trailing attribution sentence from the Description. The
paragraph above it already carries authorship and licence, so nothing
is lost, and the naming it carried is not wanted in this repo's output.
Add the required Entrypoints section. House policy wants a README
section that opens by stating adherence to Scripts to Rule Them All,
links the standard, and documents every script/ entrypoint; grep for
"script/" in the README previously returned nothing, though all twelve
entrypoints exist and the Gitea workflow runs script/cibuild. Each
entry is written from reading the script, not from its name. Two
behaviours are called out because they are unusual and were hard-won:
script/bootstrap version-checks golangci-lint against its pin and
verifies the install is the copy PATH resolves rather than a shadowed
one, and script/docker and script/cibuild pass a fresh CHECK_EPOCH so
the Dockerfile's gate layers cannot be served from cache, which is
what keeps a green build from being one that ran nothing.
Fix the stale Build section. Every Makefile target became a thin shim
into script/, so the "Makefile is the single source of truth" claim is
no longer true; point at script/ instead. Document make bootstrap and
make setup, which exist and were undocumented. Drop the make clean
clause about a legacy local files.dat, a file nothing produces any
more; the remaining files.dat references in Makefile, .gitignore and
.dockerignore are #22 and are untouched.
Fix the scan summary example, which showed 123456 files seen against
components summing to 123400. removed is deliberately excluded from
the total, as the following paragraph says and as scan.go implements,
so the total is what was wrong.
Sort the hash queue by (device, inode) so reads proceed in inode
order, which minimizes seeking on spinning disks. Paths that are hard
links to the same inode form one run: the run is read once and every
path shares the result, so link farms (rsync --link-dest backups)
cost one read per inode instead of one per path. A run that fails to
read skips all of its paths.
Zero-length files have constant head/tail hashes; return them without
opening the file.
The hash progress total now counts actual reads (runs, not paths).
Hard-linked paths still appear in reports as duplicates — their
content is identical — though they share storage; noted in README.
On a database with tens of millions of records, indexing the existing
rows before the walk takes real single-core time with no output,
which is indistinguishable from a hang. Give the load its own
spinner, and render every phase display the moment the phase starts
instead of waiting for its first completed item.
Restructure scan into three phases: walk+stat, hash, update.
The stat pass is folded into the walk workers: each regular file is
lstatted as its directory is read, while the metadata is hot. The
walk builds a scan-wide size census (walked files plus records
outside the scan roots), and unchanged already-hashed files resolve
during the walk without further work.
Only files whose size at least one other file shares are ever read:
a size-unique file cannot be a duplicate, so it is recorded without
hashes (head and tail empty). When a later scan makes its size
shared, the file is hashed then, even if otherwise unchanged. report
excludes unhashed records; trees gives them a never-matching
signature so a tree containing one never compares equal to another.
Hashed records are committed in batched transactions while the hash
phase runs, so an interrupted scan keeps everything hashed so far
and the next run resumes cheaply. The hash phase total is exact,
giving a meaningful ETA.
Memory drops accordingly: the existing-record index holds only path,
size, mtime, and a hashed flag (no hash values); the walk carries one
small record per candidate file; overlapping operands are pruned up
front instead of deduplicating every walked path in a scan-wide set.
Files no bigger than one chunk are hashed with a single read.
Plain make now builds sfdupes (previously the default was all =
check + build); make build remains as an alias, so the Dockerfile
and existing habits keep working. The sfdupes target is phony: go
build's own cache decides what to recompile.
All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.
The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
With per-operand walk/hash/update cycles, a multi-operand invocation
(e.g. scan /srv/*) showed pass totals that looked like the whole
run's: an operator watching operand 3 of 14 hash 300k files concluded
the other 20M files were being skipped. Print the operand path and
its position before each cycle.
The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.
scan will maintain a persistent database of file signatures
(default /var/lib/sfdupes/db.sqlite, overridable via
SFDUPES_DATABASE) that survives between runs; rescans hash only new
or changed files (mtime/size) and remove records for vanished files,
so scan can be cronned daily. report and trees will read the
database instead of a scan stream.
scan now takes one or more PATH operands (directories or regular
files) via cobra flags instead of the -root flag with its /srv
default; invoking scan with no operand is a usage error and a
nonexistent operand is fatal. Filesystem boundaries are crossed by
default; the new -x/--one-file-system flag (GNU du/rsync convention)
stops the walk at each operand's filesystem, implemented by comparing
lstat device IDs with build-tagged helpers for darwin's int32 Dev.
Verified against a real mounted disk image: default crosses, -x does
not, --one-file-system is identical to -x.
Adds Description (name/purpose/category/license/author), Getting
Started, Rationale, TODO, License, and Author sections; the full
normative specification is preserved under Design. The stale non-goal
about having no git repository or CI is removed, and the Build section
now documents the Makefile targets.
