Author SHA1 Message Date
clawbot 282146a409 Reformat all Markdown with prettier
check / check (push) Successful in 2m2s
Mechanical result of `make fmt` now that prettier runs over Markdown:
README.md and TODO.md rewrapped to the house settings (4-space,
proseWrap always). REPO_POLICIES.md was already compliant. No prose was
changed by hand in this commit.

Model: opus-4-8
2026-09-21 07:32:12 +00:00
clawbot 2dd4b8c401 Restore Markdown formatting in fmt/fmt-check and pin prettier (closes #19)
script/fmt and script/fmt-check now run gofmt for Go and prettier for
Markdown; fmt-check reports each independently. prettier is pinned at
3.8.1 by package.json/yarn.lock (integrity hash); .prettierrc and
.prettierignore are the house settings copied from the prompts repo.
script/bootstrap installs node and yarn from the host package manager
and runs `yarn install --frozen-lockfile`.

The Markdown check runs in CI via the Dockerfile build stage, where
bootstrap provides prettier; it is removed from the lint stage because
the golangci-lint image has no node. JS manifests are copied before
bootstrap so the yarn layer caches, and node_modules is dockerignored.

node is an unpinned host runtime like git/make/go: nvm's glibc node does
not run on the musl/Alpine build image, so the canonical nvm route is
not usable here; prettier is the hash-pinned formatter. The wholesale
Markdown reformat follows in the next commit.

Model: opus-4-8
2026-09-21 07:31:53 +00:00
19 changed files with 929 additions and 1370 deletions
+2
View File
@@ -2,6 +2,8 @@
.claude
.DS_Store
sfdupes
files.dat
node_modules
*.log
*.out
*.test
+1
View File
@@ -27,6 +27,7 @@ node_modules/
*.log
# Local scan data
files.dat
*.sqlite
*.sqlite-shm
*.sqlite-wal
+2
View File
@@ -0,0 +1,2 @@
node_modules/
yarn.lock
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+11 -6
View File
@@ -27,9 +27,12 @@ ARG CHECK_EPOCH
# target now runs `docker build -f Dockerfile.lint`, and a docker build
# cannot run a docker build: routing the gate through make would mean
# nesting docker inside this image. Same reason `make check` is gone
# from the build stage below. `make fmt-check` stays as it is — it is a
# gate, not the aggregate, and it shells out to nothing.
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
# from the build stage below.
#
# `make fmt-check` is not run in this stage: it now also runs prettier
# over Markdown, and this golangci-lint image has no node. The gate runs
# in the build stage below, where script/bootstrap installs node and
# prettier.
# The FROM above and the one in Dockerfile.lint pin the same linter
# twice, and nothing else keeps them in sync; this fails the build when
@@ -77,10 +80,12 @@ COPY --from=lint /src/go.sum /dev/null
# rather than duplicating the installs inline. Only script/ and the
# dependency manifests are copied first, nothing else, so this layer
# stays cached until the scripts or the dependencies change — bootstrap
# ends in `go mod download`, which is why there is no separate
# invocation of it here.
# runs `go mod download` and `yarn install`, which is why there is no
# separate invocation of either here. The JS manifests (package.json,
# yarn.lock) are copied too so the yarn install layer caches alongside
# the Go one.
COPY script/ script/
COPY go.mod go.sum ./
COPY go.mod go.sum package.json yarn.lock ./
RUN script/bootstrap
COPY . .
+1 -1
View File
@@ -46,4 +46,4 @@ hooks:
@script/install-precommit
clean:
rm -f $(BINARY)
rm -f $(BINARY) files.dat
+404 -515
View File
File diff suppressed because it is too large Load Diff
+335 -398
View File
@@ -1,449 +1,386 @@
# Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not
yet on the tracker gets filed as an issue first
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
tracker gets filed as an issue first
- branch (from `main`)
- do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the
same commit as the work)
- push the branch and open a PR whose title ends with
` (closes #N)`
- an independent review gates the merge; every finding is addressed
or explicitly rebutted on the PR
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
as the work)
- push the branch and open a PR whose title ends with ` (closes #N)`
- an independent review gates the merge; every finding is addressed or
explicitly rebutted on the PR
- merge to `main` once the review passes
# Status
- pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
open issues under the `1.0.0` milestone are what remains before
the tag, and this file records history and process, not the queue
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
under the `1.0.0` milestone are what remains before the tag, and this file
records history and process, not the queue
# Next Step
- take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
the source of truth for what is left before 1.0.0. Individual
issues are deliberately not restated here; a copy in this file
drifts out of date the moment the tracker moves
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
of truth for what is left before 1.0.0. Individual issues are deliberately not
restated here; a copy in this file drifts out of date the moment the tracker
moves
# Completed Steps
- replace the 1 KiB end-window sampling with the head/tail plus
content-hash ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
hashed in full and compared directly, with no end-window step — its
`head`, `tail`, and `content` all hold the whole-file hash. A file at
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
compared on a `content` hash — the whole file below 50 MiB,
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
(new `content` column); a version 1 database is rejected and must be
rescanned, which is required anyway since every stored hash changed.
`report` and `trees` group by the extended signature, so the ladder is
applied across the whole database. README "Duplicate detection"
documents every rung including the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/22)
- restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all
Markdown to the house prettier settings (2026-09-21, closes
https://git.eeqj.de/sneak/sfdupes/issues/19)
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the
redundant tag, so both pins are the policy `# image:vX.Y.Z,
YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest
unchanged. `script/verify-lint-image-pin` parses those `FROM` lines
and still matches the tagless form; its advice line lost the now
meaningless "tag and digest". With no tag in either reference, a
tag-only disagreement no longer exists — a one-sided tag is caught as
a plain mismatch.
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
parses those `FROM` lines and still matches the tagless form; its advice line
lost the now meaningless "tag and digest". With no tag in either reference, a
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint`
(2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
linter runs inside a container invoked through the `script/`
entrypoint and is never installed on a host. New root
`Dockerfile.lint` COPYs the repo into the digest-pinned
`golangci/golangci-lint:v2.12.2` image and runs
`golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build IS a clean lint; `script/lint` is
reduced to building it. `script/bootstrap` loses the `go install`,
the pin constants, the version parser and `verify_golangci_lint`
outright rather than hardening them — with nothing linting on the
host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
no subject — and now warns rather than fails when `docker` is absent.
Two traps handled. A lint build on an unchanged tree returns success
in well under a second having run no linter, which is
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling, the linter runs inside a container invoked through the `script/`
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
a successful build IS a clean lint; `script/lint` is reduced to building it.
`script/bootstrap` loses the `go install`, the pin constants, the version
parser and `verify_golangci_lint` outright rather than hardening them — with
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
motivated them has no subject — and now warns rather than fails when `docker`
is absent. Two traps handled. A lint build on an unchanged tree returns
success in well under a second having run no linter, which is
https://git.eeqj.de/sneak/sfdupes/issues/32 and
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
`Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
inside every gate `RUN` (BuildKit hashes the expanded command, not
the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
PID matters because two lint runs land inside the same second easily.
And nothing inside an image build may shell out to docker, so the
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
the expanded command, not the declaration) and `script/lint` passes
`"$(date +%s)-$$"` the PID matters because two lint runs land inside the
same second easily. And nothing inside an image build may shell out to docker,
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate (`make`, not
the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
only reaches what it invokes). `COPY --from=lint`
`/usr/bin/golangci-lint` is replaced by
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
only edge forcing BuildKit to finish linting before the build stage
starts, and dropping it without replacing the edge would have ended
fail-fast linting silently under a still-green build. That is
canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
`ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
that justified it. `script/verify-linter-pin` is retired, deleted
along with its README entry, because both of its subjects ceased to
exist in the same change: it compared a linter binary against
`GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
neither a binary crossing between stages nor a version pin in
bootstrap. The drift it guarded has not gone away, it has moved — the
linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
with nothing syncing them, which is exactly what
`make fmt-check` instead of the `make check` aggregate (`make`, not the
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
forcing BuildKit to finish linting before the build stage starts, and dropping
it without replacing the edge would have ended fail-fast linting silently
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
along with its README entry, because both of its subjects ceased to exist in
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
in `script/bootstrap`, and there is now neither a binary crossing between
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
it has moved — the linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`,
run as a gate in both files, which compares the two references to
each other and deliberately restates neither: a hardcoded expected
digest would be a third copy and the same drift one file further out.
`golangci-lint config verify` is included per the ruling, and the
concern about its unpinned live HTTPS schema fetch was measured
rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema
error, so it validates from an embedded schema and makes no network
call of its own. The README scopes that to the gate steps rather
than to linting as a whole: `Dockerfile.lint` runs `go mod download`
above them, so a cold cache still needs the network and only a warm
one lints offline. Verified: `make lint` green with every `PATH`
directory containing a `golangci-lint` removed
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
files, which compares the two references to each other and deliberately
restates neither: a hardcoded expected digest would be a third copy and the
same drift one file further out. `golangci-lint config verify` is included per
the ruling, and the concern about its unpinned live HTTPS schema fetch was
measured rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema error, so
it validates from an embedded schema and makes no network call of its own. The
README scopes that to the gate steps rather than to linting as a whole:
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
needs the network and only a warm one lints offline. Verified: `make lint`
green with every `PATH` directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs
on an untouched tree both executed the linter, 27.7s and 28.7s in the
lint step under distinct epochs with the `COPY . .` layer `CACHED`
above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
weakened to shorten that. Negative control: a planted
`var unusedIssue46Sentinel = 1` failed `script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
failed `make docker` at `[lint 9/9]` with the build stage stopped at
`[builder 3/12]``COPY --from=lint`, `script/bootstrap`, the test
gate and `make build` all zero occurrences — then reverted clean. The
drift guard fails on a tag-only disagreement, on a digest-only
disagreement, and on an unreadable reference, naming both sides.
`make docker` green in 5m35s with all six gates executing under one
epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
The non-root quirk still holds: in the builder image with the Go test
cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
(exit 1) where the unprivileged user passes (exit 0). Noted for
follow-up, not fixed here: `golangci-lint` warns that the
`gomodguard` linter is deprecated since v2.12.0 in favour of
`gomodguard_v2`.
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
`script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
`COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
zero occurrences — then reverted clean. The drift guard fails on a tag-only
disagreement, on a digest-only disagreement, and on an unreadable reference,
naming both sides. `make docker` green in 5m35s with all six gates executing
under one epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
non-root quirk still holds: in the builder image with the Go test cache off,
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running
`script/bootstrap` instead of `apk add --no-cache make` inline
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
`REPO_POLICIES.md:97` requires it, and the inline install left the
build stage maintaining its own notion of the toolchain — exactly
the divergence #24 exists to close, one layer down. The stage now
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
which ends in `go mod download`, so the separate invocation of that
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
above the bootstrap layer. It is the only edge making this stage
depend on the lint stage, so deleting it as redundant would end
fail-fast linting silently. Letting bootstrap install its own linter
here would have reintroduced the second toolchain and paid for a
from-source build of it. What makes the two stages provably one
toolchain rather than two that happen to agree is a new
`script/verify-linter-pin`, run in the build stage on the binary
that arrives from the lint stage, before bootstrap: it fails the
build naming both versions unless that binary is the version
`script/bootstrap` pins. Bootstrap's own check could not serve that
purpose — it reinstalls its pin from source and then verifies
whatever `PATH` resolves, so drift self-heals silently and a lint
stage image bumped on its own would lint at the new version while
`make check` ran at the old one, green. The linter version is pinned
in two independent places (the lint stage image digest and
- install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
it, and the inline install left the build stage maintaining its own notion of
the toolchain — exactly the divergence #24 exists to close, one layer down.
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
`script/bootstrap`, which ends in `go mod download`, so the separate
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
and moves above the bootstrap layer. It is the only edge making this stage
depend on the lint stage, so deleting it as redundant would end fail-fast
linting silently. Letting bootstrap install its own linter here would have
reintroduced the second toolchain and paid for a from-source build of it. What
makes the two stages provably one toolchain rather than two that happen to
agree is a new `script/verify-linter-pin`, run in the build stage on the
binary that arrives from the lint stage, before bootstrap: it fails the build
naming both versions unless that binary is the version `script/bootstrap`
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
pin from source and then verifies whatever `PATH` resolves, so drift
self-heals silently and a lint stage image bumped on its own would lint at the
new version while `make check` ran at the old one, green. The linter version
is pinned in two independent places (the lint stage image digest and
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
half-applied bump is now a build failure. The pin is read out of
`script/bootstrap`, which stays the single source of truth; a pin
that cannot be read is a hard failure, not a skip. The check needs
no `CHECK_EPOCH`: its only inputs are the copied binary and
`script/`, so Docker invalidates the layer exactly when a cached
result would stop being true, and it is documented with the other
entrypoints in the README. `$GOPATH/bin` joins `PATH` because
that is where bootstrap's `go install` lands and bootstrap verifies
its installs against what `PATH` resolves — nothing in the image is
shadowed by it, the directory does not exist until bootstrap runs.
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
`USER builder` still precede `make check`. Verified: the guard fails
the build with both versions named when the lint stage's linter is
faked to a different version, and an unmodified build still passes
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
installing `git` and `make` and finding the copied
linter already at the pin; a second build served the bootstrap and
dependency layers `CACHED` while both gates ran with a fresh epoch;
a planted `unused` finding failed the build at the lint gate in
48.9s with the build stage's `make check` never starting; and the
suite run in the image as `--user 0:0` fails
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
user is still load-bearing. That last check needs the Go test cache
disabled — the first attempt reported `ok ... (cached)` as root,
reusing the result the build-time run had left in the shared cache,
which would have read as a pass. Build wall time, on a shared host
running many concurrent builds and so noisy: 2m13s on an unchanged
tree, 2m17s and 4m29s for two builds after a source change, 5m14s
cold. Only the cold one breaches the policy ceiling, and not because
of this change — `chown -R builder:builder /src /home/builder` walks
the module cache and re-runs on every source change, and it alone
varied between 77s and 210s across those four builds, which is also
the whole spread in the totals. The same cold measurement against
`main` is 5m03s with a 209s `chown`. Filed as #43
- bust the Docker layer cache for the gate steps, so `script/cibuild`
and `script/docker` cannot report a green they did not earn
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
were bare `docker build` invocations with no cache control, and the
`Dockerfile` copies the tree before running its gates, so on an
unchanged tree Docker served those layers from cache and the build
exited 0 having executed nothing. That is not hypothetical here —
every merge this repo has done is a non-fast-forward merge of an
undiverged branch, so each merge commit's tree is byte-identical to
the branch head's and each merge CI run was almost certainly a full
cache hit; and PR #31's reviewer found `make docker` returning
success as a 17-layer cache hit, catching it only by being
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
it. `ARG` is scoped per stage and this `Dockerfile` has three gates
across two — `make fmt-check` and `make lint` in the lint stage,
`make check` in the build stage — so a single declaration would have
left one stage silently cacheable; it is declared in both. And
BuildKit hashes the expanded command, not the declaration, so a
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
echoes the epoch, which also puts the value in the build log as
evidence the layer really ran. Placement is below the dependency
layers on purpose — a build that goes cold every time would be a
different bug, not a fix. Verified by running each script twice back
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
three gates executed on all four runs, each with a fresh epoch in
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
then 53.4s), and twelve steps were still served `CACHED` in the
steady state — both `go mod download`s, `apk add`, `adduser`, the
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy
out of the lint stage, and the binary copy into the runtime stage.
The lint stage still gates the build stage: with a deliberate
`unused` finding planted in the tree, the build failed at
`make lint` in 36.1s and the build-stage `make check` never started.
The build stage also still drops to the unprivileged `builder` user
before `make check`, which the suite depends on rather than merely
prefers: forcing the same image to run the tests as root fails
`TestScanHardlinkRunFailsTogether`, because root reads straight
through the `chmod(0)` the test uses to prove hard links are read
once. This is the local fix only; propagating it to the canonical
templates is `prompts` #26
- check the installed golangci-lint version in `script/bootstrap`
instead of only its presence (2026-08-09, branch
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant
any linter already on `PATH` satisfied the check, so the pin was never
consulted and the v2.12.2 bump from #3 was inert on every host that
already had one — this host ran v2.10.1 against a v2.12.2 pin,
`make check` went green, and `make docker` then rejected the same
commit with findings the local gate never saw. The version now lives
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
ref derived from it so a bump cannot half-apply; a
`golangci_lint_version` helper parses `golangci-lint --version`
(taking the field after the word `version` and tolerating an optional
leading `v`, which the module ref carries and the binary's output does
not), and any version that is not the pin — older, newer, absent or
unparseable — is reinstalled. The install is then verified against the
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
stage makes — would swallow the install and leave the local gate
disagreeing with CI under an affirmative `bootstrap complete`.
Bootstrap now re-reads the effective version after installing and, on
a mismatch, prints both paths and both versions to stderr and exits
non-zero instead of claiming success; it does not reorder anyone's
`script/bootstrap`, which stays the single source of truth; a pin that cannot
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
only inputs are the copied binary and `script/`, so Docker invalidates the
layer exactly when a cached result would stop being true, and it is documented
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
that is where bootstrap's `go install` lands and bootstrap verifies its
installs against what `PATH` resolves — nothing in the image is shadowed by
it, the directory does not exist until bootstrap runs. Everything added sits
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
`make check`. Verified: the guard fails the build with both versions named
when the lint stage's linter is faked to a different version, and an
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
its `apk` branch, installing `git` and `make` and finding the copied linter
already at the pin; a second build served the bootstrap and dependency layers
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
failed the build at the lint gate in 48.9s with the build stage's `make check`
never starting; and the suite run in the image as `--user 0:0` fails
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
still load-bearing. That last check needs the Go test cache disabled — the
first attempt reported `ok ... (cached)` as root, reusing the result the
build-time run had left in the shared cache, which would have read as a pass.
Build wall time, on a shared host running many concurrent builds and so noisy:
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
because of this change — `chown -R builder:builder /src /home/builder` walks
the module cache and re-runs on every source change, and it alone varied
between 77s and 210s across those four builds, which is also the whole spread
in the totals. The same cold measurement against `main` is 5m03s with a 209s
`chown`. Filed as #43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
invocations with no cache control, and the `Dockerfile` copies the tree before
running its gates, so on an unchanged tree Docker served those layers from
cache and the build exited 0 having executed nothing. That is not hypothetical
here — every merge this repo has done is a non-fast-forward merge of an
undiverged branch, so each merge commit's tree is byte-identical to the branch
head's and each merge CI run was almost certainly a full cache hit; and PR
#31's reviewer found `make docker` returning success as a 17-layer cache hit,
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
across two — `make fmt-check` and `make lint` in the lint stage, `make check`
in the build stage — so a single declaration would have left one stage
silently cacheable; it is declared in both. And BuildKit hashes the expanded
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
nothing: each gate `RUN` echoes the epoch, which also puts the value in the
build log as evidence the layer really ran. Placement is below the dependency
layers on purpose — a build that goes cold every time would be a different
bug, not a fix. Verified by running each script twice back to back on an
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
the lint stage, and the binary copy into the runtime stage. The lint stage
still gates the build stage: with a deliberate `unused` finding planted in the
tree, the build failed at `make lint` in 36.1s and the build-stage
`make check` never started. The build stage also still drops to the
unprivileged `builder` user before `make check`, which the suite depends on
rather than merely prefers: forcing the same image to run the tests as root
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
the `chmod(0)` the test uses to prove hard links are read once. This is the
local fix only; propagating it to the canonical templates is `prompts` #26
- check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
`missing golangci-lint` meant any linter already on `PATH` satisfied the
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
on every host that already had one — this host ran v2.10.1 against a v2.12.2
pin, `make check` went green, and `make docker` then rejected the same commit
with findings the local gate never saw. The version now lives in one place,
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
bump cannot half-apply; a `golangci_lint_version` helper parses
`golangci-lint --version` (taking the field after the word `version` and
tolerating an optional leading `v`, which the module ref carries and the
binary's output does not), and any version that is not the pin — older, newer,
absent or unparseable — is reinstalled. The install is then verified against
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
the install and leave the local gate disagreeing with CI under an affirmative
`bootstrap complete`. Bootstrap now re-reads the effective version after
installing and, on a mismatch, prints both paths and both versions to stderr
and exits non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than
reinstalling forever in silence, and is bounded by `timeout(1)` where
that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
and `go` keep their presence-only checks and now say why in a
comment: they are host package-manager tools the repo deliberately
does not pin, with `go.mod` governing the language version and the
digest-pinned images covering reproducible builds. Verified on this
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
a no-op, plus stub runs of the real script under `dash` covering a
thirteen-input parse matrix (absent, older, newer, host-style,
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
binary, `(devel)`, trailing `version`), a shadowed install that must
exit non-zero, an install destination not on `PATH` at all, `GOBIN`
set, and a wedged binary that must hit the timeout; `make check` and
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
findings on `main`
connected, so a present-but-broken binary says why rather than reinstalling
forever in silence, and is bounded by `timeout(1)` where that exists, so a
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
presence-only checks and now say why in a comment: they are host
package-manager tools the repo deliberately does not pin, with `go.mod`
governing the language version and the digest-pinned images covering
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
v2.12.2 and running it again to a no-op, plus stub runs of the real script
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
wedged binary that must hit the timeout; `make check` and `make lint` are
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
moment `recordRun` failed and abandon the pool — the feeder parked
forever on a full `jobs` channel and every worker on a full
`results` channel. That only stopped being invisible when #4 landed
and `runScan` began unwinding instead of calling `os.Exit`. The
pool is now an owned, context-aware `hashPool`: every blocking send
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
closed on every path out, and `hashPhase` defers `pool.stop()`,
which cancels and then drains `results` until the last goroutine
has exited — draining is what frees a worker already parked on a
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
`syncScan`, both worker pools and the whole database layer (it is
the first parameter everywhere), so #5 can hand this path a signal
and needs to add nothing else. The walk pool never leaked, because
`walkPhase` always drains its events to close, but it has the same
unbounded-send shape and #5 will give it an early return, so it
gets the same treatment plus a `ctx.Err()` guard after the walk: a
cancelled walk yields a partial size census, and every file it never
reached looks vanished to the update phase. That phase's own
`BeginTx` fails on the same cancelled context before deleting
anything, so the guard is defence in depth rather than the only
barrier — but it is the one that survives #5 deciding an interrupted
scan may commit what it has. Tests drive `run(scan)` against a
database whose insert trigger aborts, and assert both that the scan
fails instead of hanging and that `runtime.NumGoroutine()` polls
back to its pre-scan baseline; a second set cancels a scan part-way
through the walk — deterministically, by counting the scan's own
consultations of `ctx.Done()` rather than racing a timer — and
asserts that it stops at the guard holding a partial census and a
still-populated record index, with every record intact. The
remaining cancellation branches of both pools are covered by direct
tests of `sendEvent`, the walk workers, `dispatchDirs`,
`feedHashJobs`, `hashWorker` and `hashPhase`
- guarantee the database is closed on every fatal exit path
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
it the SQLite WAL checkpoint — now actually runs when a subcommand
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
`resolveRoots` return errors instead. The single exit point is `run`
in `main.go`: it maps a `fatalError` (anything a subcommand
returned) to exit 1 and cobra's own argument and flag errors to exit
2, which keeps a runtime failure from being reported as a usage
error or printing the usage text. New `main_test.go` drives the CLI
in-process and asserts the exit codes from README §Error handling
plus the stdout/stderr split, including that a fatal error raised
after the database is open leaves no `-wal`/`-shm` sidecar behind
for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
closes #3): bumped the pinned linter in the `Dockerfile` lint
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
`.golangci.yml` with the canonical file — the linter settings
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
`recordRun` failed and abandon the pool — the feeder parked forever on a full
`jobs` channel and every worker on a full `results` channel. That only stopped
being invisible when #4 landed and `runScan` began unwinding instead of
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
and then drains `results` until the last goroutine has exited — draining is
what frees a worker already parked on a send. `ctx` is threaded from
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
database layer (it is the first parameter everywhere), so #5 can hand this
path a signal and needs to add nothing else. The walk pool never leaked,
because `walkPhase` always drains its events to close, but it has the same
unbounded-send shape and #5 will give it an early return, so it gets the same
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
partial size census, and every file it never reached looks vanished to the
update phase. That phase's own `BeginTx` fails on the same cancelled context
before deleting anything, so the guard is defence in depth rather than the
only barrier — but it is the one that survives #5 deciding an interrupted scan
may commit what it has. Tests drive `run(scan)` against a database whose
insert trigger aborts, and assert both that the scan fails instead of hanging
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
second set cancels a scan part-way through the walk — deterministically, by
counting the scan's own consultations of `ctx.Done()` rather than racing a
timer — and asserts that it stops at the guard holding a partial census and a
still-populated record index, with every record intact. The remaining
cancellation branches of both pools are covered by direct tests of
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
and `hashPhase`
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
`loadRecords` and `resolveRoots` return errors instead. The single exit point
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
runtime failure from being reported as a usage error or printing the usage
text. New `main_test.go` drives the CLI in-process and asserts the exit codes
from README §Error handling plus the stdout/stderr split, including that a
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
behind for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
and replaced `.golangci.yml` with the canonical file — the linter settings
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`linters.settings` per the v2 schema, so they are actually
applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos (2026-07-26, commit
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`) and every Makefile target is now a thin shim
over them, matching the other managed repos
`linters.settings` per the v2 schema, so they are actually applied; no new
lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`) and every Makefile target is now a thin shim over them,
matching the other managed repos
- make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes`
(previously it ran `check` plus `build`); `make build` remains as
an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
branch `scan-wide-phases`): all operands seed the shared walk pool
and every pass runs once over the whole scan, so totals and ETAs
are scan-global; the per-operand walk/hash/update cycles and their
stderr announcements are gone; the update pass commits in batched
transactions — the filesystem is authoritative and the database an
eventually-consistent reflection, so scan-level atomicity is not
required
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
`check` plus `build`); `make build` remains as an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
`scan-wide-phases`): all operands seed the shared walk pool and every pass
runs once over the whole scan, so totals and ETAs are scan-global; the
per-operand walk/hash/update cycles and their stderr announcements are gone;
the update pass commits in batched transactions — the filesystem is
authoritative and the database an eventually-consistent reflection, so
scan-level atomicity is not required
- split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk,
stat, hash, update per operand — with parallelism only inside each
phase; the walk enumerates paths with per-directory workers and the
stat pass lstats them with per-file workers, restoring the exact
total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24,
branch `scan-operand-progress`): with per-operand walk/hash/update
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
that looked like the whole run's — an operator watching operand 3 of
14 hash 300k files concluded 20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
was a single goroutine and took hours at ~20M files on a busy pool
(observed: 22M files in 4h on a ZFS server); it is now a
per-directory worker-pool traversal that records size/mtime during
the walk (folding away the separate stat pass, halving metadata
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
update per operand — with parallelism only inside each phase; the walk
enumerates paths with per-directory workers and the stat pass lstats them with
per-file workers, restoring the exact total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24, branch
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
whole run's — an operator watching operand 3 of 14 hash 300k files concluded
20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
4h on a ZFS server); it is now a per-directory worker-pool traversal that
records size/mtime during the walk (folding away the separate stat pass,
halving metadata I/O), and each `PATH` operand commits in its own transaction
so an interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`):
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
Go, cgo stays disabled) keyed by absolute path that survives between
runs — a rescan hashes only new or changed files (by mtime/size),
deletes records for files vanished from under the scanned operands,
and leaves records outside them untouched, so `scan` can be cronned
daily; `report` and `trees` read the database (no positional
arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
WAL journaling plus a single-transaction update keep a report run
during a scan safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
`v0.0.1`, and push `main` plus tags (2026-07-23)
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
disabled) keyed by absolute path that survives between runs — a rescan hashes
only new or changed files (by mtime/size), deletes records for files vanished
from under the scanned operands, and leaves records outside them untouched, so
`scan` can be cronned daily; `report` and `trees` read the database (no
positional arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
journaling plus a single-transaction update keep a report run during a scan
safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root`
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
at filesystem boundaries, which are crossed by default
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on
`main` (2026-07-22)
- `git init` with README-only first commit; code baseline committed on `main`
(2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers
- possible later features (explicitly out of scope per README): full-content
verification of candidates, removal-script helpers
# Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
standalone formatting commit is needed.
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
`files.dat` sit untracked in the tree; needs OS/editor/Go
artifacts plus secrets patterns
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
- [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed
from house convention — user to confirm)
- [x] `LICENSE` missing and README has no License section (MIT assumed from
house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then
pass `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
`fmt-check`, `docker`, `hooks`; `check` currently depends on
`build`, which writes the binary (`make check` must not modify
files)
- [x] no tests — `go test ./...` has nothing to run; policy requires
real tests with a 30-second timeout and the conditional `-v`
rerun pattern (suite covers parsing, grouping, digests,
suppression, hashing, and the scan pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
fail-fast lint stage, build stage running `make check`
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
`make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
`docker`, `hooks`; `check` currently depends on `build`, which writes the
binary (`make check` must not modify files)
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
with a 30-second timeout and the conditional `-v` rerun pattern (suite
covers parsing, grouping, digests, suppression, hashing, and the scan
pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
lint stage, build stage running `make check`
- [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
checkout action pinned by commit SHA)
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
action pinned by commit SHA)
- [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started,
Rationale, TODO, License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now
that the repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target
exists)
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the
repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists)
Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine
for a small single-binary tool per the Go styleguide; the tracker
audit agrees
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
(pure-Go builds) and the race detector requires cgo
- flat single-package layout with `.go` files in the repo root — fine for a
small single-binary tool per the Go styleguide; the tracker audit agrees
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
builds) and the race detector requires cgo
+13 -19
View File
@@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
const databaseEnv = "SFDUPES_DATABASE"
// schemaVersion is the database schema version this build reads and
// writes, stored in PRAGMA user_version. Version 2 adds the content
// column and the head/tail/content signature (replacing the version 1
// 1 KiB end windows), so a version 1 database is rejected and must be
// rescanned.
const schemaVersion = 2
// writes, stored in PRAGMA user_version.
const schemaVersion = 1
// dbDirPerm is the mode for a database parent directory created by
// scan.
@@ -39,24 +36,22 @@ const dbDirPerm = 0o755
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
const createTableSQL = `
CREATE TABLE files (
path BLOB PRIMARY KEY,
size INTEGER NOT NULL,
mtime INTEGER NOT NULL,
head TEXT NOT NULL,
tail TEXT NOT NULL,
content TEXT NOT NULL
path BLOB PRIMARY KEY,
size INTEGER NOT NULL,
mtime INTEGER NOT NULL,
head TEXT NOT NULL,
tail TEXT NOT NULL
) WITHOUT ROWID
`
// upsertSQL inserts one file record, replacing any existing record for
// the same path.
const upsertSQL = `
INSERT INTO files (path, size, mtime, head, tail, content)
VALUES (?, ?, ?, ?, ?, ?)
INSERT INTO files (path, size, mtime, head, tail)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT (path) DO UPDATE SET
size = excluded.size, mtime = excluded.mtime,
head = excluded.head, tail = excluded.tail,
content = excluded.content
head = excluded.head, tail = excluded.tail
`
// errNoDatabase reports a missing database file for report/trees.
@@ -210,7 +205,7 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
// loadFileRows reads every record from the files table.
func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, head, tail, content FROM files")
"SELECT path, size, mtime, head, tail FROM files")
if err != nil {
return nil, fmt.Errorf("read records: %w", err)
}
@@ -225,8 +220,7 @@ func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
r scanRec
)
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail,
&r.content)
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail)
if err != nil {
return nil, fmt.Errorf("read record: %w", err)
}
@@ -354,7 +348,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
for _, r := range upserts {
_, err = st.ExecContext(ctx,
[]byte(r.path), r.size, r.mtime, r.head, r.tail, r.content)
[]byte(r.path), r.size, r.mtime, r.head, r.tail)
if err != nil {
return fmt.Errorf("upsert %s: %w", r.path, err)
}
+5
View File
@@ -0,0 +1,5 @@
{
"devDependencies": {
"prettier": "3.8.1"
}
}
+10 -14
View File
@@ -17,15 +17,14 @@ const ioBufSize = 1 << 20
const minGroupSize = 2
// scanRec is one file record from the database. The signature (size,
// head, tail, content) is the duplicate key; mtime is informational
// only and used by scan for change detection.
// head, tail) is the duplicate key; mtime is informational only and
// used by scan for change detection.
type scanRec struct {
size int64
mtime int64
head string
tail string
content string
path string
size int64
mtime int64
head string
tail string
path string
}
// loadRecords opens the database and reads every file record for the
@@ -53,9 +52,8 @@ func loadRecords(ctx context.Context) ([]scanRec, error) {
}
// dupeGroup is one set of candidate-duplicate files: identical size,
// head hash, tail hash, and content hash. paths is sorted
// lexicographically; the first entry is the group's "first", the rest
// are dupes.
// head hash, and tail hash. paths is sorted lexicographically; the
// first entry is the group's "first", the rest are dupes.
type dupeGroup struct {
size int64
paths []string
@@ -124,9 +122,7 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
continue
}
k := fileSig{
size: r.size, head: r.head, tail: r.tail, content: r.content,
}
k := fileSig{size: r.size, head: r.head, tail: r.tail}
groups[k] = append(groups[k], r.path)
}
-22
View File
@@ -38,28 +38,6 @@ func TestCollectDupeGroups(t *testing.T) {
}
}
func TestCollectDupeGroupsContentSeparates(t *testing.T) {
t.Parallel()
// Same size, head, and tail, but different content hashes: the final
// rung keeps them apart, so no group forms. Matching content groups.
recs := []scanRec{
{size: 100, head: "h", tail: "t", content: "c1", path: "/a"},
{size: 100, head: "h", tail: "t", content: "c2", path: "/b"},
{size: 100, head: "h", tail: "t", content: "c1", path: "/c"},
}
groups := collectDupeGroups(recs)
if len(groups) != 1 {
t.Fatalf("len(groups) = %d, want 1 (only the matching content)",
len(groups))
}
if !slices.Equal(groups[0].paths, []string{"/a", "/c"}) {
t.Errorf("group paths = %q, want /a /c", groups[0].paths)
}
}
func TestCollectDupeGroupsMtimeExcluded(t *testing.T) {
t.Parallel()
+39 -155
View File
@@ -7,7 +7,6 @@ import (
"database/sql"
"encoding/hex"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
@@ -17,37 +16,8 @@ import (
"syscall"
)
// The duplicate ladder (see hashSignature and README "Duplicate
// detection"). A same-size candidate below headTailMin is hashed in
// full and compared directly; a larger one is separated first by the
// hashes of its end windows, then by a content hash that is exact below
// wholeFileMax and deliberately sampled at or above it.
// headTailMin is the size threshold for the end-window gate. A file
// smaller than this is hashed in full directly, with no separate head
// and tail step: its head, tail, and content all carry the whole-file
// hash. A file this size or larger is separated first by its end
// windows.
const headTailMin = 10 * 1024 * 1024
// headTailWindow is the number of bytes hashed from each end of a file
// at or above headTailMin (the head and tail rungs). Because
// headTailMin is far larger than two windows, the head and tail windows
// never overlap.
const headTailWindow = 64 * 1024
// wholeFileMax is the size boundary between the two content rungs: a
// file strictly smaller than this is content-hashed in full; a file
// this size or larger is content-hashed by sampling.
const wholeFileMax = 50 * 1024 * 1024
// sampleStride is the spacing between content samples for large files:
// one window is read at each gigabyte-aligned offset (0, 1 GiB, ...).
const sampleStride = 1024 * 1024 * 1024
// sampleWindow is the number of bytes read at each large-file sample
// offset, truncated at end of file.
const sampleWindow = 1024 * 1024
// chunk is the number of bytes hashed from each end of a file.
const chunk = 1024
// workQueueDepth bounds the job and result channels feeding the walk
// and hash worker pools.
@@ -469,12 +439,11 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
s.resolve(rec.path)
s.batch = append(s.batch, scanRec{
size: rec.size,
mtime: rec.mtime,
head: r.head,
tail: r.tail,
content: r.content,
path: rec.path,
size: rec.size,
mtime: rec.mtime,
head: r.head,
tail: r.tail,
path: rec.path,
})
}
@@ -865,15 +834,13 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
return statDev(st), st.Ino
}
// hashResult carries one inode run's signature hashes — head, tail, and
// content — (or the error that prevented hashing it) from the hash
// workers to the hash phase.
// hashResult carries one inode run's head/tail hashes (or the error
// that prevented hashing it) from the hash workers to the hash phase.
type hashResult struct {
run []fileRec
head string
tail string
content string
err error
run []fileRec
head string
tail string
err error
}
// hashPool owns every goroutine of the hash worker pool: the feeder
@@ -963,11 +930,11 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
continue
}
head, tail, content, err := hashSignature(run[0].path, run[0].size)
head, tail, err := hashHeadTail(run[0].path, run[0].size)
select {
case results <- hashResult{
run: run, head: head, tail: tail, content: content, err: err,
run: run, head: head, tail: tail, err: err,
}:
case <-ctx.Done():
return
@@ -975,138 +942,55 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
}
}
// emptyHash is the lowercase-hex SHA-256 of the empty input: the head,
// tail, and content hash of every zero-length file.
// emptyHash is the lowercase-hex SHA-256 of the empty input: the head
// and tail hash of every zero-length file.
const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
"27ae41e4649b934ca495991b7852b855"
// hashSignature computes the three content hashes that, with the file
// size, form its duplicate signature. A file below headTailMin is
// hashed in full and its whole-file SHA-256 is returned as head, tail,
// and content alike — that range takes no separate end-window step. For
// a file at or above headTailMin the head and tail are the SHA-256 of
// its first and last headTailWindow bytes, and content is the SHA-256
// of the whole file below wholeFileMax (the exact rung) or of
// gigabyte-spaced samples at or above it (the sampled, deliberately
// probabilistic rung). Two files are duplicates only when all four
// agree; any mismatch means not a duplicate. size is the value recorded
// when the file was statted; a zero-length file has constant hashes and
// is never opened.
func hashSignature(path string, size int64) (string, string, string, error) {
// hashHeadTail returns the lowercase-hex SHA-256 of the first
// min(chunk, size) bytes and of the last min(chunk, size) bytes of the
// file at path. The two reads overlap when size < 2*chunk. size is the
// value recorded when the file was statted; a zero-length file's
// hashes are constant, so it is never even opened.
func hashHeadTail(path string, size int64) (string, string, error) {
if size == 0 {
return emptyHash, emptyHash, emptyHash, nil
return emptyHash, emptyHash, nil
}
//nolint:gosec // hashing operator-supplied paths is the tool's purpose
f, err := os.Open(path)
if err != nil {
return "", "", "", err
return "", "", err
}
defer func() { _ = f.Close() }()
// Below the threshold the whole file is hashed directly, with no
// end-window step: head and tail both carry the whole-file hash.
if size < int64(headTailMin) {
content, err := hashWhole(f, size)
if err != nil {
return "", "", "", err
}
n := min(int64(chunk), size)
return content, content, content, nil
}
buf := make([]byte, n)
head, tail, err := hashEnds(f, size)
if err != nil {
return "", "", "", err
}
content, err := hashContent(f, size)
if err != nil {
return "", "", "", err
}
return head, tail, content, nil
}
// hashEnds returns the SHA-256 of the first and last headTailWindow
// bytes of f. It is called only for files at least headTailMin, which
// is far larger than two windows, so the windows never overlap and both
// reads are always full.
func hashEnds(f *os.File, size int64) (string, string, error) {
buf := make([]byte, headTailWindow)
_, err := f.ReadAt(buf, 0)
_, err = f.ReadAt(buf, 0)
if err != nil {
return "", "", err
}
h := sha256.Sum256(buf)
head := hex.EncodeToString(h[:])
_, err = f.ReadAt(buf, size-int64(headTailWindow))
// When the whole file fits in one chunk the tail window is exactly
// the bytes just read: reuse the head hash instead of issuing a
// second read for every small file.
if size <= int64(chunk) {
hh := hex.EncodeToString(h[:])
return hh, hh, nil
}
_, err = f.ReadAt(buf, size-n)
if err != nil {
return "", "", err
}
t := sha256.Sum256(buf)
return head, hex.EncodeToString(t[:]), nil
}
// hashContent returns the content-rung hash of f: the SHA-256 of the
// whole file when it is smaller than wholeFileMax, or of sampled
// windows when it is that size or larger.
func hashContent(f *os.File, size int64) (string, error) {
if size >= int64(wholeFileMax) {
return hashSamples(f, size)
}
return hashWhole(f, size)
}
// hashWhole returns the SHA-256 of the entire file. A SectionReader is
// used so the read is independent of the offset left by any end-window
// reads. Reading fewer than size bytes means the file shrank between
// the stat and the hash; that is an error rather than a hash of content
// that no longer matches the recorded size.
func hashWhole(f *os.File, size int64) (string, error) {
h := sha256.New()
n, err := io.Copy(h, io.NewSectionReader(f, 0, size))
if err != nil {
return "", err
}
if n != size {
return "", fmt.Errorf("read %d of %d bytes: %w", n, size,
io.ErrUnexpectedEOF)
}
return hex.EncodeToString(h.Sum(nil)), nil
}
// hashSamples feeds sampleWindow bytes at each gigabyte-aligned offset
// (0, sampleStride, 2*sampleStride, ... while inside the file), in
// order, into one hash, each window truncated at end of file. This is
// the probabilistic large-file rung: two files of equal size agreeing
// on every sample are reported as duplicates without every byte being
// read. Because size is part of the signature, files of different sizes
// never reach this comparison, so the sample boundaries always align.
func hashSamples(f *os.File, size int64) (string, error) {
h := sha256.New()
buf := make([]byte, sampleWindow)
for off := int64(0); off < size; off += int64(sampleStride) {
n := min(int64(sampleWindow), size-off)
_, err := f.ReadAt(buf[:n], off)
if err != nil {
return "", err
}
h.Write(buf[:n])
}
return hex.EncodeToString(h.Sum(nil)), nil
return hex.EncodeToString(h[:]), hex.EncodeToString(t[:]), nil
}
+25 -229
View File
@@ -53,23 +53,7 @@ func pattern(tag byte, n int) []byte {
return data
}
// sig returns a file's full signature (head, tail, content), failing the
// test on any error.
func sig(t *testing.T, path string, size int64) (string, string, string) {
t.Helper()
head, tail, content, err := hashSignature(path, size)
if err != nil {
t.Fatalf("hashSignature %s: %v", path, err)
}
return head, tail, content
}
// TestHashSignatureBelowThreshold verifies that a file below headTailMin
// is hashed in full and compared directly: head, tail, and content all
// carry the whole-file SHA-256, with no separate end-window step.
func TestHashSignatureBelowThreshold(t *testing.T) {
func TestHashHeadTail(t *testing.T) {
t.Parallel()
dir := t.TempDir()
@@ -78,10 +62,13 @@ func TestHashSignatureBelowThreshold(t *testing.T) {
name string
data []byte
}{
{"empty", nil},
{"one-byte", []byte("x")},
{"one-window", pattern(1, headTailWindow)},
{"several-windows", pattern(2, 3*headTailWindow)},
{"near-threshold", pattern(3, headTailMin-1)},
{"under-one-chunk", pattern(1, chunk-1)},
{"exactly-one-chunk", pattern(2, chunk)},
{"overlapping-reads", pattern(3, chunk+chunk/2)},
{"exactly-two-chunks", pattern(4, 2*chunk)},
{"beyond-two-chunks", pattern(5, 3*chunk)},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -89,240 +76,49 @@ func TestHashSignatureBelowThreshold(t *testing.T) {
p := writeFile(t, dir, c.name, c.data)
head, tail, content := sig(t, p, int64(len(c.data)))
head, tail, err := hashHeadTail(p, int64(len(c.data)))
if err != nil {
t.Fatalf("hashHeadTail: %v", err)
}
whole := hexSum(c.data)
if head != whole || tail != whole || content != whole {
t.Errorf("head=%s tail=%s content=%s, want all whole-file %s",
head, tail, content, whole)
n := min(chunk, len(c.data))
if want := hexSum(c.data[:n]); head != want {
t.Errorf("head = %s, want %s", head, want)
}
if want := hexSum(c.data[len(c.data)-n:]); tail != want {
t.Errorf("tail = %s, want %s", tail, want)
}
})
}
}
// TestHashSignatureEnds exercises the head and tail rungs, which apply
// only to files at least headTailMin. Sparse files keep the fixtures
// cheap: a difference in the first window changes only head, a
// difference in the last window changes only tail, and a difference
// between the windows changes neither end hash but does change the
// whole-file content rung (the file is below wholeFileMax).
func TestHashSignatureEnds(t *testing.T) {
func TestHashHeadTailErrors(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// Between headTailMin and wholeFileMax: the end-window gate is active
// and the content rung is a whole-file hash.
const size = int64(headTailMin + 2*1024*1024)
base := sparseFile(t, dir, "ends-base", size)
headDiff := sparseFile(t, dir, "ends-head", size)
tailDiff := sparseFile(t, dir, "ends-tail", size)
midDiff := sparseFile(t, dir, "ends-mid", size)
pokeAt(t, headDiff, 0, []byte{1})
pokeAt(t, tailDiff, size-1, []byte{1})
pokeAt(t, midDiff, size/2, []byte{1})
bHead, bTail, bContent := sig(t, base, size)
h, tl, c := sig(t, headDiff, size)
if h == bHead {
t.Error("a byte in the first window did not change head")
}
if tl != bTail {
t.Error("a byte in the first window changed tail")
}
if c == bContent {
t.Error("a byte in the first window did not change content")
}
h, tl, c = sig(t, tailDiff, size)
if tl == bTail {
t.Error("a byte in the last window did not change tail")
}
if h != bHead {
t.Error("a byte in the last window changed head")
}
if c == bContent {
t.Error("a byte in the last window did not change content")
}
h, tl, c = sig(t, midDiff, size)
if h != bHead || tl != bTail {
t.Error("a byte between the windows changed an end hash")
}
if c == bContent {
t.Error("whole-file content rung ignored a byte between the windows")
}
}
func TestHashSignatureErrors(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// A missing file: an error, and every hash left empty.
head, tail, content, err := hashSignature(filepath.Join(dir, "missing"), 1)
_, _, err := hashHeadTail(filepath.Join(dir, "missing"), 1)
if err == nil {
t.Error("no error for a missing file")
}
if head != "" || tail != "" || content != "" {
t.Errorf("missing file returned hashes: %q %q %q", head, tail, content)
}
// A zero-length file has constant hashes and is never opened: even
// a missing path succeeds.
head, tail, content, err = hashSignature(filepath.Join(dir, "missing"), 0)
if err != nil ||
head != emptyHash || tail != emptyHash || content != emptyHash {
t.Errorf("empty: head=%q tail=%q content=%q err=%v, "+
"want constant hashes", head, tail, content, err)
head, tail, err := hashHeadTail(filepath.Join(dir, "missing"), 0)
if err != nil || head != emptyHash || tail != emptyHash {
t.Errorf("empty: head=%q tail=%q err=%v, want constant hashes",
head, tail, err)
}
// A file that shrank between the stat and hash passes: reading at
// the stat-reported size must fail rather than emit wrong hashes.
p := writeFile(t, dir, "shrunk", []byte("tiny"))
head, tail, content, err = hashSignature(p, int64(2*headTailWindow))
_, _, err = hashHeadTail(p, int64(2*chunk))
if err == nil {
t.Error("no error when the stat size exceeds the file size")
}
if head != "" || tail != "" || content != "" {
t.Errorf("shrunk file returned hashes: %q %q %q", head, tail, content)
}
}
// sparseFile creates a file that is logically size bytes long without
// allocating blocks for the hole, so multi-gigabyte cases stay cheap.
func sparseFile(t *testing.T, dir, name string, size int64) string {
t.Helper()
p := filepath.Join(dir, name)
f, err := os.Create(p) //nolint:gosec // test-controlled path
if err != nil {
t.Fatal(err)
}
err = f.Truncate(size)
if err != nil {
t.Fatal(err)
}
err = f.Close()
if err != nil {
t.Fatal(err)
}
return p
}
// pokeAt writes data into an existing file at off, leaving the rest of
// the file (a sparse hole) untouched.
func pokeAt(t *testing.T, path string, off int64, data []byte) {
t.Helper()
f, err := os.OpenFile(path, os.O_WRONLY, 0o600) //nolint:gosec // test path
if err != nil {
t.Fatal(err)
}
_, err = f.WriteAt(data, off)
if err != nil {
t.Fatal(err)
}
err = f.Close()
if err != nil {
t.Fatal(err)
}
}
// contentHash returns just the content rung of a file's signature.
func contentHash(t *testing.T, path string, size int64) string {
t.Helper()
_, _, content := sig(t, path, size)
return content
}
// TestContentRungBoundary checks the 50 MiB boundary between the two
// content rungs: just below it the whole file is hashed and any byte
// difference shows; at the boundary only the gigabyte-spaced samples are
// hashed, so a difference outside a sample window is invisible.
func TestContentRungBoundary(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// A byte that lands outside the single [0, sampleWindow) sample a
// sub-gigabyte file has, but well inside the file.
const off = 10 * 1024 * 1024
// Just under the boundary: the whole-file rung sees the poked byte.
under := int64(wholeFileMax - 1)
underBase := sparseFile(t, dir, "under-base", under)
underPoked := sparseFile(t, dir, "under-poked", under)
pokeAt(t, underPoked, off, []byte{1})
if contentHash(t, underBase, under) == contentHash(t, underPoked, under) {
t.Error("whole-file rung ignored a byte difference below wholeFileMax")
}
// At the boundary: only [0, sampleWindow) is sampled, so the poked
// byte at off is invisible and the two content hashes match.
at := int64(wholeFileMax)
atBase := sparseFile(t, dir, "at-base", at)
atPoked := sparseFile(t, dir, "at-poked", at)
pokeAt(t, atPoked, off, []byte{1})
if contentHash(t, atBase, at) != contentHash(t, atPoked, at) {
t.Error("sampled rung saw a byte outside every sample window")
}
}
// TestContentRungMultiGigabyte exercises the sampled rung across several
// gigabytes using sparse files: a difference inside the third sample
// window (at offset 2*sampleStride) changes the hash, while a difference
// in the gap after it does not.
func TestContentRungMultiGigabyte(t *testing.T) {
t.Parallel()
dir := t.TempDir()
// Three sample windows (offsets 0, 1 GiB, 2 GiB) plus a trailing gap
// that no sample covers.
size := int64(2*sampleStride + 2*sampleWindow)
thirdSample := int64(2 * sampleStride)
gap := thirdSample + int64(sampleWindow)
base := sparseFile(t, dir, "g-base", size)
inSample := sparseFile(t, dir, "g-insample", size)
inGap := sparseFile(t, dir, "g-ingap", size)
pokeAt(t, inSample, thirdSample, []byte{1})
pokeAt(t, inGap, gap, []byte{1})
baseHash := contentHash(t, base, size)
if contentHash(t, inSample, size) == baseHash {
t.Error("sample at 2 GiB was not read: difference there was invisible")
}
if contentHash(t, inGap, size) != baseHash {
t.Error("a byte in an unsampled gap changed the content hash")
}
}
// collectWalk runs a walk over roots and returns the emitted records
+31
View File
@@ -11,6 +11,12 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# yarn provides prettier, which formats Markdown. yarn is a tool, like
# node/git/make/go below; the reference that governs formatting output is
# prettier, pinned by yarn.lock's integrity hash and installed by
# `yarn install --frozen-lockfile`.
YARN_VERSION="1.22.22"
PKGMGR=""
SUDO=""
APT_UPDATED=""
@@ -58,6 +64,21 @@ missing() {
! command -v "$1" >/dev/null 2>&1
}
ensure_node() {
if ! missing node; then return 0; fi
pkg_install nodejs nodejs node nodejs
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable >/dev/null 2>&1 || true
corepack prepare "yarn@$YARN_VERSION" --activate
else
pkg_install yarn yarn yarn yarn
fi
}
main() {
cd "$ROOT"
@@ -71,6 +92,16 @@ main() {
if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi
# node runs prettier and is an unpinned host tool for the same reason
# git/make/go are: it comes from the host package manager, whatever
# version it ships. It is not installed via nvm the way the canonical
# template does, because nvm's prebuilt node is glibc-linked and does
# not run on this repo's musl/Alpine build image. prettier — the tool
# whose version affects formatting output — is pinned by yarn.lock.
ensure_node
ensure_yarn
yarn install --frozen-lockfile
# Linting runs via docker only (script/lint), so docker is a lint
# prerequisite rather than something bootstrap installs. Warn, do
# not fail: everything except `make lint` — and, through it,
+12 -1
View File
@@ -1,12 +1,23 @@
#!/bin/sh
# script/fmt: format all files (writes).
# script/fmt: format all files (writes). gofmt for Go, prettier for
# Markdown. prettier is the pinned devDependency in package.json/
# yarn.lock; script/bootstrap installs it (see run_prettier).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() {
cd "$ROOT"
gofmt -s -w .
run_prettier --write '**/*.md' --tab-width 4 --prose-wrap always
}
main "$@"
+21 -2
View File
@@ -1,18 +1,37 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
# script/fmt: gofmt for Go, prettier for Markdown. Both run every time
# and each reports independently, so a failure names which formatter is
# unhappy; the script exits non-zero if either found unformatted files.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt-check: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() {
cd "$ROOT"
rc=0
files="$(gofmt -s -l .)"
if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2
echo "$files" >&2
exit 1
rc=1
fi
if ! run_prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
rc=1
fi
exit "$rc"
}
main "$@"
+5 -8
View File
@@ -13,10 +13,9 @@ import (
// fileSig is a file's duplicate signature; mtime is excluded.
type fileSig struct {
size int64
head string
tail string
content string
size int64
head string
tail string
}
// treeNode is one directory reconstructed from the scan stream.
@@ -123,9 +122,7 @@ func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
node.files = make(map[string]fileSig)
}
sig := fileSig{
size: r.size, head: r.head, tail: r.tail, content: r.content,
}
sig := fileSig{size: r.size, head: r.head, tail: r.tail}
// An unhashed record (its size was unique when last scanned)
// has unknown content: give it a signature no other file can
@@ -191,7 +188,7 @@ func (n *treeNode) compute() {
for name, sig := range n.files {
entries = append(entries,
"f\x00"+name+"\x00"+strconv.FormatInt(sig.size, 10)+
"\x00"+sig.head+"\x00"+sig.tail+"\x00"+sig.content)
"\x00"+sig.head+"\x00"+sig.tail)
n.fileCount++
n.totalSize += sig.size
}
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==