Compare commits

7 Commits
Author SHA1 Message Date
sneak 3f9f150736 Print the byte unit once in the summary line (closes #17)
check / check (push) Successful in 3m2s
formatBytes already ends in a unit, so the summary line read "9 B
bytes" or "1.0 KiB bytes". The format string in Stats.Print no longer
adds the word "bytes".

Print now takes the io.Writer to print to; its callers pass os.Stderr,
and the new test passes a buffer to read the line for one small and
one large byte count.

Model: opus-5-5
2026-10-06 06:00:11 +00:00
clawbot c4b50e2b55 Skip excluded paths before looking at their errors (closes #16)
check / check (push) Successful in 4m10s
filepath.Walk hands its callback the error for a path it could not
read, such as a directory it cannot list. Both callbacks, in
walkAndProcess and countFiles, returned that error before they checked
the exclusions, so a directory that --exclude or --exclude-dotfiles
excluded still failed the run when it could not be listed. Given an
error, each callback now checks the exclusions first and skips an
excluded path. The new test excludes a directory that cannot be listed
and runs sum add and check over its parent.

Model: opus-5-5
2026-10-06 07:44:00 +02:00
clawbot d010135618 Run lint and tests as phases of the Dockerfile (closes #5)
check / check (push) Successful in 2m45s
The Dockerfile gets a lint phase on the pinned golangci-lint v2.12.2
image and a test phase on the Debian Go image. The tests run as an
unprivileged user, because root reads a file with mode 0000 and the
permission test then fails. The build stage copies a file from each
phase, so no build finishes unless both pass, and it no longer runs
make check. script/lint and script/test each build their phase,
uncached and tagged; script/cibuild bootstraps, runs script/check, then
builds the image. script/bootstrap no longer installs golangci-lint.
README.md and TODO.md describe the new setup.

Model: opus-5-5
2026-10-06 06:59:53 +02:00
clawbot 4fd857bc32 Keep going past unreadable files with check --continue (closes #11)
check / check (push) Successful in 2m1s
With --continue, check stopped at the first file whose content or
checksum attribute it could not read, and at the first directory it
could not list. Each of these now counts as failed, its error, which
names the path, goes to stderr, and the walk goes on; the run still
exits non-zero. The count that sizes the progress bar leaves such a
path out, so the bar stays. The walk and the count, shared with sum
and clear, take the continue setting; those commands pass false and
still stop at the first error. Without --continue the first such error
still stops the run, and the summary printed before it now counts an
unreadable checksum attribute as failed, as it already did for
unreadable content.

Model: opus-5-5
2026-10-06 05:44:04 +02:00
clawbot 30b36dabe4 Make the golangci-lint settings take effect: v2 config layout, linter pinned at v2.12.2 (closes #10)
check / check (push) Successful in 3m32s
golangci-lint v2 ignores a top-level linters-settings block without a
warning, so limits such as the 88-column line length were never
applied. .golangci.yml is now the canonical copy from sneak/prompts,
verbatim, which keeps its settings under linters.settings and also
configures depguard and gomodguard_v2. golangci-lint is pinned at
v2.12.2 by commit in the Dockerfile and script/bootstrap. The long
lines in attrsum.go are rewrapped and one nolint directive moves to
its own line; behaviour is unchanged.

Model: opus-5-5
2026-10-06 03:26:51 +02:00
clawbot 6a0dfa9f2c Run make try against a temporary directory (closes #12)
check / check (push) Successful in 3m36s
`make try` ran against a fixed directory on one person's machine. It
failed for anyone else, and where that directory existed it rewrote the
checksum attributes and modification times of real files.

It now writes three small files into a new directory from `mktemp -d`,
runs the same sequence of commands on it, and removes it with an EXIT
trap. All steps run in one shell, so the trap also removes the
directory when a step fails.

`TODO.md`: the `try` sentence leaves Next Step and is recorded under
Completed Steps.

Model: opus-5-5
2026-10-06 01:59:55 +02:00
clawbot 9f3af05c91 Report the version: docker build . stamps the git tag or short commit (closes #7)
check / check (push) Successful in 59s
attrsum --version now prints the git tag or short commit. make build
stamps it with -X from git describe, and the Dockerfile takes the VERSION
build argument when given, otherwise git describe --tags --always on the
.git in the build context, failing if .git is present and no version
comes out. A new .dockerignore, the canonical one, keeps .git/config out
of the context, and also this repo's host-built /attrsum. CI checks out
full history so it sees the 1.0.0 tag and stamps what a full clone does.
script/docker is replaced with the canonical copy.

Model: opus-5-5
2026-10-02 11:02:38 +02:00
15 changed files with 582 additions and 157 deletions
+63
View File
@@ -0,0 +1,63 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
.git/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][dD]25519
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repo's host-built binary (`make build`); the image builds its own.
/attrsum
+4
View File
@@ -6,4 +6,8 @@ jobs:
steps: steps:
# actions/checkout v4.2.2, 2026-02-28 # actions/checkout v4.2.2, 2026-02-28
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# Full history and tags, so the build stamps the same
# `git describe` version as a full clone.
with:
fetch-depth: 0
- run: script/cibuild - run: script/cibuild
+80 -14
View File
@@ -1,32 +1,98 @@
version: "2" version: "2"
# Config schema uses the golangci-lint v2 layout (settings live under
# linters.settings, not top-level linters-settings) so that the
# thresholds below are actually applied by golangci-lint >= v2.
run: run:
timeout: 5m timeout: 5m
modules-download-mode: readonly modules-download-mode: readonly
linters: linters:
default: all default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
linters-settings: # silenced by disabling that name, not by enabling the successor.
lll: - wsl # Deprecated, replaced by wsl_v5
line-length: 88 - gomodguard # Deprecated, replaced by gomodguard_v2
funlen: settings:
lines: 80 lll:
statements: 50 line-length: 88
cyclop: funlen:
max-complexity: 15 lines: 80
dupl: statements: 50
threshold: 100 cyclop:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues: issues:
exclude-use-default: false
max-issues-per-linter: 0 max-issues-per-linter: 0
max-same-issues: 0 max-same-issues: 0
+48 -26
View File
@@ -1,35 +1,57 @@
# Build stage # Lint phase
# golang 1.25-alpine, 2026-02-28 # golangci/golangci-lint:v2.12.2, 2026-10-05
FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
RUN apk add --no-cache git make gcc musl-dev binutils-gold
# golangci-lint v2.10.1
RUN go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@5d1e709b7be35cb2025444e19de266b056b7b7ee
# goimports v0.42.0
RUN go install golang.org/x/tools/cmd/goimports@009367f5c17a8d4c45a961a3a509277190a9a6f0
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not. The tests run as an
# unprivileged user: root can read a file with mode 0000, so the
# permission test would fail.
# golang:1.25.7-trixie, 2026-10-06
FROM golang@sha256:2b174ffcf56c7ad0c47d30d2630693265639ddf2a5141149c2da34db921791b4 AS test
RUN useradd --create-home testuser
USER testuser
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang 1.25-alpine, 2026-02-28
FROM golang@sha256:f6751d823c26342f9506c03797d2527668d095b0a15f1862cddb4d927a7a4ced AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git make
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# Run the checks as an unprivileged user. Root bypasses file mode bits, which # The version stamped into the binary: the VERSION build argument when one
# would make the permission tests (expecting EACCES on a 0000 file) spuriously # is given, otherwise `git describe --tags --always` of the .git the build
# pass with no error. Caches live under /tmp (world-writable) so the user needs # context carries: the tag on a tagged commit, tag-N-gHASH on a commit after
# no home directory of its own. # one, the short commit when no tag is reachable. A context that carries .git
ENV GOCACHE=/tmp/gocache # and still yields no version fails the build. With neither, as from a source
ENV XDG_CACHE_HOME=/tmp/xdgcache # tarball, the binary reports dev.
RUN adduser -D -u 1000 builder && chown -R builder:builder /src /go ARG VERSION
USER builder RUN version="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
# Run all checks - build fails if any check fails [ "$version" = unknown ]; }; then \
RUN make check echo "version is '$version' although .git is present" >&2; \
exit 1; \
# Build the binary (still as the unprivileged user: it owns /src, so git VCS fi; \
# stamping sees consistent ownership). make build VERSION="${version:-dev}"
RUN make build
# Runtime stage # Runtime stage
# alpine 3.21, 2026-02-28 # alpine 3.21, 2026-02-28
+23 -11
View File
@@ -1,6 +1,9 @@
.PHONY: default bootstrap setup test lint fmt fmt-check check docker hooks build clean try .PHONY: default bootstrap setup test lint fmt fmt-check check docker hooks build clean try
TESTDIR := $(HOME)/Documents/_SYSADMIN/cyberdyne # The version `make build` stamps into the binary: the git tag or short
# commit, -dirty with uncommitted changes. `make build VERSION=x` stamps x,
# which is how the Dockerfile passes its version in.
VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
# Standard targets are thin shims; the implementations live in script/ # Standard targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern. # per the scripts-to-rule-them-all pattern.
@@ -35,18 +38,27 @@ hooks:
@script/install-precommit @script/install-precommit
build: clean build: clean
@go build . @go build -ldflags "-X main.Version=$(VERSION)" .
clean: clean:
@rm -f attrsum @rm -f attrsum
# Runs the binary on three small files in a new temporary directory, which
# the trap removes when the recipe ends, also when a step fails. The check
# after clear is expected to fail, so its exit status is ignored.
try: build try: build
./attrsum sum add -v $(TESTDIR) @set -ex; \
./attrsum check -v $(TESTDIR) dir=$$(mktemp -d); \
./attrsum clear -v $(TESTDIR) trap 'rm -rf "$$dir"' EXIT; \
-./attrsum check -v $(TESTDIR) echo one >"$$dir/a"; \
./attrsum sum add -v $(TESTDIR) echo two >"$$dir/b"; \
./attrsum check -v $(TESTDIR) echo three >"$$dir/c"; \
touch $(TESTDIR)/* ./attrsum sum add -v "$$dir"; \
./attrsum sum update -v $(TESTDIR) ./attrsum check -v "$$dir"; \
./attrsum check -v $(TESTDIR) ./attrsum clear -v "$$dir"; \
./attrsum check -v "$$dir" || true; \
./attrsum sum add -v "$$dir"; \
./attrsum check -v "$$dir"; \
touch "$$dir"/*; \
./attrsum sum update -v "$$dir"; \
./attrsum check -v "$$dir"
+3 -1
View File
@@ -95,7 +95,9 @@ Future improvements under consideration:
* Author & maintainer: **sneak** – <sneak@sneak.berlin> * Author & maintainer: **sneak** – <sneak@sneak.berlin>
* Issues / PRs: <https://git.eeqj.de/sneak/attrsum/> * Issues / PRs: <https://git.eeqj.de/sneak/attrsum/>
* Code must pass `go vet`, `go test ./...`, and `go fmt`. * Code must pass `make check`, which runs the tests and golangci-lint as
phases of the `Dockerfile` (Docker is required) and checks formatting
with `gofmt`.
* No CLA; contributions are under WTFPL v2. * No CLA; contributions are under WTFPL v2.
--- ---
+36 -12
View File
@@ -17,15 +17,40 @@ have landed since the tag.
# Next Step # Next Step
Policy scaffold commit: add LICENSE, REPO_POLICIES.md, .editorconfig, Re-vendor the canonical files from `sneak/prompts` at `dd4027b`
.golangci.yml, and a comprehensive .gitignore (currently only the (https://git.eeqj.de/sneak/attrsum/issues/13): add `REPO_POLICIES.md`
attrsum binary), and extend the Makefile (only test/build/clean/try and `.editorconfig`, refresh `.gitignore` (only the `attrsum` binary
today) with lint, fmt, fmt-check, check, and hooks targets. Fix the try today), `.dockerignore`, `.gitea/workflows/check.yml` and
target to use a temp fixture instead of the hardcoded `.golangci.yml`, and move the lint phase to golangci-lint v2.14.0.
$(HOME)/Documents/_SYSADMIN/cyberdyne path.
# Completed Steps # Completed Steps
* 2026-10-06: the summary line after `sum`, `check` and `clear` prints
the byte unit once (`1.5 KiB`, not `1.5 KiB bytes`)
* 2026-10-06: a path that `--exclude` or `--exclude-dotfiles` excludes
is skipped even when it cannot be read, so an excluded directory
that cannot be listed no longer fails the run
* 2026-10-06: lint and test run as phases of the `Dockerfile`, and the
build stage depends on both; `script/lint` and `script/test` each
build their phase with `--no-cache`; `script/cibuild` bootstraps,
runs `script/check`, then builds the image; golangci-lint is no
longer installed on the host
* 2026-10-06: `check --continue` keeps going past a file or directory
it cannot read: it counts it as failed, prints the error and the
path on stderr, and checks the rest of the tree
* 2026-10-05: golangci-lint settings take effect: canonical
`.golangci.yml` (v2 layout, settings under `linters.settings`),
golangci-lint pinned at v2.12.2 in `Dockerfile` and
`script/bootstrap`, and the code fixed for what the settings now
report (long lines in `attrsum.go` rewrapped)
* 2026-10-05: `make try` runs on three small files in a temporary
directory that it removes afterwards, also when a step fails,
instead of on a fixed directory on one person's machine
* 2026-10-02: `attrsum --version` reports the git tag or short commit,
stamped by `make build` and by a plain `docker build .` of a clone;
`.dockerignore` sends `.git` without `.git/config` and keeps a
host-built `attrsum` out; CI checks out full history so it stamps
the same version
* 2026-02-02: correctness pass: track actual bytes read instead of * 2026-02-02: correctness pass: track actual bytes read instead of
stale file size, atomic failure tracking in ProcessCheck, detect stale file size, atomic failure tracking in ProcessCheck, detect
file modification during checksum (TOCTOU), propagate countFiles file modification during checksum (TOCTOU), propagate countFiles
@@ -39,13 +64,12 @@ $(HOME)/Documents/_SYSADMIN/cyberdyne path.
# Future Steps # Future Steps
* Add Dockerfile and .dockerignore that run make check, images pinned
by sha256, plus a Makefile docker target
* Add .gitea/workflows/check.yml
* Restructure README.md into the standard sections: Description, * Restructure README.md into the standard sections: Description,
Getting Started, Rationale, Design, TODO, License, Author (Getting Getting Started, Entrypoints, Rationale, Design, TODO, License,
Started, Why?, TODO, License exist; Description, Design, Author are Author (Getting Started, Why?, TODO, License exist; Description,
missing) Entrypoints, Design, Author are missing)
* Add a `LICENSE` file matching the README's WTFPL v2; sneak's to add,
not an agent's
* Tag a patch release to ship the 2026-02-02 correctness fixes * Tag a patch release to ship the 2026-02-02 correctness fixes
* Dry-run mode (--dry-run, -n): show what would be done without making * Dry-run mode (--dry-run, -n): show what would be done without making
changes (from README TODO) changes (from README TODO)
+133 -74
View File
@@ -36,6 +36,10 @@ const (
progressThrottle = 250 * time.Millisecond progressThrottle = 250 * time.Millisecond
) )
// Version is the git tag or short commit, set at link time with -X by
// `make build`. A build that does not set it reports dev.
var Version = "dev" //nolint:gochecknoglobals // set at link time with -X
// Sentinel errors returned by the command implementations. // Sentinel errors returned by the command implementations.
var ( var (
errNoPaths = errors.New("no paths provided") errNoPaths = errors.New("no paths provided")
@@ -67,12 +71,13 @@ func (s *Stats) Duration() time.Duration {
return time.Since(s.StartTime) return time.Since(s.StartTime)
} }
func (s *Stats) Print(opts *options, operation string) { func (s *Stats) Print(w io.Writer, opts *options, operation string) {
if opts.quiet { if opts.quiet {
return return
} }
fmt.Fprintf(os.Stderr, "\n%s complete: %d files processed, %d skipped, %d failed, %s bytes in %s\n", _, _ = fmt.Fprintf(w,
"\n%s complete: %d files processed, %d skipped, %d failed, %s in %s\n",
operation, operation,
s.FilesProcessed, s.FilesProcessed,
s.FilesSkipped, s.FilesSkipped,
@@ -101,8 +106,9 @@ func main() {
opts := &options{} opts := &options{}
rootCmd := &cobra.Command{ rootCmd := &cobra.Command{
Use: "attrsum", Use: "attrsum",
Short: "Compute and verify file checksums via xattrs", Short: "Compute and verify file checksums via xattrs",
Version: Version,
} }
rootCmd.SilenceUsage = true rootCmd.SilenceUsage = true
rootCmd.SilenceErrors = true rootCmd.SilenceErrors = true
@@ -166,12 +172,16 @@ func expandPaths(args []string) ([]string, error) {
} }
// processFunc processes a single path within a command's run loop. // processFunc processes a single path within a command's run loop.
type processFunc func(opts *options, path string, stats *Stats, bar *progressbar.ProgressBar) error type processFunc func(
opts *options, path string, stats *Stats, bar *progressbar.ProgressBar,
) error
// countAndBar counts the files under paths and returns a progress bar sized // countAndBar counts the files under paths and returns a progress bar sized
// to that total. It always returns either a non-nil bar or a non-nil error. // to that total. It always returns either a non-nil bar or a non-nil error.
func countAndBar(opts *options, paths []string, desc string) (*progressbar.ProgressBar, error) { func countAndBar(
total, err := countFilesMultiple(opts, paths) opts *options, paths []string, desc string, cont bool,
) (*progressbar.ProgressBar, error) {
total, err := countFilesMultiple(opts, paths, cont)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -188,7 +198,9 @@ func finishBar(bar *progressbar.ProgressBar) {
// runOverPaths runs process over each path, sharing the progress/stats // runOverPaths runs process over each path, sharing the progress/stats
// bookkeeping common to the sum-add, sum-update and clear commands. // bookkeeping common to the sum-add, sum-update and clear commands.
func runOverPaths(opts *options, args []string, desc, op string, process processFunc) error { func runOverPaths(
opts *options, args []string, desc, op string, process processFunc,
) error {
paths, err := expandPaths(args) paths, err := expandPaths(args)
if err != nil { if err != nil {
return err return err
@@ -199,7 +211,7 @@ func runOverPaths(opts *options, args []string, desc, op string, process process
var bar *progressbar.ProgressBar var bar *progressbar.ProgressBar
if !opts.quiet { if !opts.quiet {
bar, err = countAndBar(opts, paths, desc) bar, err = countAndBar(opts, paths, desc, false)
if err != nil { if err != nil {
return err return err
} }
@@ -215,7 +227,7 @@ func runOverPaths(opts *options, args []string, desc, op string, process process
} }
finishBar(bar) finishBar(bar)
stats.Print(opts, op) stats.Print(os.Stderr, opts, op)
return nil return nil
} }
@@ -253,46 +265,54 @@ func newSumCmd(opts *options) *cobra.Command {
return cmd return cmd
} }
func processSumAdd(opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar) error { func processSumAdd(
return walkAndProcess(opts, dir, stats, bar, func(p string, info os.FileInfo, s *Stats) error { opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar,
if hasXattr(p, checksumKey) { ) error {
atomic.AddInt64(&s.FilesSkipped, 1) return walkAndProcess(opts, dir, false, stats, bar,
func(p string, info os.FileInfo, s *Stats) error {
if hasXattr(p, checksumKey) {
atomic.AddInt64(&s.FilesSkipped, 1)
return nil return nil
} }
err := writeChecksumAndTime(opts, p, info, s) err := writeChecksumAndTime(opts, p, info, s)
if err != nil { if err != nil {
atomic.AddInt64(&s.FilesFailed, 1)
return err
}
return nil
})
}
func processSumUpdate(opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar) error {
return walkAndProcess(opts, dir, stats, bar, func(p string, info os.FileInfo, s *Stats) error {
t, err := readSumTime(p)
if err != nil || info.ModTime().After(t) {
werr := writeChecksumAndTime(opts, p, info, s)
if werr != nil {
atomic.AddInt64(&s.FilesFailed, 1) atomic.AddInt64(&s.FilesFailed, 1)
return werr return err
} }
return nil return nil
} })
atomic.AddInt64(&s.FilesSkipped, 1)
return nil
})
} }
func writeChecksumAndTime(opts *options, path string, info os.FileInfo, stats *Stats) error { func processSumUpdate(
opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar,
) error {
return walkAndProcess(opts, dir, false, stats, bar,
func(p string, info os.FileInfo, s *Stats) error {
t, err := readSumTime(p)
if err != nil || info.ModTime().After(t) {
werr := writeChecksumAndTime(opts, p, info, s)
if werr != nil {
atomic.AddInt64(&s.FilesFailed, 1)
return werr
}
return nil
}
atomic.AddInt64(&s.FilesSkipped, 1)
return nil
})
}
func writeChecksumAndTime(
opts *options, path string, info os.FileInfo, stats *Stats,
) error {
// Record mtime before hashing to detect modifications during hash. // Record mtime before hashing to detect modifications during hash.
mtimeBefore := info.ModTime() mtimeBefore := info.ModTime()
@@ -363,24 +383,27 @@ func newClearCmd(opts *options) *cobra.Command {
} }
} }
func processClear(opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar) error { func processClear(
return walkAndProcess(opts, dir, stats, bar, func(p string, info os.FileInfo, s *Stats) error { opts *options, dir string, stats *Stats, bar *progressbar.ProgressBar,
cleared, err := clearOne(opts, p) ) error {
if err != nil { return walkAndProcess(opts, dir, false, stats, bar,
atomic.AddInt64(&s.FilesFailed, 1) func(p string, info os.FileInfo, s *Stats) error {
cleared, err := clearOne(opts, p)
if err != nil {
atomic.AddInt64(&s.FilesFailed, 1)
return err return err
} }
if cleared { if cleared {
atomic.AddInt64(&s.FilesProcessed, 1) atomic.AddInt64(&s.FilesProcessed, 1)
atomic.AddInt64(&s.BytesProcessed, info.Size()) atomic.AddInt64(&s.BytesProcessed, info.Size())
} else { } else {
atomic.AddInt64(&s.FilesSkipped, 1) atomic.AddInt64(&s.FilesSkipped, 1)
} }
return nil return nil
}) })
} }
// clearOne removes both checksum xattrs from a single path, reporting // clearOne removes both checksum xattrs from a single path, reporting
@@ -428,7 +451,8 @@ func newCheckCmd(opts *options) *cobra.Command {
return runCheck(opts, a, cont) return runCheck(opts, a, cont)
}, },
} }
cmd.Flags().BoolVar(&cont, "continue", false, "continue after errors and report each file") cmd.Flags().BoolVar(&cont, "continue", false,
"continue after errors and report each file")
return cmd return cmd
} }
@@ -444,7 +468,7 @@ func runCheck(opts *options, args []string, cont bool) error {
var bar *progressbar.ProgressBar var bar *progressbar.ProgressBar
if !opts.quiet { if !opts.quiet {
bar, err = countAndBar(opts, paths, "Verifying checksums") bar, err = countAndBar(opts, paths, "Verifying checksums", cont)
if err != nil { if err != nil {
return err return err
} }
@@ -457,7 +481,7 @@ func runCheck(opts *options, args []string, cont bool) error {
if perr != nil { if perr != nil {
if !cont { if !cont {
finishBar(bar) finishBar(bar)
stats.Print(opts, "check") stats.Print(os.Stderr, opts, "check")
return perr return perr
} }
@@ -467,18 +491,21 @@ func runCheck(opts *options, args []string, cont bool) error {
} }
finishBar(bar) finishBar(bar)
stats.Print(opts, "check") stats.Print(os.Stderr, opts, "check")
return finalErr return finalErr
} }
func processCheck(opts *options, dir string, cont bool, stats *Stats, bar *progressbar.ProgressBar) error { func processCheck(
opts *options, dir string, cont bool, stats *Stats, bar *progressbar.ProgressBar,
) error {
// Track initial failed count to detect failures during this walk. // Track initial failed count to detect failures during this walk.
initialFailed := atomic.LoadInt64(&stats.FilesFailed) initialFailed := atomic.LoadInt64(&stats.FilesFailed)
err := walkAndProcess(opts, dir, stats, bar, func(p string, _ os.FileInfo, s *Stats) error { err := walkAndProcess(opts, dir, cont, stats, bar,
return checkOne(opts, p, cont, s) func(p string, _ os.FileInfo, s *Stats) error {
}) return checkOne(opts, p, cont, s)
})
if err != nil { if err != nil {
if errors.Is(err, errVerification) { if errors.Is(err, errVerification) {
return errVerification return errVerification
@@ -500,7 +527,7 @@ func checkOne(opts *options, p string, cont bool, s *Stats) error {
exp, err := xattr.Get(p, checksumKey) exp, err := xattr.Get(p, checksumKey)
if err != nil { if err != nil {
if !errors.Is(err, xattr.ENOATTR) { if !errors.Is(err, xattr.ENOATTR) {
return err return unreadable(cont, s, err)
} }
return missingChecksum(opts, p, cont, s) return missingChecksum(opts, p, cont, s)
@@ -508,9 +535,7 @@ func checkOne(opts *options, p string, cont bool, s *Stats) error {
act, bytesRead, err := fileMultihash(p) act, bytesRead, err := fileMultihash(p)
if err != nil { if err != nil {
atomic.AddInt64(&s.FilesFailed, 1) return unreadable(cont, s, err)
return err
} }
ok := bytes.Equal(exp, act) ok := bytes.Equal(exp, act)
@@ -546,6 +571,21 @@ func missingChecksum(opts *options, p string, cont bool, s *Stats) error {
return errVerification return errVerification
} }
// unreadable counts a file or directory that could not be read as failed.
// With --continue it prints err, which names the path, to stderr and
// returns nil so the walk goes on; otherwise it returns err.
func unreadable(cont bool, s *Stats, err error) error {
atomic.AddInt64(&s.FilesFailed, 1)
if !cont {
return err
}
log.Print(err)
return nil
}
// reportCheck prints a per-file verification result when verbose output is on. // reportCheck prints a per-file verification result when verbose output is on.
func reportCheck(opts *options, p, actual string, ok bool) { func reportCheck(opts *options, p, actual string, ok bool) {
if !opts.verbose || opts.quiet { if !opts.verbose || opts.quiet {
@@ -565,12 +605,24 @@ func reportCheck(opts *options, p, actual string, ok bool) {
/////////////////////////////////////////////////////////////////////////////// ///////////////////////////////////////////////////////////////////////////////
// countFiles counts the total number of regular files that will be processed. // countFiles counts the total number of regular files that will be processed.
func countFiles(opts *options, root string) (int64, error) { // With cont, a path it cannot read is left out of the count instead of ending
// it; the walk that follows reports that path as failed.
func countFiles(opts *options, root string, cont bool) (int64, error) {
var count int64 var count int64
root = filepath.Clean(root) root = filepath.Clean(root)
err := filepath.Walk(root, func(p string, info os.FileInfo, err error) error { err := filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
// An excluded path is skipped whatever went wrong reading it.
rel, _ := filepath.Rel(root, p)
if err != nil && shouldExclude(opts, rel) {
return nil
}
if err != nil && cont {
return nil
}
if err != nil { if err != nil {
return err return err
} }
@@ -581,7 +633,6 @@ func countFiles(opts *options, root string) (int64, error) {
return nil return nil
} }
rel, _ := filepath.Rel(root, p)
if shouldExclude(opts, rel) { if shouldExclude(opts, rel) {
if info.IsDir() { if info.IsDir() {
return filepath.SkipDir return filepath.SkipDir
@@ -605,11 +656,11 @@ func countFiles(opts *options, root string) (int64, error) {
} }
// countFilesMultiple counts files across multiple roots. // countFilesMultiple counts files across multiple roots.
func countFilesMultiple(opts *options, roots []string) (int64, error) { func countFilesMultiple(opts *options, roots []string, cont bool) (int64, error) {
var total int64 var total int64
for _, root := range roots { for _, root := range roots {
count, err := countFiles(opts, root) count, err := countFiles(opts, root, cont)
if err != nil { if err != nil {
return total, err return total, err
} }
@@ -645,6 +696,7 @@ func newProgressBar(total int64, description string) *progressbar.ProgressBar {
func walkAndProcess( func walkAndProcess(
opts *options, opts *options,
root string, root string,
cont bool,
stats *Stats, stats *Stats,
bar *progressbar.ProgressBar, bar *progressbar.ProgressBar,
fn func(string, os.FileInfo, *Stats) error, fn func(string, os.FileInfo, *Stats) error,
@@ -652,8 +704,14 @@ func walkAndProcess(
root = filepath.Clean(root) root = filepath.Clean(root)
return filepath.Walk(root, func(p string, info os.FileInfo, err error) error { return filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
// An excluded path is skipped whatever went wrong reading it.
rel, _ := filepath.Rel(root, p)
if err != nil && shouldExclude(opts, rel) {
return nil
}
if err != nil { if err != nil {
return err return unreadable(cont, stats, err)
} }
skip, skipErr := walkSkip(opts, root, p, info) skip, skipErr := walkSkip(opts, root, p, info)
@@ -742,7 +800,8 @@ func hasXattr(path, key string) bool {
func fileMultihash(path string) ([]byte, int64, error) { func fileMultihash(path string) ([]byte, int64, error) {
// The path is supplied by the operator as the tree to checksum; reading // The path is supplied by the operator as the tree to checksum; reading
// it is the entire purpose of the tool. // it is the entire purpose of the tool.
f, err := os.Open(path) //nolint:gosec // G304: operator-specified path is the intended input //nolint:gosec // G304: operator-specified path is the intended input
f, err := os.Open(path)
if err != nil { if err != nil {
return nil, 0, err return nil, 0, err
} }
+130
View File
@@ -1,6 +1,8 @@
package main package main
import ( import (
"bytes"
"errors"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -213,6 +215,54 @@ func TestExcludeDotfilesAndPatterns(t *testing.T) {
} }
} }
func TestExcludeUnreadableDir(t *testing.T) {
t.Parallel()
opts := &options{excludePatterns: []string{"locked"}}
dir := t.TempDir()
skipIfNoXattr(t, dir)
keep := writeFile(t, dir, "keep.txt", "keep")
hidden := writeFile(t, dir, "locked/a.txt", "hidden")
// The excluded directory cannot be listed.
sub := filepath.Join(dir, "locked")
err := os.Chmod(sub, noPerm)
if err != nil {
t.Fatalf("chmod dir: %v", err)
}
defer func() { _ = os.Chmod(sub, dirPerm) }()
err = processSumAdd(opts, dir, newTestStats(), nil)
if err != nil {
t.Fatalf("add: %v", err)
}
_, err = xattr.Get(keep, checksumKey)
if err != nil {
t.Fatalf("expected xattr on keep.txt: %v", err)
}
// Without --quiet, runCheck counts the files for the progress bar
// before it checks any, so this also covers the count.
err = runCheck(opts, []string{dir}, false)
if err != nil {
t.Fatalf("check: %v", err)
}
err = os.Chmod(sub, dirPerm)
if err != nil {
t.Fatalf("chmod dir back: %v", err)
}
_, err = xattr.Get(hidden, checksumKey)
if err == nil {
t.Fatalf("locked/a.txt should have been excluded")
}
}
func TestSkipBrokenSymlink(t *testing.T) { func TestSkipBrokenSymlink(t *testing.T) {
t.Parallel() t.Parallel()
@@ -271,3 +321,83 @@ func TestPermissionErrors(t *testing.T) {
t.Fatalf("expected permission error on check, got nil") t.Fatalf("expected permission error on check, got nil")
} }
} }
func TestCheckContinuePastUnreadable(t *testing.T) {
t.Parallel()
opts := &options{}
dir := t.TempDir()
skipIfNoXattr(t, dir)
writeFile(t, dir, "a.txt", "one")
secret := writeFile(t, dir, "b.txt", "two")
writeFile(t, dir, "c/d.txt", "three")
writeFile(t, dir, "e.txt", "four")
err := processSumAdd(opts, dir, newTestStats(), nil)
if err != nil {
t.Fatalf("add: %v", err)
}
// An unreadable file and an unlistable directory sit between the
// readable files a.txt and e.txt.
sub := filepath.Join(dir, "c")
err = os.Chmod(secret, noPerm)
if err != nil {
t.Fatalf("chmod file: %v", err)
}
defer func() { _ = os.Chmod(secret, filePerm) }()
err = os.Chmod(sub, noPerm)
if err != nil {
t.Fatalf("chmod dir: %v", err)
}
defer func() { _ = os.Chmod(sub, dirPerm) }()
stats := newTestStats()
err = processCheck(opts, dir, true, stats, nil)
if !errors.Is(err, errVerification) {
t.Fatalf("expected verification error, got %v", err)
}
if stats.FilesProcessed != 2 || stats.FilesFailed != 2 {
t.Fatalf("expected 2 verified and 2 failed, got %d and %d",
stats.FilesProcessed, stats.FilesFailed)
}
// Without --quiet, runCheck counts the files for the progress bar
// before it checks any, so the count reaches the unlistable directory
// first.
err = runCheck(opts, []string{dir}, true)
if !errors.Is(err, errVerification) {
t.Fatalf("expected verification error from runCheck, got %v", err)
}
}
func TestSummaryPrintsByteUnitOnce(t *testing.T) {
t.Parallel()
tests := []struct {
bytesProcessed int64
want string
}{
{9, ", 9 B in "},
{1536, ", 1.5 KiB in "},
}
for _, tt := range tests {
var out bytes.Buffer
stats := newTestStats()
stats.BytesProcessed = tt.bytesProcessed
stats.Print(&out, &options{}, "check")
if !strings.Contains(out.String(), tt.want) {
t.Errorf("summary %q does not contain %q", out.String(), tt.want)
}
}
}
+5 -7
View File
@@ -3,15 +3,14 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. # or apk (detected in that order); assumes nothing is present.
# golangci-lint and goimports are installed via `go install` at the same # goimports is installed via `go install` at a pinned commit (never
# pinned commits the Dockerfile uses (never "latest"). # "latest"). The linter is not installed: it runs only as the lint phase
# of the Dockerfile.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-07 (same pins as the Dockerfile) # Pinned versions, 2026-10-05
# golangci-lint v2.10.1
GOLANGCI_LINT_REF="github.com/golangci/golangci-lint/v2/cmd/golangci-lint@5d1e709b7be35cb2025444e19de266b056b7b7ee"
# goimports v0.42.0 # goimports v0.42.0
GOIMPORTS_REF="golang.org/x/tools/cmd/goimports@009367f5c17a8d4c45a961a3a509277190a9a6f0" GOIMPORTS_REF="golang.org/x/tools/cmd/goimports@009367f5c17a8d4c45a961a3a509277190a9a6f0"
@@ -69,9 +68,8 @@ main() {
if missing make; then pkg_install gnumake make make make; fi if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# Lint/format tools, pinned via go install (installs into # Format tool, pinned via go install (installs into
# "$(go env GOPATH)/bin"; ensure that is on your PATH). # "$(go env GOPATH)/bin"; ensure that is on your PATH).
if missing golangci-lint; then go install "$GOLANGCI_LINT_REF"; fi
if missing goimports; then go install "$GOIMPORTS_REF"; fi if missing goimports; then go install "$GOIMPORTS_REF"; fi
go mod download go mod download
+3 -1
View File
@@ -1,6 +1,8 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files. # extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+19 -4
View File
@@ -1,13 +1,28 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs make check, so # script/cibuild: run the CI build. It bootstraps first: a CI runner
# a successful build implies all checks pass. # checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build . "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+11 -1
View File
@@ -1,6 +1,8 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -8,7 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" . # Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+14 -3
View File
@@ -1,12 +1,23 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter. # script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
golangci-lint run --config .golangci.yml ./... docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
} }
main "$@" main "$@"
+10 -3
View File
@@ -1,12 +1,19 @@
#!/bin/sh #!/bin/sh
# script/test: run the test suite. # script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
go test -v -race -timeout 30s -cover ./... docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
} }
main "$@" main "$@"