Author SHA1 Message Date
sneak 114ba0a29f Disable the deprecated gomodguard linter (closes #116)
check / check (push) Failing after 2s
golangci-lint v2.12 deprecates gomodguard in favour of gomodguard_v2.
With `default: all` both ran, and every lint run printed the
deprecation warning. Disable the old name, the same way wsl is
disabled for wsl_v5; gomodguard_v2 stays enabled. The old linter had
no settings here, so there is nothing to carry over.

Model: opus-5-5
2026-10-04 08:06:44 +00:00
clawbot 64eb5cbd40 Scanner status tests no longer depend on a 100 ms timer (closes #124)
check / check (push) Failing after 3s
The two status-send tests now call the send directly on a channel nobody
receives from, so a blocking send hangs the test into its timeout instead
of racing a 100 ms timer that a loaded host can miss.

The gpg test for a child holding gpg's output had the same problem: its
100 ms deadline could fire before the fake gpg wrote the PID of sleep,
and the cleanup then failed. The fake gpg now writes that PID to a named
pipe, and the test cancels only after reading it. It then waits up to
10 s for the call, so a call that waits for sleep fails the test and the
cleanup still kills sleep.

Model: opus-5-5
2026-10-04 09:48:51 +02:00
clawbot 45eac1f6f8 Run all linting in Docker via the Dockerfile lint stage (closes #90)
check / check (push) Failing after 3s
script/lint now builds only the lint stage of the main Dockerfile
(docker build --no-cache --target lint), whose build runs the linter, so
a successful build is a clean lint. It is uncached because a cached
build runs no linter, and a trap removes the image it tagged; the tag
carries the process ID so concurrent runs do not collide. The lint stage
calls golangci-lint directly, since make lint now needs Docker. Nothing
installs or runs golangci-lint on the host any more: bootstrap and the
Makefile drop the install, and script/fmt drops golangci-lint run --fix.

Model: opus-5-5
2026-10-04 09:31:54 +02:00
clawbot b91e92b070 Build a static binary so the scratch image runs (closes #126)
check / check (push) Failing after 1s
The final stage is scratch, which has no C library, but the builder
compiled mfer with cgo on (the golang image's default). The binary
imports net (mfer's own HTTP code does, and so does google/uuid), so
with cgo on it came out dynamically linked and the image could not
start. The image's go build now sets CGO_ENABLED=0; nothing in mfer
needs cgo. A new builder step runs ldd on the binary and fails the
build unless it reports a static executable.

Model: opus-5-5
2026-10-04 08:48:52 +02:00
clawbot a2732cf8da Add the required README sections (closes #75)
check / check (push) Successful in 1m28s
The Description first line now names the project, purpose, category,
WTFPL license, and author. A new Getting Started section gives a
copy-pasteable build-from-source block and gen/check/fetch usage. Problem
Statement and Proposed Solution move under a new Rationale heading, the
prose kept. A new Design section documents the package layout: the mfer/
library with its committed protobuf code, the internal/cli commands,
internal/log and internal/bork, and the cmd/mfer entrypoint. Authors is
renamed Author with the canonical link.

Getting Started uses go build because the make target that builds the
binary runs protoc, which script/bootstrap does not install.

Model: opus-4-8 (implementation); opus-5-5 (rebase, rework)
2026-10-04 06:32:07 +02:00
clawbot a789eb3083 Give -v to verbose, move --version to -V (closes #64)
check / check (push) Successful in 1m45s
urfave/cli's built-in version flag claimed -v, so "mfer -v --version"
failed to parse, and -v meant version at the root but verbose on
generate, check, freshen and fetch. Verbose is the more common meaning,
so the root now takes -v and -q too, and the version flag takes -V. A
-v or -q before generate, check, freshen, fetch or export applies to
that subcommand; list keeps its fixed quiet logging.

User-visible changes: -v no longer prints the version; use -V or
--version. "mfer version" prints "mfer version 0.1.0 (...)", the same
line as --version, instead of the bare "0.1.0 (...)".

Tests: runCLI now points the logger at io.Discard after each run, so
other tests' log lines cannot land in a finished run's output.

Model: opus-4-8 (implementation); opus-5-5 (rework)
2026-10-04 06:02:24 +02:00
clawbot d00982b329 Fuzz NewManifestFromReader and cap the zstd decoder (closes #65)
check / check (push) Successful in 1m5s
FuzzNewManifestFromReader fails when the parser returns both or neither
of a manifest and an error, or allocates more than a fixed multiple of
its input and the decompressed data it may read, plus room for the
decoder's window buffers. make test runs the seed corpus; make fuzz
fuzzes for one minute.

Parser bug: MaxDecompressedSize did not bound decompression; the zstd
decoder decoded a payload under 128 KiB in full before the LimitReader
read any of it. It now decodes only what the LimitReader reads and
refuses windows over the 8 MiB mfer writes with. Seeds: a frame claiming
8 GiB, two frames together over the limit, empty frames with growing
windows.

Model: opus-5-5
2026-10-04 05:31:51 +02:00
clawbot 51f69c960d Enforce real timeouts on gpg subprocess calls (closes #62)
check / check (push) Successful in 2m2s
Every gpg run now has a one-minute deadline (gpgTimeout) on top of its
caller's context and is killed when either ends. A timeout is reported
as "gpg timed out" under the failing operation instead of "signal:
killed". Only gpg itself is killed; WaitDelay (one second) stops the run
from waiting on a process gpg left behind that still holds its output,
such as a wrapper script that does not exec the real gpg.
Builder.Build and Checker.ExtractEmbeddedSigningKeyFP take a context, so
ToManifest's context now reaches signing and the contextcheck
suppression calling signing non-cancellable is gone. Manifest loading
takes no context, so its signature check is bounded by the timeout
alone.

Model: opus-5-5
2026-10-04 04:31:53 +02:00
clawbot 1adad7d3bc Remove TODO.md; track open work in the issues (closes #76)
check / check (push) Successful in 1m29s
TODO.md is deleted and the README TODO section now only points to the
issue tracker, where open work and open design questions live from now
on. AGENTS.md says so too, and says this overrides the shared policy's
README todo list for this repo. The README Open Questions section
becomes Original Design Questions, each noting where it now stands.

Every open item in both lists was already done or already owned by an
issue, apart from one, now #121,
and the chunk checksum question, now on
#81.

Model: opus-5-5
2026-10-04 03:32:09 +02:00
clawbot c31796998f Pin CLI error messages by driving their real call sites (closes #87)
check / check (push) Successful in 58s
errmsg_test.go now calls the function that emits each user-visible CLI
error message and asserts on the full text it returns, instead of
rendering format strings copied from production, so rewording any pinned
message fails the suite. The unknown-command message is driven through
the root command's action.

The signer-mismatch case signs a manifest with a throwaway gpg key and
is skipped where gpg is absent, like the repo's other signing tests.

The freshen mtime-presence test gives the scanned file an epoch mtime,
so it fails if recordEntry reads an absent manifest mtime as the epoch.

Model: opus-4-8 (first implementation); opus-5-5 (completion, this message)
2026-10-03 18:24:30 +02:00
clawbot a3749e9e9b cibuild: build with --no-cache like script/docker (closes #89)
check / check (push) Successful in 1m1s
A bare `docker build .` served the Dockerfile's check steps from the
layer cache on an unchanged tree and exited 0 without running them.
script/cibuild now computes the version on its own line and runs the
same build command as script/docker and the shared policy, --no-cache
included, so every CI build runs the checks. README.md describes
script/cibuild accordingly.

Model: opus-5-5
2026-10-03 17:41:55 +02:00
clawbot fa97c4519c Never write fetch's temp file into an existing file (closes #115)
check / check (push) Successful in 53s
downloadFile opened the temp file with os.Create, which opens and
empties a file already at that name. If that file was a hard link to a
file outside the destination directory, fetch overwrote the outside
file. It now removes whatever is at the temp name, which removes only
that name, and creates the temp file with O_EXCL, so the create fails if
anything is still or again there. A leftover temp file from an
interrupted run is still replaced. The new test puts a hard link at the
temp name and checks that fetch succeeds and the outside file is
unchanged. The command name is now the constant cmdFetch, like the other
command names, because lint requires it once a third test uses it.

Model: opus-5-5
2026-10-03 16:58:35 +02:00
clawbot 7e601929c8 Refuse fetch writes through a symlink in the destination (closes #86)
check / check (push) Successful in 1m10s
sanitizePath checks manifest paths only as text, so a symlink already
inside the destination directory could send fetch's writes outside it.
checkNoSymlinks now looks at each existing part of a path with os.Lstat
and refuses the path if any part is a symlink, wherever it points.
fetch runs it immediately before each write: creating the parent
directories, creating the temp file, and renaming it into place. The new
test puts such a symlink at each of those three places, and once inside
a plain directory, and checks that the fetch fails and nothing outside
changes. The G304 comment now states what holds. A symlink swapped in
between a check and its write is not caught; os.Root closes that once
the Go version is raised.

Model: opus-5-5
2026-10-03 16:08:09 +02:00
clawbot c3b5fe651a Stamp the tag or short commit in a plain docker build (closes #112)
check / check (push) Successful in 48s
.dockerignore now sends .git but not .git/config, which can hold a
credential and which git describe does not need. The build stage stamps
main.Gitrev from the VERSION build argument when one is given, otherwise
from git describe --tags --always, and fails if .git is present and no
version comes out. git there trusts /src whoever owns it, since a
context sent as a tar archive keeps its files' owners. script/docker is
replaced by the canonical copy, which passes VERSION; bin/gitrev.sh uses
--tags too, so every entrypoint stamps the same value for a clean
commit.

Model: opus-5-5
2026-10-02 08:53:34 +02:00
clawbot 0deacfc7ed Validate manifest entry paths on deserialize (closes #61)
check / check (push) Failing after 1s
Untrusted .mf files were parsed with no path validation, so an entry like ../../etc/passwd reached filepath.Join against the checker's base path and mfer check could stat and read outside it. ValidatePath ran only when building a manifest. It now runs on every entry as the manifest loads, so every consumer is covered. The whole manifest is rejected on the first bad entry instead of dropping it, which could hide files from a check; the error wraps errInvalidManifestPath and names the path.

Disclosure: a path that is not valid UTF-8 is refused earlier, by the protobuf string decoder, so that error does not name the path.

Model: opus-4-8 (implementation); fable-5-1 (summary)
2026-09-22 00:47:26 +02:00
clawbot 7de4d6ec1c Rewrite script/test to the canonical race-enabled pattern (closes #67)
check / check (push) Successful in 51s
script/test now runs go test -timeout 30s -race -cover ./..., quiet on success and rerunning verbose on failure. -race exposed a data race on the process-global apex/log logger: Init reconfigured it on every CLI run while other goroutines logged through it. internal/log now mutates the global under the write lock and reads it under the read lock; WithError is dropped because its Entry logged outside that lock, and run() reports a failed command's error via log.Errorf instead (still shown under -q). The corruption fuzz test is scaled to 1500 files / 100 iterations to fit the budget under -race; it pins the same claims at reduced breadth. Independent review ran the Docker lint gate uncached.

Model: opus-4-8 (implementation, review)
model: claude-fable-5
2026-09-21 15:17:50 +02:00
clawbot 6de3f1d714 Add .editorconfig, broaden .gitignore, drop dead Drone refs (closes #72)
check / check (push) Failing after 1s
Add the canonical .editorconfig from sneak/prompts verbatim. Broaden .gitignore to cover secrets (.env, .env.*, *.key, *.pem), OS files, editor files, and Go artifacts while keeping every repo-specific entry; no tracked file becomes ignored, and bin/gitrev.sh stays tracked despite the /bin/ rule. Remove the stale .drone.yml ignore entry and the DRONE_COMMIT_SHA branch from bin/gitrev.sh, left from the Gitea Actions migration; the GITREV and git-describe paths still work.

A reader may trip over .editorconfig saying four spaces for every file while Go files are tab-indented: gofmt governs Go, editorconfig does not, and the file is taken verbatim by policy.

Disclosure: the worker's cold docker build hit the 10s test timeout in the gpg signature test; this change has no Go code, and that timeout is tracked by issues 67 and 62.

Model: opus-4-8 (implementation); fable-5-1 (merge)
2026-09-21 09:56:15 +02:00
clawbot 5683d0f4ff Configure prettier and make fmt-check cover markdown (closes #69)
check / check (push) Successful in 4s
Adds .prettierrc and .prettierignore, a single script/prettier entrypoint shared by fmt and fmt-check so the two cannot drift, and prettier 3.9.6 pinned by yarn.lock integrity hash.

Markdown formatting is now gated in the authoritative Docker build via a new mdfmt stage, since the golangci-lint image has no node. REPO_POLICIES.md is ignored so local tooling cannot drift it from upstream.

Removes the || true that made the previous prettier invocation unable to fail.
2026-08-10 16:16:42 +02:00
clawbot de476708e9 Update golangci-lint to v2.12.2 with canonical config (closes #60)
check / check (push) Has been cancelled
Adopts golangci-lint v2.12.2 and the canonical .golangci.yml (default: all), and fixes all resulting findings across the tree.

Two intended behavior changes: absent MFFilePath.Mtime is handled explicitly in freshen, list and export rather than dereferenced (main panicked); gpg positional key IDs now follow an explicit -- end-of-options marker.

All twelve reworded user-visible error messages restored to byte-identical parity with main and pinned by tests.
2026-08-10 16:06:12 +02:00
64 changed files with 1834 additions and 842 deletions
+62 -1
View File
@@ -1,3 +1,64 @@
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
.git/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][dD]25519
# Dependencies: restored inside the image, never copied in.
**/node_modules
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# This repo's own host-built archives (Makefile).
*.tmp
*.dockerimage
.git
+12
View File
@@ -0,0 +1,12 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
+22 -2
View File
@@ -1,5 +1,6 @@
/bin/
/tmp
/node_modules/
*.tmp
*.dockerimage
/vendor
@@ -9,5 +10,24 @@ modcache.tzst
# Generated manifest files
.index.mf
# Stale files
.drone.yml
# Secrets
.env
.env.*
*.key
*.pem
# OS files
.DS_Store
Thumbs.db
# Editor files
*.swp
*.swo
*~
.idea/
.vscode/
# Go build artifacts
*.log
*.out
*.test
+1
View File
@@ -16,6 +16,7 @@ linters:
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
settings:
+14
View File
@@ -0,0 +1,14 @@
# REPO_POLICIES.md is a verbatim copy of an authoritative upstream
# document (sneak/prompts). Local tooling must never rewrite it: any
# reformatting is silent drift from the source of truth.
REPO_POLICIES.md
# User-owned configuration, copied verbatim from upstream. Not matched by
# the current prettier file set, but listed so widening that set can never
# start rewriting it.
.golangci.yml
# Dependencies and build output
node_modules/
vendor/
bin/
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+7 -3
View File
@@ -13,7 +13,8 @@ source for coding standards, formatting, linting, and workflow rules.
- After each change, run `make fmt`, then `make test`, then `make lint`. Fix any
failures before committing.
- After each change, commit only the files you've changed. Push after committing.
- After each change, commit only the files you've changed. Push after
committing.
## Attribution
@@ -26,5 +27,8 @@ source for coding standards, formatting, linting, and workflow rules.
- The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are
committed (required for `go get` compatibility).
- The format specification is in `FORMAT.md`.
- See the TODO section in `README.md` for the 1.0 implementation plan
and open design questions.
- Open work, open design questions included, is tracked only in the repo's
issues: https://git.eeqj.de/sneak/mfer/issues. There is no `TODO.md` and no
TODO list in `README.md`; do not add either. For this repo this overrides the
`REPO_POLICIES.md` rule to put the todo list in the README, per sneak's
ruling: https://git.eeqj.de/sneak/mfer/issues/76#issuecomment-118130.
+45 -4
View File
@@ -11,15 +11,34 @@ COPY . .
# Touch .pb.go so make does not try to regenerate via protoc (file is committed)
RUN touch mfer/mf.pb.go
RUN make fmt-check
RUN make lint
# Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below.
RUN make fmt-check-go
# The linter directly, not `make lint`: script/lint builds this stage, and
# there is no docker inside this build.
RUN golangci-lint run --config .golangci.yml ./...
# Markdown/JSON format stage — prettier needs node, which the Go images
# do not have. node:22.17.0-bookworm-slim (2026-08-09); ships node
# 22.17.0 and yarn 1.22.22, the versions script/bootstrap pins.
FROM node@sha256:b04ce4ae4e95b522112c2e5c52f781471a5cbc3b594527bcddedee9bc48c03a0 AS mdfmt
WORKDIR /src
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
# No make in this image; call the script entrypoint directly.
RUN script/prettier --check
# Build stage — tests and compilation
# golang:1.23 (2026-03-14)
FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder
# Force BuildKit to run the lint stage by creating a stage dependency
# Force BuildKit to run the lint and mdfmt stages by creating stage dependencies
COPY --from=lint /src/go.sum /dev/null
COPY --from=mdfmt /src/go.sum /dev/null
WORKDIR /src
COPY go.mod go.sum ./
@@ -31,7 +50,29 @@ COPY . .
RUN touch mfer/mf.pb.go
RUN make test
RUN cd cmd/mfer && go build -tags urfave_cli_no_docs -o /mfer .
# A build context sent as a tar archive, as upaas sends it, keeps its files'
# owners, and git refuses to read a checkout owned by another user.
RUN git config --system --add safe.directory /src
# The revision `mfer version` prints, stamped into main.Gitrev: the VERSION
# build argument when one is given (script/docker passes one), otherwise
# `git describe --tags --always` of the .git the build context carries: the
# tag on a tagged commit, tag-N-gHASH on a commit after one, the short commit
# when no tag is reachable. git ships in this base image. A context that
# carries .git and still yields no version fails the build.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "no version could be derived although the build context carries .git" >&2; \
exit 1; \
fi; \
cd cmd/mfer && \
CGO_ENABLED=0 go build -tags urfave_cli_no_docs -ldflags "-X main.Gitrev=$version" -o /mfer .
# Fail unless /mfer is statically linked: scratch has no C library to run it.
RUN ldd /mfer 2>&1 | grep -q 'not a dynamic executable'
FROM scratch
COPY --from=builder /mfer /mfer
+19 -16
View File
@@ -5,9 +5,9 @@ Version 1.0
## Overview
An `.mf` file is a binary manifest that describes a directory tree of files,
including their paths, sizes, and cryptographic checksums. It supports
optional GPG signatures for integrity verification and optional timestamps
for metadata preservation.
including their paths, sizes, and cryptographic checksums. It supports optional
GPG signatures for integrity verification and optional timestamps for metadata
preservation.
## File Structure
@@ -39,15 +39,17 @@ The outer message contains:
### SHA-256 Hash
The `sha256` field (104) covers the **compressed** `innerMessage` bytes.
This allows verifying data integrity before decompression.
The `sha256` field (104) covers the **compressed** `innerMessage` bytes. This
allows verifying data integrity before decompression.
## Compression
The `innerMessage` field is compressed with [Zstandard (zstd)](https://facebook.github.io/zstd/).
Implementations must enforce a decompression size limit to prevent
decompression bombs. The reference implementation limits decompressed size to
256 MB.
The `innerMessage` field is compressed with
[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must
enforce a decompression size limit to prevent decompression bombs. The reference
implementation limits decompressed size to 256 MB. It writes zstd frames with a
window of at most 8 MiB, the largest window the zstd format recommends decoders
support, and refuses frames that ask for a larger one.
## Inner Message (`MFFile`)
@@ -114,11 +116,11 @@ Where:
- `ZNAVSRFG` is the magic bytes string (literal ASCII)
- `<UUID>` is the hex-encoded UUID from the outer message
- `<SHA256>` is the hex-encoded SHA-256 hash from the outer message (covering compressed data)
- `<SHA256>` is the hex-encoded SHA-256 hash from the outer message (covering
compressed data)
Components are separated by hyphens. The signature is produced by GPG over
this canonical string and stored in the `signature` field of the outer
message.
Components are separated by hyphens. The signature is produced by GPG over this
canonical string and stored in the `signature` field of the outer message.
## Deterministic Serialization
@@ -134,10 +136,11 @@ changed).
## MIME Type
The recommended MIME type for `.mf` files is `application/octet-stream`.
The `.mf` file extension is the canonical identifier.
The recommended MIME type for `.mf` files is `application/octet-stream`. The
`.mf` file extension is the canonical identifier.
## Reference
- Proto definition: [`mfer/mf.proto`](mfer/mf.proto)
- Reference implementation: [git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
- Reference implementation:
[git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
+12 -4
View File
@@ -13,7 +13,7 @@ GOLDFLAGS += -X main.Version=$(VERSION)
GOLDFLAGS += -X main.Gitrev=$(GITREV_BUILD)
GOFLAGS := -ldflags "$(GOLDFLAGS)"
.PHONY: bootstrap setup docker default run ci test check lint fmt fmt-check hooks fixme
.PHONY: bootstrap setup docker default run ci test fuzz check lint fmt fmt-check fmt-check-go fmt-check-md hooks fixme
default: fmt test
@@ -32,6 +32,9 @@ ci: test
test:
@script/test
fuzz:
@script/fuzz
$(PROTOC_GEN_GO):
test -e $(PROTOC_GEN_GO) || go install -v google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.1
@@ -44,12 +47,17 @@ check:
fmt-check:
@script/fmt-check
# Halves of fmt-check, for environments that have only one toolchain:
# the Docker lint stage has Go but no node, the markdown stage the reverse.
fmt-check-go:
@script/fmt-check-go
fmt-check-md:
@script/prettier --check
hooks:
@script/install-precommit
devprereqs:
which golangci-lint || go install -v github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2
mfer/mf.pb.go: mfer/mf.proto
cd mfer && go generate .
+202 -340
View File
@@ -1,138 +1,200 @@
# mfer
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library
and thin wrapper command-line utility written in [Go](https://golang.org)
and first published in 2022 under the [WTFPL](https://wtfpl.net) (public
domain) license. It specifies and generates `.mf` manifest files over a
directory tree of files to encapsulate metadata about them (such as
cryptographic checksums or signatures over same) to aid in archiving,
downloading, and streaming, or mirroring. The manifest files' data is
serialized with Google's [protobuf serialization
format](https://developers.google.com/protocol-buffers). The structure of
these files can be found [in the format
specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
which is included in the [project
repository](https://git.eeqj.de/sneak/mfer).
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
(public domain) [Go](https://golang.org) library and command-line tool by
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
over a directory tree to encapsulate metadata about the files — such as
cryptographic checksums and signatures over same — to aid in archiving,
downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
which is included in the [project repository](https://git.eeqj.de/sneak/mfer).
The current version is pre-1.0 and while the repo was published in 2022,
there has not yet been any versioned release. [SemVer](https://semver.org)
will be used for releases.
The current version is pre-1.0 and while the repo was published in 2022, there
has not yet been any versioned release. [SemVer](https://semver.org) will be
used for releases.
This project was started by [@sneak](https://sneak.berlin) to scratch an
itch in 2022 and is currently a one-person effort, though the goal is for
this to emerge as a de-facto standard and be incorporated into other
software. A compatible javascript library is planned.
This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
2022 and is currently a one-person effort, though the goal is for this to emerge
as a de-facto standard and be incorporated into other software. A compatible
javascript library is planned.
# Getting Started
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
is committed, so no `protoc` toolchain is required:
```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
go build -o bin/mfer ./cmd/mfer
```
Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:
```sh
# Write .index.mf, a manifest of the files under the current directory.
bin/mfer gen .
# Verify the files on disk against the manifest. Exits nonzero if any file
# is missing or corrupted.
bin/mfer check .index.mf
# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/
```
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
for a single command's options.
# Build Status
CI runs via `script/cibuild` (`docker build .`), which executes `make
check` (formatting, linting, tests). The `main` branch must always be
green.
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
the formatting, lint and test steps in the `Dockerfile` run on every build. The
`main` branch must always be green.
# Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call
them. We provide:
development workflow, and the Makefile targets are thin shims that call them. We
provide:
- `script/bootstrap` — install all dependencies (Go, golangci-lint, Go
module download), idempotently
- `script/bootstrap` — install all dependencies (Go, Go module download, and
node/yarn plus the prettier version pinned in `package.json`/`yarn.lock`),
idempotently; golangci-lint is not installed, it runs only in Docker
- `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (`mfer`); used by other
scripts such as `script/docker`
- `script/test` — run the test suite (`go test`), regenerating the
protobuf code first if it is stale
- `script/lint` — run `golangci-lint` and verify `gofmt` cleanliness
- `script/fmt` — format all code and docs (writes): `gofumpt`,
`golangci-lint run --fix`, and prettier for JSON/Markdown
- `script/fmt-check` — check formatting without writing
- `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`
- `script/projectname` — output the project name (`mfer`); used by other scripts
such as `script/docker`
- `script/test` — run the test suite (`go test`), regenerating the protobuf code
first if it is stale
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
as ordinary tests
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
every time, then removes the image
- `script/fmt` — format all code and docs (writes): `gofumpt` and
`script/prettier --write`
- `script/prettier` — run prettier over the repository's canonical file set
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
`--check`; the single definition of that file set, so `script/fmt` and
`script/fmt-check` cannot disagree about it
- `script/fmt-check` — check formatting without writing: `script/fmt-check-go`
plus `script/prettier --check`
- `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the
Docker lint stage, whose image has no node
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/docker` — build the Docker image tagged with the project name
- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile
runs the checks)
- `script/precommit` — pre-commit checks: `go mod tidy` verification,
then `script/check`
- `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
- `script/cibuild` — CI entrypoint: builds the image with the same command as
`script/docker`, uncached, so the checks in the Dockerfile run every time
- `script/precommit` — pre-commit checks: `go mod tidy` verification, then
`script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`
# Participation
The community is as yet nonexistent so there are no defined policies or
norms yet. Primary development happens on a privately-run Gitea instance at
[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues
are [tracked there](https://git.eeqj.de/sneak/mfer/issues).
The community is as yet nonexistent so there are no defined policies or norms
yet. Primary development happens on a privately-run Gitea instance at
[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues are
[tracked there](https://git.eeqj.de/sneak/mfer/issues).
Changes must always be formatted with a standard `go fmt`, syntactically
valid, and must pass the linting defined in the repository (presently only
the `golangci-lint` defaults), which can be run with a `make lint`. The
`main` branch is protected and all changes must be made via [pull
requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged.
Any changes submitted to this project must also be
Changes must always be formatted with a standard `go fmt`, syntactically valid,
and must pass the linting defined in the repository's `.golangci.yml`, which
`make lint` runs in Docker. The `main` branch is protected and all changes must
be made via [pull requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to
be merged. Any changes submitted to this project must also be
[WTFPL-licensed](https://wtfpl.net) to be considered.
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions.
# Problem Statement
# Rationale
## The problem
Given a plain URL, there is no standard way to safely and programmatically
download everything "under" that URL path. `wget -r` can traverse directory
listings if they're enabled, but every server has a different format, and
this does not verify cryptographic integrity of the files, or enable them to
be fetched using a different protocol other than HTTP/s.
listings if they're enabled, but every server has a different format, and this
does not verify cryptographic integrity of the files, or enable them to be
fetched using a different protocol other than HTTP/s.
Currently, the solution that people are using are sidecar files in the
format of `SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached
signature. This is not checksum-algorithm-agnostic and the sidecar file is
not always consistently named.
Currently, the solution that people are using are sidecar files in the format of
`SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached signature.
This is not checksum-algorithm-agnostic and the sidecar file is not always
consistently named.
Real issues I face:
- when I plug in an ExFAT hard drive, I don't know if any files on the
filesystem are corrupted or missing
- current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files
- current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files
- when I want to mirror an HTTP archive, I have to use special tools like
debmirror that understand the archive format
- the debian repository metadata structure is hot garbage
- when I download a large file via HTTP, I have no way of knowing if the
file content is what it's supposed to be
- the debian repository metadata structure is hot garbage
- when I download a large file via HTTP, I have no way of knowing if the file
content is what it's supposed to be
# Proposed Solution
## The solution
A standard, a manifest file format, and a tool for generating same.
The manifest file would be called `index.mf`, and the tool for generating such would be called `mfer`.
The manifest file would be called `index.mf`, and the tool for generating such
would be called `mfer`.
The manifest file would do several important things:
- have a standard filename, so if given
`https://example.com/downloadpackage/` one could fetch
`https://example.com/downloadpackage/index.mf` to enumerate the full
directory listing.
- have a standard filename, so if given `https://example.com/downloadpackage/`
one could fetch `https://example.com/downloadpackage/index.mf` to enumerate
the full directory listing.
- contain a version field for extensibility
- contain structured data (protobuf, json, or cbor)
- provide an inner signed container, so that the manifest file itself can
embed a signature and a public key alongside in a single file
- provide an inner signed container, so that the manifest file itself can embed
a signature and a public key alongside in a single file
- contain a list of files, each with a relative path to the manifest
- contain manifest timestamp
- contain ctime/mtime information for files so that file metadata can be
preserved
- contain cryptographic checksums in several different algorithms for each
file
- probably encoded with multihash to indicate algo + hash
- sha256 at the minimum
- would be nice to include an IPFS/IPLD CIDv1 root hash for each file,
which likely involves doing an ipfs file object chunking
- maybe even including the complete IPFS/IPLD directory tree objects and
chunklists?
- this is because generating an `index.mf` does not imply publishing on
ipfs at that time
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
top-level infohash for the whole manifest?
- contain cryptographic checksums in several different algorithms for each file
- probably encoded with multihash to indicate algo + hash
- sha256 at the minimum
- would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which
likely involves doing an ipfs file object chunking
- maybe even including the complete IPFS/IPLD directory tree objects and
chunklists?
- this is because generating an `index.mf` does not imply publishing on
ipfs at that time
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
top-level infohash for the whole manifest?
# Design
The repository is split into a reusable library and a thin command-line wrapper
around it.
- `mfer/` is the reusable library and the heart of the project: it defines the
manifest format and implements building, scanning, checking, serialization,
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
with `go get` and needs no `protoc` toolchain.
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
`check`, `freshen`, `export`, `list` (alias `ls`), `fetch`, and `version` —
that wire the library to the command-line interface.
- `internal/log/` provides the logging used by the commands and the library.
- `internal/bork/` provides the error the library returns when a manifest's
decompressed contents are not the size the manifest records.
- `cmd/mfer/` is the entrypoint: its `main` package runs `internal/cli` and
exits with the status it returns.
Everything under `internal/` is private to this repository; only the `mfer/`
package is intended for import by other software.
# Design Goals
@@ -146,37 +208,42 @@ The manifest file would do several important things:
# Non-Goals
- Manifest generation speed
- likely involves IPFS chunking, bittorrent chunking, and several
different cryptographic hash functions over the entirety of each and
every file
- likely involves IPFS chunking, bittorrent chunking, and several different
cryptographic hash functions over the entirety of each and every file
- Small manifest file size (within reason)
- 30MiB files are "small" these days, given modern storage/bandwidth
- metadata size should not be used as an excuse to sacrifice utility (such
as providing checksums over each chunk of a large file)
- 30MiB files are "small" these days, given modern storage/bandwidth
- metadata size should not be used as an excuse to sacrifice utility (such
as providing checksums over each chunk of a large file)
# Open Questions
# Original Design Questions
- Should the manifest file include checksums of individual file chunks, or just for the whole assembled file?
These were the open questions when the project started; open design questions
are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- If so, should the chunksize be fixed or dynamic?
- Should the manifest file include checksums of individual file chunks, or just
for the whole assembled file? If so, should the chunk size be fixed or
dynamic? Still open, on
[issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698).
- Should the manifest signature format be GnuPG signatures, or those from
OpenBSD's signify (of which there is a good [golang
implementation](https://github.com/frankbraun/gosignify)?
OpenBSD's signify (of which there is a good
[golang implementation](https://github.com/frankbraun/gosignify))? Still open,
as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).
- Should the on-disk serialization format be proto3 or json?
- Should the on-disk serialization format be proto3 or json? Settled: it is
proto3, see `FORMAT.md` and `mfer/mf.proto`.
# Tool Examples
- `mfer gen` / `mfer gen .`
- recurses under current directory and writes out an `index.mf`
- recurses under current directory and writes out an `index.mf`
- `mfer check` / `mfer check .`
- verifies checksums of all files in manifest, displaying error and
exiting nonzero if any files are missing or corrupted
- verifies checksums of all files in manifest, displaying error and exiting
nonzero if any files are missing or corrupted
- `mfer fetch https://example.com/stuff/`
- fetches `/stuff/index.mf` and downloads all files listed in manifest,
optionally resuming any that already exist locally, and assures
cryptographic integrity of downloaded files.
- fetches `/stuff/index.mf` and downloads all files listed in manifest,
optionally resuming any that already exist locally, and assures
cryptographic integrity of downloaded files.
# Implementation Plan
@@ -193,30 +260,31 @@ The manifest file would do several important things:
# Hopes And Dreams
- `aria2c https://example.com/manifestdirectory/`
- (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
checksums all files, resumes any that exist locally already)
- (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
checksums all files, resumes any that exist locally already)
- `mfer fetch https://example.com/manifestdirectory/`
- a command line option to zero/omit mtime/ctime, as well as manifest
timestamp, and sort all directory listings so that manifest file
generation is deterministic/reproducible
- URL format `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`
to assert in the URL which PGP signing key should be used in the manifest,
so that shared URLs have a cryptographic trust root
- a "well-known" key in the manifest that maps well known keys (could reuse
the http spec) to specific file paths in the manifest.
- example: a `berlin.sneak.app.slideshow` key that maps to a json
slideshow config listing what image paths to show, and for how long, and
in what order
- a command line option to zero/omit mtime/ctime, as well as manifest timestamp,
and sort all directory listings so that manifest file generation is
deterministic/reproducible
- URL format
`mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`
to assert in the URL which PGP signing key should be used in the manifest, so
that shared URLs have a cryptographic trust root
- a "well-known" key in the manifest that maps well known keys (could reuse the
http spec) to specific file paths in the manifest.
- example: a `berlin.sneak.app.slideshow` key that maps to a json slideshow
config listing what image paths to show, and for how long, and in what
order
# Use Cases
## Web Images
I'd like to be able to put a bunch of images into a directory, generate a
manifest, and then point a slideshow client (such as an ambient display, or
a react app with the target directory in a query string arg) at that
statically hosted directory, and have it discover the full list of images
available at that URL.
manifest, and then point a slideshow client (such as an ambient display, or a
react app with the target directory in a query string arg) at that statically
hosted directory, and have it discover the full list of images available at that
URL.
## Software Distribution
@@ -226,232 +294,25 @@ resumably by either HTTP or IPFS/BitTorrent without a .torrent file.
## Filesystem Archive Integrity
I use filesystems that don't include data checksums, and I would like a
cryptographically signed checksum file so that I can later verify that a set
of archive files have not been modified, none are missing, and that the
checksums have not been altered in storage by a second party.
cryptographically signed checksum file so that I can later verify that a set of
archive files have not been modified, none are missing, and that the checksums
have not been altered in storage by a second party.
## Filesystem-Independent Checksums
I would like to be able to plug in a hard drive or flash drive and, if there
is an `index.mf` in the root, automatically detect missing/corrupted files,
I would like to be able to plug in a hard drive or flash drive and, if there is
an `index.mf` in the root, automatically detect missing/corrupted files,
regardless of filesystem format.
# Collaboration
Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your
desired username for an account on this Gitea instance.
Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired
username for an account on this Gitea instance.
# TODO: Remaining Work for 1.0
# TODO
## Design Questions (Owner Decision Required)
These require @sneak's input before implementation. Answers should be added
inline below each question.
### Format Design
**1. Should `MFFileChecksum` be simplified?** Currently it's a separate
message wrapping a single `bytes multiHash` field. Since multihash
already self-describes the algorithm, `repeated bytes hashes` directly on
`MFFilePath` would be simpler and reduce per-file protobuf overhead. Is
the extra message layer intentional (e.g. planning to add per-hash
metadata like `verified_at`)?
> _answer:_
**2. Should file permissions/mode be stored?** The format stores
mtime/ctime but not Unix file permissions. For archival use this may not
matter, but for software distribution or filesystem restoration it's a
gap. Should we reserve a field now (e.g. `optional uint32 mode = 305`)
even if we don't populate it yet?
> _answer:_
**3. Should `atime` be removed from the schema?** Access time is
volatile, non-deterministic, and often disabled (`noatime`). Including it
means two manifests of the same directory at different times will differ,
which conflicts with the determinism goal. Remove it, or document it as
"never set by default"?
> _answer:_
**4. What are the path normalization rules?** The proto has `string path`
with no specification about: always forward-slash? Must be relative? No
`..` components allowed? UTF-8 NFC vs NFD normalization (macOS vs
Linux)? Max path length? This is a security issue (path traversal) and a
cross-platform compatibility issue. What rules should the spec mandate?
> _answer:_
**5. Should we add a version byte after the magic?** Currently
`ZNAVSRFG` is followed immediately by protobuf. Adding a version byte
(`ZNAVSRFG\x01`) would allow future framing changes without requiring
protobuf parsing to detect the version. `MFFileOuter.Version` serves
this purpose but requires successful deserialization to read. Worth the
extra byte?
> _answer:_
**6. Should we add a length-prefix after the magic?** Protobuf is not
self-delimiting. If we ever want to concatenate manifests or append data
after the protobuf, the current framing is insufficient. Add a varint or
fixed-width length-prefix?
> _answer:_
### Signature Design
**7. What does the outer SHA-256 hash cover — compressed or uncompressed
data?** The code currently hashes compressed data (good for verifying
before decompression), but this should be explicitly documented. Which is
the intended behavior?
> _answer:_
**8. Should `signatureString()` sign raw bytes instead of a hex-encoded
string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with
hex encoding, which adds a transformation layer. Signing the raw `sha256`
bytes (or compressed `innerMessage` directly) would be simpler. Keep the
string format or switch to raw bytes?
> _answer:_
**9. Should we support detached signature files (`.mf.sig`)?** Embedded
signatures are better for single-file distribution. Detached `.mf.sig`
files follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are
simpler for HTTP serving. Support both modes?
> _answer:_
**10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is
fragile (may not be installed, version-dependent output).
`github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could
use Ed25519/signify (simpler, no key management). Which direction?
> _answer:_
### Implementation Design
**11. Should manifests be deterministic by default?** This means: sort
file entries by path, omit `createdAt` timestamp (or make it opt-in), no
`atime`. Should determinism be the default, with a
`--include-timestamps` flag to opt in?
> _answer:_
**12. Should we consolidate or keep both scanner/checker
implementations?** There are two parallel implementations:
`mfer/scanner.go` + `mfer/checker.go` (typed with `FileSize`,
`RelFilePath`) and `internal/scanner/` + `internal/checker/` (raw
`int64`, `string`). The `mfer/` versions are superior. Delete the
`internal/` versions?
> _answer:_
**13. Should the `manifest` type be exported?** Currently unexported with
exported constructors (`NewManifestFromReader`, `NewManifestFromFile`).
Consumers can't declare `var m *mfer.manifest`. Export the type, or
define an interface?
> _answer:_
**14. What should the Go module path be for 1.0?** Currently
`sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in
the proto `go_package` option. Which is canonical?
> _answer:_
## Implementation Tasks
### Repo Infrastructure
- [ ] Add `.golangci.yml` (fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`)
- [ ] Add `.editorconfig`
- [ ] Add `.gitea/workflows/check.yml` that runs `docker build .`
### Format & Correctness
- [ ] Resolve proto `go_package` path inconsistency
(`git.eeqj.de/sneak/mfer/mfer` vs `sneak.berlin/go/mfer`)
- [ ] Specify path invariants — add proto comments requiring UTF-8,
forward-slash, relative paths, no `..`, no leading `/`; validate
in `Builder.AddFile` and `Builder.AddFileWithHash` (pending design
question answer)
- [ ] Remove or deprecate `atime` from proto (pending design question
answer)
- [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future
file permissions (pending design question answer)
- [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version
1 (pending design question answer)
- [ ] Write format specification document — separate from README:
magic, outer structure, compression, inner structure, path
invariants, signature scheme, canonical serialization
### Library
- [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate
on `mfer/` package versions; update CLI code (pending design
question answer)
- [ ] Add deterministic file ordering — sort entries by path
(lexicographic, byte-order) in `Builder.Build()`; add test
asserting byte-identical output from two runs
- [ ] Add decompression size limit — `io.LimitReader` in
`deserializeInner()` with `m.pbOuter.Size` as bound
- [ ] Fix `errors.Is` dead code in checker — replace with
`os.IsNotExist(err)` or `errors.Is(err, fs.ErrNotExist)`
- [ ] Fix `AddFile` to verify size — check `totalRead == size` after
reading, return error on mismatch
- [ ] Export the `manifest` type or define a public interface (pending
design question answer) — currently consumers cannot hold a reference
to a loaded manifest in their own type declarations
- [ ] Replace GPG subprocess calls with pure-Go crypto (pending design
question answer) — current implementation shells out to `gpg` which
may not be installed
- [ ] Add timeout to any remaining subprocess calls
### CLI
- [ ] Fix flag naming — all CLI flags should use kebab-case as primary
(`--include-dotfiles`, `--follow-symlinks`)
- [ ] Fix URL construction in fetch — use `BaseURL.JoinPath()` or
`url.JoinPath()` instead of string concatenation
- [ ] Add progress rate-limiting to Checker — throttle to once per
second, matching Scanner
- [ ] Add `--deterministic` flag or make it default — omit `createdAt`,
sort files (pending design question answer)
- [ ] Wire `--version` flag properly (currently only a `version`
subcommand exists; top-level `--version` shows urfave/cli generic
output)
- [ ] Add retry logic to `fetch` — currently no retries on transient
HTTP errors; needs exponential backoff
- [ ] `fetch` command uses bare `http.Get` with no timeout — needs
`http.Client` with configurable timeout
### Testing & Robustness
- [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf
deserialization of untrusted input needs fuzz coverage
- [ ] Add integration test for `freshen` CLI command — current tests
only verify setup, not the actual freshen operation end-to-end
- [ ] Add test for `fetch` CLI command end-to-end (currently only
`downloadFile` is tested)
### Documentation
- [ ] Promote `FORMAT.md` as primary spec reference; README should link
to it more prominently
- [ ] Audit and update all error messages for consistency and
helpfulness
- [ ] Document the signature scheme more thoroughly (canonical string
format, verification steps)
### Release
- [ ] Finalize Go module path
- [ ] Update version constant in `mfer/constants.go`
- [ ] Add `--version` output matching SemVer
- [ ] Tag `v1.0.0`
Open work, open design questions included, is tracked in this repo's issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues).
# See Also
@@ -465,11 +326,12 @@ the proto `go_package` option. Which is canonical?
## Links
- Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
- Issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
- Issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
# Authors
# Author
- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
- [@sneak](https://sneak.berlin)
# License
-111
View File
@@ -1,111 +0,0 @@
# Workflow
- branch (from `main`)
- do the work in Next Step
- move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work)
- merge to `main` if the branch is not protected, otherwise open a PR
- push
# Status
pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists
open design questions and implementation tasks; policy compliance work is
in flight and unmerged.
# Next Step
Work through the remaining compliance items folded from the 2026-07-02
audit (the first group under Future Steps): `.editorconfig`, `.gitignore`
coverage, gofumpt-based `fmt-check`, README "Getting Started", and the
rest. `.golangci.yml` and `TODO.md` are tracked and committed as of
2026-08-07, so the only thing left of the `chore/align-repo-policies`
branch is the list below.
# Completed Steps
- 2026-08-07: updated golangci-lint to v2.12.2 everywhere it is pinned
(`Makefile`, `Dockerfile`), added the canonical `.golangci.yml`
(`default: all`), and fixed all resulting lint findings across the
codebase
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
Makefile shims, README Entrypoints section
- 2026-07-03: aligned repo tooling, docs, and config with standardized
policies (7d9a138, on chore/align-repo-policies, unmerged)
- 2026-06-28: moved to standardized repo policies (#56, on main)
- 2026-04-07: added 1.0 roadmap as README TODO section, removed old
TODO.md (#54)
- 2026-03-20: added Gitea Actions CI workflow (#53)
- 2026-03-17: added REPO_POLICIES.md, renamed CLAUDE.md to AGENTS.md (#51);
removed committed .index.mf (#52)
- 2026-03-15: split Dockerfile with pre-built golangci-lint stage for
faster CI (#45)
- 2026-03-01: 1.0 quality polish: code review, tests, bug fixes, docs (#32)
- 2026-02-20: deterministic file ordering in Builder.Build() (#28);
removed committed vendor/modcache archives (#35)
- 2026-02-08: added --seed flag for deterministic manifest UUID
# Future Steps
- Compliance (fold of TODO.md audit 2026-07-02; verify which items the
in-flight branch already closes, then check off):
- Add .editorconfig (canonical copy from sneak/prompts)
- Make .gitignore cover secrets (.env, _.key, _.pem), OS files
(.DS_Store), and editor files (_.swp, _~)
- Make fmt-check/lint verify with gofumpt, not gofmt -l, so
`make check` matches what `make fmt` writes
- Add README "Getting Started" section with copy-pasteable
install/usage block
- Move FORMAT.md from repo root to docs/ and update the AGENTS.md
reference
- Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`,
`golangci-lint@v2.12.2`) by module hash, not mutable tag
- Set `make test` timeout to 30s (currently 10s)
- Add explicit README "Rationale" heading (content exists under other
names); name the author in the README Description first line
- Reconcile root-level AGENTS.md with directory-hygiene policy (keep
or relocate)
- Add a `make build` target
- Rewrite `make hooks` to use printf or a heredoc instead of
non-portable `echo '...\n...'`
- Answer the 14 owner design questions in the README 1.0 roadmap:
- Format: simplify MFFileChecksum; store file mode; drop atime;
specify path normalization rules; version byte after magic;
length-prefix after magic
- Signatures: hash covers compressed or uncompressed data; sign raw
bytes vs hex canonical string; detached .mf.sig support; GPG
subprocess vs pure-Go crypto
- Implementation: deterministic manifests by default; consolidate
duplicate scanner/checker implementations; export the manifest
type; canonical Go module path for 1.0
- Format and correctness:
- Resolve proto go_package vs go.mod module path inconsistency
- Specify and validate path invariants (UTF-8, forward-slash,
relative, no .., no leading /)
- Remove or deprecate atime; reserve mode field; add version byte
(all pending design answers)
- Write a standalone format specification document
- Library:
- Delete internal/scanner and internal/checker; consolidate on the
mfer/ package versions (pending design answer)
- Add decompression size limit via io.LimitReader in
deserializeInner()
- Fix errors.Is dead code in checker; make AddFile verify
totalRead == size
- Export manifest type or define a public interface (pending)
- Replace GPG subprocess with pure-Go crypto (pending); add timeouts
to remaining subprocess calls
- CLI:
- Kebab-case primary flag names; fix fetch URL construction with
url.JoinPath; add http.Client timeout and retry with backoff to
fetch; rate-limit Checker progress output; add --deterministic
flag or default; wire top-level --version properly
- Testing:
- Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch
- Documentation:
- Promote docs/FORMAT.md as primary spec reference; audit error
messages; document the signature scheme fully
- Release:
- Finalize module path, bump version constant, SemVer --version
output, tag v1.0.0
+1 -6
View File
@@ -1,12 +1,7 @@
#!/bin/bash
#
if [[ ! -z "$DRONE_COMMIT_SHA" ]]; then
echo "${DRONE_COMMIT_SHA:0:7}"
exit 0
fi
if [[ ! -z "$GITREV" ]]; then
echo $GITREV
else
git describe --always --dirty=-dirty
git describe --tags --always --dirty=-dirty
fi
+6 -3
View File
@@ -2,6 +2,7 @@
package cli
import (
"context"
"encoding/hex"
"errors"
"fmt"
@@ -126,7 +127,9 @@ func (mfa *CLIApp) fetchManifestToTemp(url string) (string, error) {
// verifyRequiredSigner enforces the --require-signature fingerprint
// against the manifest's embedded signing key.
func verifyRequiredSigner(chk *mfer.Checker, requiredSigner string) error {
func verifyRequiredSigner(
ctx context.Context, chk *mfer.Checker, requiredSigner string,
) error {
// Validate fingerprint format: must be exactly 40 hex characters
if len(requiredSigner) != fingerprintHexLen {
return fmt.Errorf("%w, got %d", errInvalidFingerprint, len(requiredSigner))
@@ -145,7 +148,7 @@ func verifyRequiredSigner(chk *mfer.Checker, requiredSigner string) error {
// Extract fingerprint from the embedded public key (not from the
// signer field). This validates the key is importable and gets its
// actual fingerprint.
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(ctx)
if err != nil {
return fmt.Errorf(
"failed to extract fingerprint from embedded signing key: %w", err)
@@ -303,7 +306,7 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
// Check signature requirement
requiredSigner := ctx.String("require-signature")
if requiredSigner != "" {
err = verifyRequiredSigner(chk, requiredSigner)
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
if err != nil {
return err
}
+136 -12
View File
@@ -5,6 +5,7 @@ import (
"bytes"
"errors"
"fmt"
"io"
"math/rand"
"os"
"sync"
@@ -14,6 +15,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2"
"sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer"
)
@@ -27,6 +29,7 @@ const (
testManifest = "/manifest.mf"
testFlagBase = "--base"
testFlagNoExtra = "--no-extra-files"
testFlagVersion = "--version"
)
var errSimulatedWrite = errors.New("simulated write failure")
@@ -39,12 +42,20 @@ var errSimulatedWrite = errors.New("simulated write failure")
var runMu sync.Mutex
// runCLI invokes RunWithOptions while holding runMu so parallel tests
// capture their own output.
// capture their own output. Before releasing the lock it points the
// process-global logger at io.Discard: other tests log outside the lock
// (manifest loads, scans), and those lines must not land in this run's
// buffers once it has returned and its test is reading them.
func runCLI(opts *RunOptions) int {
runMu.Lock()
defer runMu.Unlock()
return RunWithOptions(opts)
exitCode := RunWithOptions(opts)
log.SetOutput(io.Discard, io.Discard)
log.Init()
return exitCode
}
func TestMain(m *testing.M) {
@@ -102,7 +113,7 @@ func TestVersionCommand(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
opts := testOpts([]string{testApp, "version"}, fs)
opts := testOpts([]string{testApp, cmdVersion}, fs)
exitCode := runCLI(opts)
@@ -113,6 +124,117 @@ func TestVersionCommand(t *testing.T) {
assert.Contains(t, stdout, "abc123")
}
// TestVFlagCollision covers the -v/--verbose vs --version flag interaction
// (issue #64). Verbose owns -v; version answers to --version and -V. None of
// these invocations may produce a parser error, and the two ways of asking
// for the version must print the same thing.
func TestVFlagCollision(t *testing.T) {
t.Parallel()
// Invocations that must print the version and exit 0.
versionCases := map[string][]string{
"long version flag": {testApp, testFlagVersion},
"short version flag": {testApp, "-V"},
"verbose then version": {testApp, "-v", testFlagVersion},
"long verbose and version": {testApp, "--verbose", testFlagVersion},
}
for name, args := range versionCases {
t.Run(name, func(t *testing.T) {
t.Parallel()
opts := testOpts(args, afero.NewMemMapFs())
exitCode := runCLI(opts)
assert.Equal(t, 0, exitCode, "stderr: %s", testStderr(t, opts))
assert.Contains(t, testStdout(t, opts), mfer.Version)
assert.NotContains(t, testStderr(t, opts), "two forms of the same flag")
})
}
// Invocations that must enable verbose and exit 0 without a parser error.
verboseCases := map[string][]string{
"short verbose flag": {testApp, "-v"},
"long verbose flag": {testApp, "--verbose"},
}
for name, args := range verboseCases {
t.Run(name, func(t *testing.T) {
t.Parallel()
opts := testOpts(args, afero.NewMemMapFs())
exitCode := runCLI(opts)
assert.Equal(t, 0, exitCode, "stderr: %s", testStderr(t, opts))
assert.Contains(t, testStdout(t, opts), cmdGenerate,
"root should show help listing subcommands")
})
}
}
// TestVersionFlagAndCommandMatch asserts that "mfer --version" and
// "mfer version" produce identical output (issue #64).
func TestVersionFlagAndCommandMatch(t *testing.T) {
t.Parallel()
flagOpts := testOpts([]string{testApp, testFlagVersion}, afero.NewMemMapFs())
require.Equal(t, 0, runCLI(flagOpts))
cmdOpts := testOpts([]string{testApp, cmdVersion}, afero.NewMemMapFs())
require.Equal(t, 0, runCLI(cmdOpts))
assert.Equal(t, testStdout(t, flagOpts), testStdout(t, cmdOpts))
}
// TestRootVerbosityFlags asserts that -q and -v given before any subcommand
// name take effect: in the root itself, and in the fetch and export
// subcommands (issue #64). It does not run in parallel: other tests log
// outside runMu (manifest loads, scans), and a line logged while one of these
// runs is in progress lands in its output.
//
//nolint:paralleltest // see above
func TestRootVerbosityFlags(t *testing.T) {
t.Run("quiet with no subcommand hides the banner", func(t *testing.T) {
loud := testOpts([]string{testApp}, afero.NewMemMapFs())
require.Equal(t, 0, runCLI(loud))
assert.Contains(t, testStdout(t, loud), banner)
quiet := testOpts([]string{testApp, "-q"}, afero.NewMemMapFs())
require.Equal(t, 0, runCLI(quiet))
assert.Contains(t, testStdout(t, quiet), cmdGenerate)
assert.NotContains(t, testStdout(t, quiet), banner)
})
t.Run("quiet before fetch hides the banner", func(t *testing.T) {
opts := testOpts([]string{testApp, "-q", cmdFetch}, afero.NewMemMapFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts), errURLRequired.Error())
assert.NotContains(t, testStdout(t, opts), banner)
})
t.Run("verbose twice before fetch enables debug logging", func(t *testing.T) {
opts := testOpts([]string{testApp, "-v", "-v", cmdFetch}, afero.NewMemMapFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts), "fetchManifestOperation()")
})
t.Run("quiet before export hides the manifest summary", func(t *testing.T) {
fs := afero.NewMemMapFs()
manifest := buildTestManifest(t, map[string][]byte{"a.txt": []byte("a")})
require.NoError(t, afero.WriteFile(fs, testManifest, manifest, 0o644))
loud := testOpts([]string{testApp, cmdExport, testManifest}, fs)
require.Equal(t, 0, runCLI(loud))
assert.Contains(t, testStderr(t, loud), "loaded manifest")
quiet := testOpts([]string{testApp, "-q", cmdExport, testManifest}, fs)
require.Equal(t, 0, runCLI(quiet))
assert.NotContains(t, testStderr(t, quiet), "loaded manifest")
})
}
func TestHelpCommand(t *testing.T) {
t.Parallel()
@@ -126,7 +248,7 @@ func TestHelpCommand(t *testing.T) {
stdout := testStdout(t, opts)
assert.Contains(t, stdout, cmdGenerate)
assert.Contains(t, stdout, cmdCheck)
assert.Contains(t, stdout, "fetch")
assert.Contains(t, stdout, cmdFetch)
}
func TestGenerateCommand(t *testing.T) {
@@ -679,11 +801,13 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
fs := afero.NewMemMapFs()
rng := rand.New(rand.NewSource(42)) //nolint:gosec // deterministic test data
// Create many small files with random names to generate a ~1MB manifest
// Each manifest entry is roughly 50-60 bytes, so we need ~20000 files
// Create many small files with random names so the manifest has many
// entries and random single-byte flips land at varied offsets. Each
// manifest entry is roughly 50-60 bytes. Kept modest so the suite stays
// within its wall-clock budget under -race.
require.NoError(t, fs.MkdirAll(testDir, 0o755))
numFiles := 20000
numFiles := 1500
for range numFiles {
// Generate random filename
filename := fmt.Sprintf("/testdir/%08x%08x%08x.dat",
@@ -699,11 +823,11 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
exitCode := runCLI(opts)
require.Equal(t, 0, exitCode, "generate should succeed")
// Read the valid manifest and verify it's approximately 1MB
// Read the valid manifest and verify it has real size.
validManifest, err := afero.ReadFile(fs, testManifest)
require.NoError(t, err)
require.GreaterOrEqual(t, len(validManifest), 1024*1024,
"manifest should be at least 1MB, got %d bytes", len(validManifest))
require.GreaterOrEqual(t, len(validManifest), 64*1024,
"manifest should be at least 64KB, got %d bytes", len(validManifest))
t.Logf("manifest size: %d bytes (%d files)", len(validManifest), numFiles)
// First corruption: truncate the manifest
@@ -726,8 +850,8 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
exitCode = runCLI(opts)
require.Equal(t, 0, exitCode, "check should pass with valid manifest")
// Now do 500 random corruption iterations
for i := range 500 {
// Now do 100 random corruption iterations
for i := range 100 {
// Corrupt: write a random byte at a random offset
corrupted := make([]byte, len(validManifest))
copy(corrupted, validManifest)
+322 -135
View File
@@ -2,167 +2,354 @@
package cli
import (
"fmt"
"bytes"
"context"
"flag"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"testing"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2"
"sneak.berlin/go/mfer/mfer"
)
// errMsgCase is one pinned user-visible error message.
type errMsgCase struct {
name string
err error
want string
}
// These tests pin the exact rendered text of the CLI's user-visible error
// messages. The messages are grepped for in CI pipelines and quoted in bug
// reports, so a reword is a deliberate change, never a refactoring side
// effect.
//
// Every case drives the real function that emits the message and asserts on
// what it returns. No production format string is restated here: a test that
// only re-rendered a copied format string would keep passing after the real
// message changed, which is exactly the regression these tests exist to
// catch.
// Full 40-hex fingerprints used where a message embeds one.
const (
msgFpA = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
msgFpB = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
)
func checkErrMsgCases(t *testing.T, cases []errMsgCase) {
// runLocked runs fn while holding runMu, so operations that write to the
// process-global logger do not race the other CLI runs.
func runLocked(fn func() error) error {
runMu.Lock()
defer runMu.Unlock()
return fn()
}
// unsignedChecker builds a Checker over a freshly scanned, unsigned manifest.
func unsignedChecker(t *testing.T) *mfer.Checker {
t.Helper()
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
assert.Equal(t, tc.want, tc.err.Error())
})
}
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/d", 0o755))
require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644))
s := mfer.NewScannerWithOptions(&mfer.ScannerOptions{Fs: fs})
require.NoError(t, s.EnumeratePath("/d", nil))
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
require.NoError(t, afero.WriteFile(fs, "/d/index.mf", buf.Bytes(), 0o644))
chk, err := mfer.NewChecker("/d/index.mf", "/d", fs)
require.NoError(t, err)
require.False(t, chk.IsSigned())
return chk
}
// TestErrorMessagesVerbatim pins the exact rendered text of the CLI's
// user-visible error messages.
func TestNoManifestFoundMessage(t *testing.T) {
t.Parallel()
_, err := findManifest(afero.NewMemMapFs(), "/tmp/x")
require.ErrorIs(t, err, errNoManifestFound)
assert.EqualError(t, err,
"no manifest found in /tmp/x (looked for index.mf and .index.mf)")
}
func TestVerifyRequiredSignerMessages(t *testing.T) {
t.Parallel()
t.Run("invalid fingerprint length", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), "12345678")
require.ErrorIs(t, err, errInvalidFingerprint)
assert.EqualError(t, err,
"invalid fingerprint: must be exactly 40 hex characters, got 8")
})
t.Run("manifest not signed", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), msgFpA)
require.ErrorIs(t, err, errManifestNotSigned)
assert.EqualError(t, err,
"manifest is not signed, but signature from "+msgFpA+" is required")
})
}
// TestSignerMismatchMessage drives verifyRequiredSigner against a real signed
// manifest. The embedded fingerprint is whatever the generated key produced,
// so it is read back from the checker and substituted into the expected
// string; the required signer is a fixed value that cannot match it. Requires
// gpg and is skipped where it is absent, as the other signing tests are.
//
// These strings are an interface: they are grepped for in CI pipelines
// and quoted in bug reports. The messages are assembled by wrapping
// static sentinels, and it is easy to change what a user sees while
// only meaning to make an error matchable with errors.Is - which is
// precisely what happened once already. Any change to a string below is
// therefore a deliberate, separately stated change, never a side effect
// of a refactor.
func TestErrorMessagesVerbatim(t *testing.T) {
//nolint:paralleltest // signedChecker calls t.Setenv, which bars t.Parallel
func TestSignerMismatchMessage(t *testing.T) {
chk := signedChecker(t)
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
require.NoError(t, err)
err = verifyRequiredSigner(context.Background(), chk, msgFpB)
require.ErrorIs(t, err, errSignerMismatch)
assert.EqualError(t, err,
"embedded signing key fingerprint "+embeddedFP+
" does not match required "+msgFpB)
}
// signedChecker builds a Checker over a manifest signed by a throwaway GPG
// key generated in a temporary GNUPGHOME.
func signedChecker(t *testing.T) *mfer.Checker {
t.Helper()
_, err := exec.LookPath("gpg")
if err != nil {
t.Skip("gpg not installed, skipping signing test")
}
gpgHome := t.TempDir()
params := "%no-protection\n" +
"Key-Type: RSA\nKey-Length: 2048\n" +
"Name-Real: MFER Test Key\nName-Email: test@mfer.test\n" +
"Expire-Date: 0\n%commit\n"
paramsFile := filepath.Join(gpgHome, "key-params")
require.NoError(t, os.WriteFile(paramsFile, []byte(params), 0o600))
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
cmd := exec.CommandContext(context.Background(), "gpg",
"--batch", "--gen-key", paramsFile)
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
out, err := cmd.CombinedOutput()
if err != nil {
t.Skipf("failed to generate test GPG key: %v: %s", err, out)
}
t.Setenv("GNUPGHOME", gpgHome)
b := mfer.NewBuilder()
b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
content := []byte("signed file")
_, err = b.AddFile("f.txt", mfer.FileSize(len(content)), mfer.ModTime{},
bytes.NewReader(content), nil)
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
chk, err := mfer.NewChecker("/index.mf", "/", fs)
require.NoError(t, err)
require.True(t, chk.IsSigned())
return chk
}
func TestPathDoesNotExistMessage(t *testing.T) {
t.Parallel()
checkErrMsgCases(t, []errMsgCase{
{
name: "check: no manifest found",
err: fmt.Errorf("%w in %s (looked for index.mf and .index.mf)",
errNoManifestFound, "/tmp/x"),
want: "no manifest found in /tmp/x " +
"(looked for index.mf and .index.mf)",
},
{
name: "check: invalid fingerprint length",
err: fmt.Errorf("%w, got %d", errInvalidFingerprint, 8),
want: "invalid fingerprint: must be exactly 40 hex characters, got 8",
},
{
name: "check: manifest not signed",
err: fmt.Errorf("%w, but signature from %s is required",
errManifestNotSigned, msgFpA),
want: "manifest is not signed, but signature from " + msgFpA +
" is required",
},
{
name: "check: signer mismatch",
err: fmt.Errorf("embedded signing key fingerprint %s %w %s",
msgFpA, errSignerMismatch, msgFpB),
want: "embedded signing key fingerprint " + msgFpA +
" does not match required " + msgFpB,
},
{
name: "gen: path does not exist",
err: fmt.Errorf("%w: %s", errPathNotExist, "nope"),
want: "path does not exist: nope",
},
{
name: "gen: output file exists",
err: fmt.Errorf("output file %s %w", "index.mf", errOutputExists),
want: "output file index.mf already exists " +
"(use --force to overwrite)",
},
{
name: "mfer: unknown command",
err: fmt.Errorf("%w %q", errUnknownCommand, "bogus"),
want: `unknown command "bogus"`,
},
set := flag.NewFlagSet("gen", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{"nope"}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil)
_, err := mfa.collectInputPaths(ctx.Args())
require.ErrorIs(t, err, errPathNotExist)
assert.EqualError(t, err, "path does not exist: nope")
}
func TestOutputFileExistsMessage(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/d", 0o755))
require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644))
require.NoError(t, afero.WriteFile(fs, "/out.mf", []byte("old"), 0o644))
set := flag.NewFlagSet("gen", flag.ContinueOnError)
set.String("output", "", "")
set.Bool("force", false, "")
require.NoError(t, set.Parse([]string{"/d"}))
require.NoError(t, set.Set("output", "/out.mf"))
mfa := &CLIApp{Fs: fs}
ctx := urfcli.NewContext(nil, set, nil)
// generateManifestOperation writes to the process-global logger during
// enumeration, so serialize with the other CLI runs.
err := runLocked(func() error { return mfa.generateManifestOperation(ctx) })
require.ErrorIs(t, err, errOutputExists)
assert.EqualError(t, err,
"output file /out.mf already exists (use --force to overwrite)")
}
// TestUnknownCommandMessage drives the root command's action. run only logs
// the error that action returns, so the test lets run build the app with no
// command given and then runs that same app on an unknown command to get the
// error itself.
func TestUnknownCommandMessage(t *testing.T) {
t.Parallel()
mfa := &CLIApp{
appname: testApp,
Stdout: &bytes.Buffer{},
Stderr: &bytes.Buffer{},
Fs: afero.NewMemMapFs(),
}
// run points the process-global logger at this app's output, so
// serialize with the other CLI runs.
err := runLocked(func() error {
mfa.run([]string{testApp})
return mfa.app.Run([]string{testApp, "bogus"})
})
require.ErrorIs(t, err, errUnknownCommand)
assert.EqualError(t, err, `unknown command "bogus"`)
}
func TestManifestLoaderHTTPStatusMessage(t *testing.T) {
t.Parallel()
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer server.Close()
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
_, err := mfa.openManifestReader(server.URL + "/foo.mf")
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err,
"failed to fetch "+server.URL+"/foo.mf: HTTP 404")
}
func TestFetchManifestHTTPStatusMessage(t *testing.T) {
t.Parallel()
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer server.Close()
set := flag.NewFlagSet("fetch", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{server.URL}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil)
// fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) })
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "failed to fetch manifest: HTTP 404")
}
func TestFetchFileHTTPStatusMessage(t *testing.T) {
t.Parallel()
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer server.Close()
err := downloadFile(context.Background(), server.URL+"/x", "x",
&mfer.MFFilePath{}, nil)
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "HTTP 500")
}
func TestURLRequiredMessage(t *testing.T) {
t.Parallel()
set := flag.NewFlagSet("fetch", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil)
// fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) })
require.ErrorIs(t, err, errURLRequired)
assert.EqualError(t, err, "URL argument required")
}
func TestSanitizePathMessages(t *testing.T) {
t.Parallel()
t.Run("empty", func(t *testing.T) {
t.Parallel()
_, err := sanitizePath("")
require.ErrorIs(t, err, errEmptyPath)
assert.EqualError(t, err, "empty path")
})
t.Run("absolute", func(t *testing.T) {
t.Parallel()
_, err := sanitizePath("/etc/passwd")
require.ErrorIs(t, err, errAbsolutePath)
assert.EqualError(t, err, "absolute path not allowed: /etc/passwd")
})
t.Run("traversal", func(t *testing.T) {
t.Parallel()
_, err := sanitizePath("../x")
require.ErrorIs(t, err, errPathTraversal)
assert.EqualError(t, err, "path traversal not allowed: ../x")
})
}
// TestFetchErrorMessagesVerbatim pins the fetch and manifest-loader
// messages; see TestErrorMessagesVerbatim for why.
func TestFetchErrorMessagesVerbatim(t *testing.T) {
func TestSizeMismatchMessage(t *testing.T) {
t.Parallel()
checkErrMsgCases(t, []errMsgCase{
{
name: "manifest_loader: http status",
err: fmt.Errorf("failed to fetch %s: %w %d",
"https://example.com/index.mf", errHTTPStatus, 404),
want: "failed to fetch https://example.com/index.mf: HTTP 404",
},
{
name: "fetch: manifest http status",
err: fmt.Errorf("failed to fetch manifest: %w %d",
errHTTPStatus, 404),
want: "failed to fetch manifest: HTTP 404",
},
{
name: "fetch: file http status",
err: fmt.Errorf("%w %d", errHTTPStatus, 500),
want: "HTTP 500",
},
{
name: "fetch: empty path",
err: errEmptyPath,
want: "empty path",
},
{
name: "fetch: absolute path",
err: fmt.Errorf("%w: %s", errAbsolutePath, "/etc/passwd"),
want: "absolute path not allowed: /etc/passwd",
},
{
name: "fetch: path traversal",
err: fmt.Errorf("%w: %s", errPathTraversal, "../x"),
want: "path traversal not allowed: ../x",
},
{
name: "fetch: size mismatch",
err: fmt.Errorf("%w: expected %d bytes, got %d",
errSizeMismatch, 10, 9),
want: "size mismatch: expected 10 bytes, got 9",
},
{
name: "fetch: url required",
err: errURLRequired,
want: "URL argument required",
},
{
name: "fetch: hash mismatch",
err: errHashMismatch,
want: "hash mismatch",
},
})
// finishDownload returns the size-mismatch error before it touches the
// paths, digest, or entry, so those can be zero here.
err := finishDownload("", "", 9, 10, nil, nil, nil, nil)
require.ErrorIs(t, err, errSizeMismatch)
assert.EqualError(t, err, "size mismatch: expected 10 bytes, got 9")
}
// TestSentinelsAreMatchable checks that the wrapped forms of the
// messages above remain matchable with errors.Is, which is the reason
// the sentinels exist at all.
func TestSentinelsAreMatchable(t *testing.T) {
func TestHashMismatchMessage(t *testing.T) {
t.Parallel()
wrapped := fmt.Errorf("embedded signing key fingerprint %s %w %s",
"a", errSignerMismatch, "b")
require.ErrorIs(t, wrapped, errSignerMismatch)
wrapped = fmt.Errorf("output file %s %w", "index.mf", errOutputExists)
require.ErrorIs(t, wrapped, errOutputExists)
wrapped = fmt.Errorf("failed to fetch manifest: %w %d", errHTTPStatus, 404)
require.ErrorIs(t, wrapped, errHTTPStatus)
assert.NotErrorIs(t, errHashMismatch, errSizeMismatch)
// A 32-byte digest that matches none of the (empty) manifest hashes.
err := verifyDownloadedHash(make([]byte, 32), &mfer.MFFilePath{})
require.ErrorIs(t, err, errHashMismatch)
require.NotErrorIs(t, err, errSizeMismatch)
assert.EqualError(t, err, "hash mismatch")
}
+67 -10
View File
@@ -36,6 +36,11 @@ const (
// traversal bit for group and other must stay set.
dirPerms os.FileMode = 0o755
// filePerms is the permission mode, before the umask, for downloaded
// files. It is the mode os.Create uses; like dirPerms, it keeps group
// and other read access.
filePerms os.FileMode = 0o666
// Bitrate unit thresholds in bits per second.
bpsPerGbps = 1e9
bpsPerMbps = 1e6
@@ -53,6 +58,9 @@ var (
// errPathTraversal indicates a manifest path escaping the target
// directory.
errPathTraversal = errors.New("path traversal not allowed")
// errSymlinkInPath indicates a manifest path running through a
// symlink that already exists in the target directory.
errSymlinkInPath = errors.New("symlink in path not allowed")
// errSizeMismatch indicates a downloaded file with an unexpected
// size.
errSizeMismatch = errors.New("size mismatch")
@@ -274,13 +282,42 @@ func sanitizePath(p string) (string, error) {
return cleaned, nil
}
// checkNoSymlinks returns an error if any part of the relative path p
// already exists as a symlink. sanitizePath checks p only as text, so
// without this a symlink inside the target directory could send a write
// to p outside of it. Parts that do not exist yet are fine: fetch creates
// them as plain directories and files. Call it immediately before each
// write: a symlink created after it returns is not caught.
func checkNoSymlinks(p string) error {
current := ""
for _, part := range strings.Split(p, string(filepath.Separator)) {
current = filepath.Join(current, part)
info, err := os.Lstat(current)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return fmt.Errorf("failed to check %s for a symlink: %w", current, err)
}
if info.Mode()&os.ModeSymlink != 0 {
return fmt.Errorf("%w: %s", errSymlinkInPath, current)
}
}
return nil
}
// resolveManifestURL takes a URL and returns the manifest URL.
// If the URL already ends with .mf, it's returned as-is.
// Otherwise, index.mf is appended.
func resolveManifestURL(inputURL string) (string, error) {
parsed, err := url.Parse(inputURL)
if err != nil {
return "", fmt.Errorf("failed to parse URL: %w", err)
return "", err
}
// Check if URL already ends with .mf
@@ -419,7 +456,12 @@ func downloadFile(
// Create parent directories if needed
dir := filepath.Dir(localPath)
if dir != "" && dir != "." {
err := os.MkdirAll(dir, dirPerms)
err = checkNoSymlinks(dir)
if err != nil {
return err
}
err = os.MkdirAll(dir, dirPerms)
if err != nil {
return fmt.Errorf("failed to create directory %s: %w", dir, err)
}
@@ -447,15 +489,25 @@ func downloadFile(
totalBytes = expectedSize
}
// Create temp file.
err = checkNoSymlinks(tmpPath)
if err != nil {
return err
}
// Remove whatever is at tmpPath, such as a leftover from an
// interrupted run, rather than write into it: it may be a hard link
// to a file outside the target directory, and removing a hard link
// removes only this name. If the removal fails, O_EXCL below makes
// the create fail.
_ = os.Remove(tmpPath)
// Create the temp file only if nothing is at tmpPath (O_EXCL).
//
// G304: tmpPath is derived from localPath, which sanitizePath above
// constrains lexically to a relative path that does not escape the
// destination directory. That is a purely lexical guarantee: it does
// not resolve symlinks, so a pre-existing symlink inside the
// destination tree can still redirect this write outside of it
// (tracked in issue #86).
out, err := os.Create(tmpPath) //nolint:gosec // G304: see comment above
// G304: tmpPath is a relative path that sanitizePath keeps inside the
// target directory as text, and checkNoSymlinks just found no symlink
// in it.
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
tmpPath, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
@@ -519,6 +571,11 @@ func finishDownload(
return err
}
err = checkNoSymlinks(localPath)
if err != nil {
return err
}
// Rename temp file to final path
err = os.Rename(tmpPath, localPath)
if err != nil {
+97
View File
@@ -21,6 +21,10 @@ const (
testFileTxt = "file.txt"
testDirFile = "dir/file.txt"
testIndexMF = "https://example.com/path/index.mf"
// Exactly what url.Parse renders, with no wrapper of our own.
urlParseControlCharErr = `parse "http://example.com/\x7f": ` +
`net/url: invalid control character in URL`
)
func TestEncodeFilePath(t *testing.T) {
@@ -137,6 +141,16 @@ func TestResolveManifestURL(t *testing.T) {
assert.Equal(t, tt.expected, result)
})
}
// The sole caller wraps this error as "invalid URL: %w", so
// resolveManifestURL must return url.Parse's error unadorned.
t.Run("invalid:control character", func(t *testing.T) {
t.Parallel()
_, err := resolveManifestURL("http://example.com/\x7f")
require.ErrorContains(t, err, urlParseControlCharErr)
assert.NotContains(t, err.Error(), "failed to parse URL")
})
}
// scanToManifest scans sourceFs and returns the serialized manifest bytes.
@@ -426,3 +440,86 @@ func TestFetchProgress(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, content, downloaded)
}
// TestFetchRefusesSymlinks runs fetch into a destination directory that
// holds a symlink pointing outside it, in each of the three places fetch
// writes: a parent directory, the temp file, and the file itself, which
// the temp file is renamed onto; and once as a directory inside a plain
// directory. The fetch must fail and nothing outside may change.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchRefusesSymlinks(t *testing.T) {
tests := []struct {
name string
entry string // the manifest's only file
link string // symlink placed in the destination directory
target string // what link points to, relative to the outside directory
}{
{"parent directory", "sub/deeper/file.txt", "sub", "."},
{"directory inside a plain directory", "docs/data/passwd", "docs/data", "."},
{"temp file", testFileTxt, ".file.txt.tmp", "new.txt"},
{"file", testFileTxt, testFileTxt, "new.txt"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
content := []byte("fetched")
sourceFs := afero.NewMemMapFs()
require.NoError(t, sourceFs.MkdirAll(filepath.Dir("/"+tt.entry), 0o755))
require.NoError(t, afero.WriteFile(sourceFs, "/"+tt.entry, content, 0o644))
server := httptest.NewServer(fetchTestHandler(
scanToManifest(t, sourceFs), map[string][]byte{tt.entry: content}))
defer server.Close()
outside := t.TempDir()
chdirTemp(t)
require.NoError(t, os.MkdirAll(filepath.Dir(tt.link), 0o750))
require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), tt.link))
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts), "failed to download "+tt.entry+
": symlink in path not allowed: "+tt.link)
written, err := os.ReadDir(outside)
require.NoError(t, err)
assert.Empty(t, written, "fetch wrote outside the destination")
})
}
}
// TestFetchReplacesHardLinkAtTempName runs fetch into a destination
// directory that holds, at the temp file's name, a hard link to a file
// outside it. To fetch that is an ordinary leftover from an interrupted
// earlier run: it must replace it and succeed, and the outside file must
// not change.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchReplacesHardLinkAtTempName(t *testing.T) {
content := []byte("fetched")
sourceFs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(sourceFs, "/"+testFileTxt, content, 0o644))
server := httptest.NewServer(fetchTestHandler(
scanToManifest(t, sourceFs), map[string][]byte{testFileTxt: content}))
defer server.Close()
outsideFile := filepath.Join(t.TempDir(), "secret.txt")
require.NoError(t, os.WriteFile(outsideFile, []byte("outside"), 0o600))
chdirTemp(t)
require.NoError(t, os.Link(outsideFile, ".file.txt.tmp"))
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
fetched, err := os.ReadFile(testFileTxt)
require.NoError(t, err)
assert.Equal(t, content, fetched)
outside, err := os.ReadFile(outsideFile) //nolint:gosec // test-controlled path
require.NoError(t, err)
assert.Equal(t, "outside", string(outside), "fetch wrote outside the destination")
}
+4 -3
View File
@@ -1,6 +1,7 @@
package cli
import (
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -304,7 +305,7 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
// writeFreshenedManifest writes the manifest atomically (write to a
// temp file, then rename over the target).
func writeFreshenedManifest(
afs afero.Fs, builder *mfer.Builder, manifestPath string,
ctx context.Context, afs afero.Fs, builder *mfer.Builder, manifestPath string,
) error {
tmpPath := manifestPath + ".tmp"
@@ -313,7 +314,7 @@ func writeFreshenedManifest(
return fmt.Errorf("failed to create temp file: %w", err)
}
err = builder.Build(outFile)
err = builder.Build(ctx, outFile)
_ = outFile.Close()
if err != nil {
@@ -530,7 +531,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
}
// Write updated manifest atomically (write to temp, then rename)
err = writeFreshenedManifest(mfa.Fs, hasher.builder, manifestPath)
err = writeFreshenedManifest(ctx.Context, mfa.Fs, hasher.builder, manifestPath)
if err != nil {
return err
}
+4 -1
View File
@@ -110,7 +110,10 @@ func TestFreshenRecordEntryMtimePresence(t *testing.T) {
const relPath = "file1.txt"
mtime := time.Unix(1_700_000_000, 0)
// The scanned file's mtime is the Unix epoch. If recordEntry ever misreads
// an absent manifest mtime as the epoch, the "absent" case below would
// compare equal to this and be classified unchanged, so the test fails.
mtime := time.Unix(0, 0)
info := stubFileInfo{size: 8, mtime: mtime}
for _, tc := range []struct {
+48 -7
View File
@@ -18,6 +18,8 @@ const (
cmdGenerate = "generate"
cmdCheck = "check"
cmdExport = "export"
cmdFetch = "fetch"
cmdVersion = "version"
flagProgress = "progress"
@@ -67,6 +69,12 @@ func (mfa *CLIApp) VersionString() string {
return mfer.Version
}
// printVersion writes the version line shared by the --version flag and the
// version subcommand, so both produce identical output.
func (mfa *CLIApp) printVersion() {
_, _ = fmt.Fprintf(mfa.Stdout, "%s version %s\n", mfa.appname, mfa.VersionString())
}
func (mfa *CLIApp) printBanner() {
if log.GetLevel() <= log.InfoLevel {
_, _ = fmt.Fprintln(mfa.Stdout, banner)
@@ -77,20 +85,34 @@ func (mfa *CLIApp) printBanner() {
}
}
// setVerbosity sets the log level from -v and -q, given before the subcommand
// name (the root's copies), after it (the subcommand's own copies), or both.
// urfave/cli reads a flag from the nearest command that defines it, so each
// command in the lineage is asked. The highest -v count wins rather than the
// sum, because a subcommand without its own copies reads the root's.
func (mfa *CLIApp) setVerbosity(c *cli.Context) {
_, present := os.LookupEnv("MFER_DEBUG")
verbosity := 0
quiet := false
for _, ctx := range c.Lineage() {
verbosity = max(verbosity, ctx.Count("verbose"))
quiet = quiet || ctx.Bool("quiet")
}
switch {
case present:
log.EnableDebugLogging()
case c.Bool("quiet"):
case quiet:
log.SetLevel(log.ErrorLevel)
default:
log.SetLevelFromVerbosity(c.Count("verbose"))
log.SetLevelFromVerbosity(verbosity)
}
}
// commonFlags returns the flags shared by most commands (-v, -q)
// commonFlags returns the -v and -q flags taken by the root and by the
// generate, check, freshen and fetch subcommands.
func commonFlags() []cli.Flag {
return []cli.Flag{
&cli.BoolFlag{
@@ -258,6 +280,8 @@ func (mfa *CLIApp) exportCommand() *cli.Command {
Usage: "Export manifest contents as JSON",
ArgsUsage: "[manifest file or URL]",
Action: func(c *cli.Context) error {
mfa.setVerbosity(c)
return mfa.exportManifestOperation(c)
},
}
@@ -265,10 +289,10 @@ func (mfa *CLIApp) exportCommand() *cli.Command {
func (mfa *CLIApp) versionCommand() *cli.Command {
return &cli.Command{
Name: "version",
Name: cmdVersion,
Usage: "Show version",
Action: func(_ *cli.Context) error {
_, _ = fmt.Fprintln(mfa.Stdout, mfa.VersionString())
mfa.printVersion()
return nil
},
@@ -300,7 +324,7 @@ func (mfa *CLIApp) listCommand() *cli.Command {
func (mfa *CLIApp) fetchCommand() *cli.Command {
return &cli.Command{
Name: "fetch",
Name: cmdFetch,
Usage: "fetch manifest and referenced files",
Action: func(c *cli.Context) error {
mfa.setVerbosity(c)
@@ -324,6 +348,21 @@ func (mfa *CLIApp) run(args []string) {
log.SetOutput(mfa.Stdout, mfa.Stderr)
log.Init()
// -v means verbose, not version. urfave/cli's built-in version flag
// claims -v by default, which made "mfer -v --version" fail to parse and
// gave -v a different meaning at the root than on the generate, check,
// freshen and fetch subcommands, where it means verbose. Verbose is the
// more common meaning of -v in tools that offer both, so -v means verbose
// at the root too and the version flag takes the capital -V.
// VersionFlag and VersionPrinter are urfave/cli package globals; run() is
// serialized in tests, so assigning them here is safe.
cli.VersionFlag = &cli.BoolFlag{
Name: cmdVersion,
Aliases: []string{"V"},
Usage: "print the version",
}
cli.VersionPrinter = func(_ *cli.Context) { mfa.printVersion() }
mfa.app = &cli.App{
Name: mfa.appname,
Usage: "Manifest generator",
@@ -331,11 +370,13 @@ func (mfa *CLIApp) run(args []string) {
EnableBashCompletion: true,
Writer: mfa.Stdout,
ErrWriter: mfa.Stderr,
Flags: commonFlags(),
Action: func(c *cli.Context) error {
if c.Args().Len() > 0 {
return fmt.Errorf("%w %q", errUnknownCommand, c.Args().First())
}
mfa.setVerbosity(c)
mfa.printBanner()
return cli.ShowAppHelp(c)
@@ -357,6 +398,6 @@ func (mfa *CLIApp) run(args []string) {
if err != nil {
mfa.exitCode = 1
log.WithError(err).Debugf("exiting")
log.Errorf("%s", err)
}
}
+34 -41
View File
@@ -112,13 +112,16 @@ func DisableStyling() {
}
// Init initializes the logger with the CLI handler and default log level.
//
// It reconfigures the process-global apex/log logger under the write lock so
// the global is never mutated while another goroutine holds the read lock to
// read it in emit. Without this, parallel callers (e.g. the test suite) race
// Init's SetLevel/SetHandler against concurrent log calls.
func Init() {
mu.RLock()
mu.Lock()
defer mu.Unlock()
w := stderr
mu.RUnlock()
log.SetHandler(acli.New(w))
log.SetHandler(acli.New(stderr))
log.SetLevel(log.DebugLevel) // Let apex/log pass everything; we filter ourselves
}
@@ -130,74 +133,66 @@ func isEnabled(l Level) bool {
return l >= currentLevel
}
// emit calls fn while holding the read lock if messages at level l are
// enabled. Holding the read lock across the apex/log call keeps the global
// logger from being read while Init reconfigures it under the write lock.
func emit(l Level, fn func()) {
mu.RLock()
defer mu.RUnlock()
if l >= currentLevel {
fn()
}
}
// Fatalf logs a formatted message at fatal level.
func Fatalf(format string, args ...any) {
if isEnabled(FatalLevel) {
log.Fatalf(format, args...)
}
emit(FatalLevel, func() { log.Fatalf(format, args...) })
}
// Fatal logs a message at fatal level.
func Fatal(arg string) {
if isEnabled(FatalLevel) {
log.Fatal(arg)
}
emit(FatalLevel, func() { log.Fatal(arg) })
}
// Errorf logs a formatted message at error level.
func Errorf(format string, args ...any) {
if isEnabled(ErrorLevel) {
log.Errorf(format, args...)
}
emit(ErrorLevel, func() { log.Errorf(format, args...) })
}
// Error logs a message at error level.
func Error(arg string) {
if isEnabled(ErrorLevel) {
log.Error(arg)
}
emit(ErrorLevel, func() { log.Error(arg) })
}
// Warnf logs a formatted message at warn level.
func Warnf(format string, args ...any) {
if isEnabled(WarnLevel) {
log.Warnf(format, args...)
}
emit(WarnLevel, func() { log.Warnf(format, args...) })
}
// Warn logs a message at warn level.
func Warn(arg string) {
if isEnabled(WarnLevel) {
log.Warn(arg)
}
emit(WarnLevel, func() { log.Warn(arg) })
}
// Infof logs a formatted message at info level.
func Infof(format string, args ...any) {
if isEnabled(InfoLevel) {
log.Infof(format, args...)
}
emit(InfoLevel, func() { log.Infof(format, args...) })
}
// Info logs a message at info level.
func Info(arg string) {
if isEnabled(InfoLevel) {
log.Info(arg)
}
emit(InfoLevel, func() { log.Info(arg) })
}
// Verbosef logs a formatted message at verbose level.
func Verbosef(format string, args ...any) {
if isEnabled(VerboseLevel) {
log.Infof(format, args...)
}
emit(VerboseLevel, func() { log.Infof(format, args...) })
}
// Verbose logs a message at verbose level.
func Verbose(arg string) {
if isEnabled(VerboseLevel) {
log.Info(arg)
}
emit(VerboseLevel, func() { log.Info(arg) })
}
// Debugf logs a formatted message at debug level with caller location.
@@ -216,7 +211,10 @@ func Debug(arg string) {
// DebugReal logs at debug level with caller info from the specified stack depth.
func DebugReal(arg string, cs int) {
if !isEnabled(DebugLevel) {
mu.RLock()
defer mu.RUnlock()
if DebugLevel < currentLevel {
return
}
@@ -275,11 +273,6 @@ func GetLevel() Level {
return currentLevel
}
// WithError returns a log entry with the error attached.
func WithError(e error) *log.Entry {
return log.Log.WithError(e)
}
// Progressf prints a progress message that overwrites the current line.
// Use ProgressDone() when progress is complete to move to the next line.
func Progressf(format string, args ...any) {
+6 -4
View File
@@ -3,6 +3,7 @@
package mfer
import (
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -281,8 +282,9 @@ func (b *Builder) SetSigningOptions(opts *SigningOptions) {
b.signingOptions = opts
}
// Build finalizes the manifest and writes it to the writer.
func (b *Builder) Build(w io.Writer) error {
// Build finalizes the manifest and writes it to the writer. ctx bounds the
// gpg runs that sign the manifest when signing options are set.
func (b *Builder) Build(ctx context.Context, w io.Writer) error {
b.mu.Lock()
defer b.mu.Unlock()
@@ -308,13 +310,13 @@ func (b *Builder) Build(w io.Writer) error {
}
// Generate outer wrapper
err := m.generateOuter()
err := m.generateOuter(ctx)
if err != nil {
return fmt.Errorf("build: generate outer: %w", err)
}
// Generate final output
err = m.generate()
err = m.generate(ctx)
if err != nil {
return fmt.Errorf("build: generate: %w", err)
}
+34 -8
View File
@@ -3,6 +3,8 @@ package mfer
import (
"bytes"
"context"
"fmt"
"strings"
"testing"
"time"
@@ -113,7 +115,7 @@ func TestBuilderBuild(t *testing.T) {
var buf bytes.Buffer
err = b.Build(&buf)
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Should have magic bytes
@@ -177,7 +179,7 @@ func TestBuilderDeterministicOutput(t *testing.T) {
var buf bytes.Buffer
err := b.Build(&buf)
err := b.Build(context.Background(), &buf)
require.NoError(t, err)
return buf.Bytes()
@@ -325,7 +327,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
}
var buf bytes.Buffer
require.NoError(t, b.Build(&buf))
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -348,6 +350,30 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
}
}
// A manifest whose payload is larger than one zstd block (128 KiB) is
// written as a frame asking for the writer's whole window, zstdWindowSize,
// which is the most the parser accepts.
func TestBuilderBuildRoundTripLargeManifest(t *testing.T) {
t.Parallel()
hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
b := NewBuilder()
for i := range 4000 {
path := RelFilePath(fmt.Sprintf("dir/file-%05d.txt", i))
require.NoError(t, b.AddFileWithHash(path, FileSize(i), ModTime{}, hash))
}
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.Greater(t, m.pbOuter.GetSize(), int64(128<<10))
assert.Len(t, m.Files(), 4000)
}
func TestNewManifestFromReaderInvalidMagic(t *testing.T) {
t.Parallel()
@@ -383,7 +409,7 @@ func TestManifestString(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(&buf))
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -397,7 +423,7 @@ func TestBuilderBuildEmpty(t *testing.T) {
var buf bytes.Buffer
err := b.Build(&buf)
err := b.Build(context.Background(), &buf)
require.NoError(t, err)
// Should still produce valid manifest with 0 files
@@ -416,7 +442,7 @@ func TestBuilderOmitsCreatedAtByDefault(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(&buf))
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -438,7 +464,7 @@ func TestBuilderIncludesCreatedAtWhenRequested(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(&buf))
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -464,7 +490,7 @@ func TestBuilderDeterministicFileOrder(t *testing.T) {
}
var buf bytes.Buffer
require.NoError(t, b.Build(&buf))
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
+6 -2
View File
@@ -164,12 +164,12 @@ func (c *Checker) SigningPubKey() []byte {
// ExtractEmbeddedSigningKeyFP imports the manifest's embedded public key into a
// temporary keyring and extracts its fingerprint. This validates the key and
// returns its actual fingerprint from the key material itself.
func (c *Checker) ExtractEmbeddedSigningKeyFP() (string, error) {
func (c *Checker) ExtractEmbeddedSigningKeyFP(ctx context.Context) (string, error) {
if len(c.signingPubKey) == 0 {
return "", errNoSigningPubKey
}
return gpgExtractPubKeyFingerprint(c.signingPubKey)
return gpgExtractPubKeyFingerprint(ctx, c.signingPubKey)
}
// Check verifies all files against the manifest.
@@ -312,6 +312,10 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
}
func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
// entry.GetPath() is safe to join here: a manifest's entry paths are
// validated against the path invariants when it is loaded (see
// deserializeInner) or built (see Builder.AddFile), so a traversal or
// absolute path can never reach this point.
absPath := filepath.Join(string(c.basePath), entry.GetPath())
relPath := RelFilePath(entry.GetPath())
+1 -1
View File
@@ -61,7 +61,7 @@ func createTestManifest(
}
var buf bytes.Buffer
require.NoError(t, builder.Build(&buf))
require.NoError(t, builder.Build(context.Background(), &buf))
require.NoError(t, afero.WriteFile(fs, manifestPath, buf.Bytes(), 0o644))
}
+3
View File
@@ -12,6 +12,9 @@ const (
// memory.
MaxDecompressedSize int64 = 256 * 1024 * 1024
// zstdWindowSize is the zstd window zstd.SpeedBestCompression gives mfer's writer.
zstdWindowSize = 8 << 20
// uuidLength is the length in bytes of a binary UUID.
uuidLength = 16
)
+29 -1
View File
@@ -2,6 +2,7 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -25,6 +26,7 @@ var (
errDecompressedTooLarge = errors.New("decompressed data exceeds maximum allowed size")
errUUIDMismatch = errors.New("outer and inner UUID mismatch")
errInvalidFileFormat = errors.New("invalid file format")
errInvalidManifestPath = errors.New("manifest contains invalid path")
)
// validateUUID checks that the byte slice is a valid UUID (16 bytes, parseable).
@@ -91,7 +93,9 @@ func (m *manifest) verifyOuterIntegrity() error {
)
}
// Loading a manifest takes no context; gpgTimeout still bounds gpg.
err = gpgVerify(
context.Background(),
[]byte(sigString),
m.pbOuter.GetSignature(),
m.pbOuter.GetSigningPubKey(),
@@ -110,7 +114,18 @@ func (m *manifest) verifyOuterIntegrity() error {
func (m *manifest) decompressInner() ([]byte, error) {
bb := bytes.NewBuffer(m.pbOuter.GetInnerMessage())
zr, err := zstd.NewReader(bb)
// By default the decoder decodes a payload under 128 KiB in full,
// each frame up to the decoder's limit, before the LimitReader below
// reads any of it. Decoding synchronously and never in full makes it
// decode only what the LimitReader asks for. It also sets aside a new
// buffer for each frame that asks for a larger window than the frames
// before it, even a frame holding no data. Refusing windows above
// zstdWindowSize keeps each buffer to a little over zstdWindowSize, and
// the buffers of frames holding no data to about 16 times it in total.
zr, err := zstd.NewReader(bb,
zstd.WithDecoderConcurrency(1),
zstd.WithDecodeBuffersBelow(0),
zstd.WithDecoderMaxWindow(zstdWindowSize))
if err != nil {
return nil, fmt.Errorf("deserialize: zstd reader: %w", err)
}
@@ -181,6 +196,19 @@ func (m *manifest) deserializeInner() error {
return errUUIDMismatch
}
// Enforce the manifest path invariants on every entry as it is loaded,
// so that no consumer of a manifest — Checker today, any restore or
// extract path tomorrow — acts on a traversal or absolute path from an
// untrusted .mf. Reject loudly on the first offender rather than
// dropping entries, which would let a hostile manifest hide files from a
// check.
for _, f := range m.pbInner.GetFiles() {
err = ValidatePath(f.GetPath())
if err != nil {
return fmt.Errorf("%w: %w", errInvalidManifestPath, err)
}
}
log.Infof("loaded manifest with %d files", len(m.pbInner.GetFiles()))
return nil
+83
View File
@@ -0,0 +1,83 @@
//nolint:testpackage // white-box tests exercise unexported internals
package mfer
import (
"bytes"
"runtime"
"testing"
"google.golang.org/protobuf/proto"
)
// FuzzNewManifestFromReader feeds arbitrary bytes to the manifest parser.
// `make test` runs it on the seed corpus in
// testdata/fuzz/FuzzNewManifestFromReader; `make fuzz` searches for new
// inputs.
//
// For every input the parser must return a manifest or an error, not both
// and not neither, and must not allocate more than a fixed multiple of its
// input and of the decompressed data it may read, plus room for the
// decoder's window buffers. A panic or a hang fails the test on its own.
func FuzzNewManifestFromReader(f *testing.F) {
// A signed manifest makes the parser write the key and signature to a
// temporary directory and run gpg on them. With gpg off the PATH and
// temporary files kept in the test's own directory, no process is
// started and nothing is written elsewhere; such input ends in an
// error instead.
f.Setenv("PATH", "")
f.Setenv("TMPDIR", f.TempDir())
f.Fuzz(func(t *testing.T, data []byte) {
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
m, err := NewManifestFromReader(bytes.NewReader(data))
runtime.ReadMemStats(&after)
if (m == nil) == (err == nil) {
t.Fatalf("got manifest %p and error %v, want exactly one", m, err)
}
// The parser reads at most the declared size plus one byte of
// decompressed data, and never more than MaxDecompressedSize.
decompressed := uint64(MaxDecompressedSize)
outer := new(MFFileOuter)
if validateMagic(data) &&
proto.Unmarshal(data[len(MAGIC):], outer) == nil {
size := outer.GetSize()
if size > 0 && size < MaxDecompressedSize {
decompressed = uint64(size) + 1
}
}
// It also keeps a few copies of its input. Buffers grow by
// copying, so reaching those sizes allocates a few times them in
// total: sixteen times the input and the decompressed data leaves
// room for that.
//
// The decoder also sets aside a new buffer of one to two times the
// window for each frame that asks for a larger window than the
// frames before it, and refuses windows above zstdWindowSize. A
// frame that gives its content size instead of a window has that
// size as its window, refused above zstdWindowSize like any other.
// Frames asking for every window size up to that make it set aside
// about 16 times zstdWindowSize in total; 24 times leaves room.
//
// The seed whose frame claims 8 GiB fails if the decoder sets that
// size aside; the seed whose frames ask for ever larger windows
// fails if the decoder accepts windows of twice zstdWindowSize; the
// seed whose two frames together exceed MaxDecompressedSize fails
// if the decoder decodes them in full instead of stopping at the
// declared size.
limit := 16*(uint64(len(data))+decompressed) + 24*zstdWindowSize
allocated := after.TotalAlloc - before.TotalAlloc
if allocated > limit {
t.Fatalf("allocated %d bytes for %d bytes of input, limit %d",
allocated, len(data), limit)
}
})
}
+146
View File
@@ -0,0 +1,146 @@
//nolint:testpackage // white-box tests exercise unexported internals
package mfer
import (
"bytes"
"context"
"crypto/sha256"
"fmt"
"testing"
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/encoding/protowire"
"google.golang.org/protobuf/proto"
)
// craftInnerBytes builds the wire bytes of an inner MFFile holding a single
// file entry whose path is exactly pathBytes. It writes the wire form by hand
// so a hostile path — including one that is not valid UTF-8 — can be embedded
// without proto.Marshal's own UTF-8 enforcement rejecting it first.
func craftInnerBytes(id uuid.UUID, pathBytes string) []byte {
entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
entry = protowire.AppendString(entry, pathBytes)
inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
inner = protowire.AppendBytes(inner, entry)
inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
return inner
}
// wrapInner wraps inner MFFile wire bytes in a complete, well-formed .mf
// envelope (magic prefix, zstd-compressed payload, matching hash and UUID) so
// that deserialization reaches path validation rather than failing earlier on
// an integrity check.
func wrapInner(t *testing.T, id uuid.UUID, innerData []byte) []byte {
t.Helper()
var cbuf bytes.Buffer
zw, err := zstd.NewWriter(&cbuf, zstd.WithEncoderLevel(zstd.SpeedBestCompression))
require.NoError(t, err)
_, err = zw.Write(innerData)
require.NoError(t, err)
require.NoError(t, zw.Close())
compressed := cbuf.Bytes()
sum := sha256.Sum256(compressed)
outer := &MFFileOuter{
InnerMessage: compressed,
Size: int64(len(innerData)),
Sha256: sum[:],
Uuid: id[:],
Version: MFFileOuter_VERSION_ONE,
CompressionType: MFFileOuter_COMPRESSION_ZSTD,
}
ob, err := proto.Marshal(outer)
require.NoError(t, err)
return append([]byte(MAGIC), ob...)
}
func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
t.Parallel()
tests := []struct {
name string
path string
}{
{"parent traversal", "../escape"},
{"interior traversal", "a/../../escape"},
{"absolute path", "/etc/passwd"},
{"backslash path", `a\b`},
{"double slash", "a//b"},
{"empty path", ""},
{"invalid utf-8", "abc\xff"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
id := uuid.New()
data := wrapInner(t, id, craftInnerBytes(id, tt.path))
_, err := NewManifestFromReader(bytes.NewReader(data))
require.Error(t, err)
if tt.path == "abc\xff" {
// A path that is not valid UTF-8 cannot survive the proto3
// string decoder, which rejects it before path validation
// runs; the manifest is still refused at load time.
return
}
require.ErrorIs(t, err, errInvalidManifestPath)
if tt.path != "" {
// ValidatePath quotes the path with %q; assert against the
// same rendering so escaped characters (e.g. a backslash)
// still match.
assert.Contains(t, err.Error(), fmt.Sprintf("%q", tt.path),
"error must name the offending path")
}
})
}
}
func TestDeserializeValidManifestRoundTrips(t *testing.T) {
t.Parallel()
hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
b := NewBuilder()
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(bytes.NewReader(buf.Bytes()))
require.NoError(t, err)
files := m.Files()
require.Len(t, files, 1)
assert.Equal(t, "dir/file.txt", files[0].GetPath())
assert.Equal(t, int64(123), files[0].GetSize())
}
// TestValidatePathRejectsInvalidUTF8 pins the ValidatePath rule that a manifest
// path must be valid UTF-8, independent of the proto decoder that also enforces
// it on the wire.
func TestValidatePathRejectsInvalidUTF8(t *testing.T) {
t.Parallel()
err := ValidatePath("abc\xff")
require.ErrorIs(t, err, errPathNotUTF8)
assert.Contains(t, err.Error(), "UTF-8")
}
+4 -2
View File
@@ -2,6 +2,7 @@
package mfer
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
@@ -80,6 +81,7 @@ func TestSerializeInternalErrorMessagesVerbatim(t *testing.T) {
t.Parallel()
m := &manifest{}
require.EqualError(t, m.generate(), "internal error: pbInner not set")
require.EqualError(t, m.generateOuter(), "internal error")
require.EqualError(t, m.generate(context.Background()),
"internal error: pbInner not set")
require.EqualError(t, m.generateOuter(context.Background()), "internal error")
}
+48 -15
View File
@@ -10,9 +10,21 @@ import (
"os/exec"
"path/filepath"
"strings"
"time"
)
const (
// gpgTimeout bounds every gpg run, which can otherwise wait forever on
// a passphrase prompt or a stalled gpg-agent. A minute leaves a person
// time to type a passphrase or touch a smartcard.
gpgTimeout = time.Minute
// gpgWaitDelay is how long a gpg run keeps waiting for gpg's stdout
// and stderr to close once gpg has been killed or has exited. Reading
// what gpg itself wrote takes far less; only a process gpg left behind
// holds them open longer.
gpgWaitDelay = time.Second
// privateDirPerms is the permission mode for temporary GPG home
// directories.
privateDirPerms os.FileMode = 0o700
@@ -66,8 +78,20 @@ func gpgArgs(opts []string, positional ...string) []string {
}
// runGPG runs the gpg binary in batch mode with the given arguments and
// optional stdin, returning captured stdout and stderr.
func runGPG(stdin io.Reader, args ...string) (*bytes.Buffer, *bytes.Buffer, error) {
// optional stdin, returning captured stdout and stderr. gpg is killed when
// ctx ends or gpgTimeout passes, whichever comes first.
func runGPG(
ctx context.Context, stdin io.Reader, args ...string,
) (*bytes.Buffer, *bytes.Buffer, error) {
// exec.CommandContext kills only gpg itself. A gpg-agent that gpg
// starts runs detached and holds none of gpg's output, but another
// process gpg leaves behind (a wrapper script that runs the real gpg
// without exec, for example) can keep gpg's stdout or stderr open, and
// Run would wait for it to exit. WaitDelay stops that wait
// gpgWaitDelay after the kill; that process is left running.
ctx, cancel := context.WithTimeout(ctx, gpgTimeout)
defer cancel()
fullArgs := append([]string{"--batch", "--no-tty"}, args...)
// G204: the executable name is a compile-time constant. The arguments
@@ -76,7 +100,8 @@ func runGPG(stdin io.Reader, args ...string) (*bytes.Buffer, *bytes.Buffer, erro
// option or after the "--" end-of-options marker inserted by gpgArgs,
// and therefore cannot be reinterpreted by gpg as an option.
cmd := exec.CommandContext( //nolint:gosec // G204: see comment above
context.Background(), "gpg", fullArgs...)
ctx, "gpg", fullArgs...)
cmd.WaitDelay = gpgWaitDelay
cmd.Stdin = stdin
var stdout, stderr bytes.Buffer
@@ -85,6 +110,14 @@ func runGPG(stdin io.Reader, args ...string) (*bytes.Buffer, *bytes.Buffer, erro
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil && ctx.Err() != nil {
// gpg was killed because ctx ended, which Run reports only as
// "signal: killed"; return the reason instead.
err = ctx.Err()
if errors.Is(err, context.DeadlineExceeded) {
err = fmt.Errorf("gpg timed out: %w", err)
}
}
return &stdout, &stderr, err
}
@@ -105,8 +138,8 @@ func parseFingerprint(colonOutput string) (string, bool) {
// gpgSign creates a detached signature of the data using the specified key.
// Returns the armored detached signature.
func gpgSign(data []byte, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(bytes.NewReader(data),
func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, bytes.NewReader(data),
"--detach-sign",
gpgOptArmor,
"--local-user", string(keyID),
@@ -120,8 +153,8 @@ func gpgSign(data []byte, keyID GPGKeyID) ([]byte, error) {
// gpgExportPublicKey exports the public key for the specified key ID.
// Returns the armored public key.
func gpgExportPublicKey(keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(nil,
func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
gpgArgs([]string{"--export", gpgOptArmor}, string(keyID))...,
)
if err != nil {
@@ -136,8 +169,8 @@ func gpgExportPublicKey(keyID GPGKeyID) ([]byte, error) {
}
// gpgGetKeyFingerprint gets the full fingerprint for a key ID.
func gpgGetKeyFingerprint(keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(nil,
func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
gpgArgs([]string{"--with-colons", "--fingerprint"}, string(keyID))...,
)
if err != nil {
@@ -157,7 +190,7 @@ func gpgGetKeyFingerprint(keyID GPGKeyID) ([]byte, error) {
// gpgExtractPubKeyFingerprint imports a public key into a temporary keyring
// and extracts its fingerprint. This verifies the key is valid and returns
// the actual fingerprint from the key material.
func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, error) {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-fingerprint-*")
if err != nil {
@@ -181,7 +214,7 @@ func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(nil,
_, importStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
@@ -191,7 +224,7 @@ func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
}
// List keys to get fingerprint
listStdout, listStderr, err := runGPG(nil,
listStdout, listStderr, err := runGPG(ctx, nil,
"--homedir", tmpDir,
"--with-colons",
"--fingerprint",
@@ -212,7 +245,7 @@ func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
// gpgVerify verifies a detached signature against data using the provided public key.
// It creates a temporary keyring to import the public key for verification.
func gpgVerify(data, signature, pubKey []byte) error {
func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-verify-*")
if err != nil {
@@ -252,7 +285,7 @@ func gpgVerify(data, signature, pubKey []byte) error {
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(nil,
_, importStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
@@ -262,7 +295,7 @@ func gpgVerify(data, signature, pubKey []byte) error {
}
// Verify the signature
_, verifyStderr, err := runGPG(nil,
_, verifyStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, gpgOptVerify},
sigFile, dataFile)...,
)
+116 -20
View File
@@ -4,11 +4,15 @@ package mfer
import (
"bytes"
"context"
"io"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
@@ -43,8 +47,11 @@ Expire-Date: 0
paramsFile := filepath.Join(gpgHome, "key-params")
require.NoError(t, os.WriteFile(paramsFile, []byte(keyParams), 0o600))
ctx, cancel := context.WithTimeout(context.Background(), gpgTimeout)
defer cancel()
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
cmd := exec.CommandContext(context.Background(), "gpg",
cmd := exec.CommandContext(ctx, "gpg",
"--batch", "--gen-key", paramsFile)
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
@@ -55,7 +62,7 @@ Expire-Date: 0
}
// Get the key fingerprint
cmd = exec.CommandContext(context.Background(), "gpg",
cmd = exec.CommandContext(ctx, "gpg",
"--list-keys", "--with-colons", "test@mfer.test")
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
@@ -90,7 +97,7 @@ func TestGPGSign(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(data, keyID)
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
assert.NotEmpty(t, sig)
assert.Contains(t, string(sig), "-----BEGIN PGP SIGNATURE-----")
@@ -101,7 +108,7 @@ func TestGPGExportPublicKey(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(keyID)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
assert.NotEmpty(t, pubKey)
assert.Contains(t, string(pubKey), "-----BEGIN PGP PUBLIC KEY BLOCK-----")
@@ -112,7 +119,7 @@ func TestGPGGetKeyFingerprint(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
fingerprint, err := gpgGetKeyFingerprint(keyID)
fingerprint, err := gpgGetKeyFingerprint(context.Background(), keyID)
require.NoError(t, err)
assert.NotEmpty(t, fingerprint)
// The fingerprint should be 40 hex chars
@@ -146,12 +153,12 @@ func TestGPGOptionLikeKeyIDIsNotAnOption(t *testing.T) {
_, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(GPGKeyID("--version"))
pubKey, err := gpgExportPublicKey(context.Background(), GPGKeyID("--version"))
require.Error(t, err)
require.ErrorIs(t, err, errGPGKeyNotFound)
assert.NotContains(t, string(pubKey), "gpg (GnuPG)")
fpr, err := gpgGetKeyFingerprint(GPGKeyID("--version"))
fpr, err := gpgGetKeyFingerprint(context.Background(), GPGKeyID("--version"))
require.Error(t, err)
assert.NotContains(t, string(fpr), "gpg (GnuPG)")
}
@@ -162,7 +169,8 @@ func TestGPGSignInvalidKey(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
_, err := gpgSign(data, GPGKeyID("NONEXISTENT_KEY_ID_12345"))
_, err := gpgSign(context.Background(), data,
GPGKeyID("NONEXISTENT_KEY_ID_12345"))
assert.Error(t, err)
}
@@ -185,7 +193,7 @@ func TestBuilderWithSigning(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(&buf)
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are populated
@@ -251,14 +259,14 @@ func TestGPGVerify(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign and verify")
sig, err := gpgSign(data, keyID)
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(keyID)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
// Verify the signature
err = gpgVerify(data, sig, pubKey)
err = gpgVerify(context.Background(), data, sig, pubKey)
require.NoError(t, err)
}
@@ -267,15 +275,15 @@ func TestGPGVerifyInvalidSignature(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(data, keyID)
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(keyID)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
// Try to verify with different data - should fail
wrongData := []byte("different data")
err = gpgVerify(wrongData, sig, pubKey)
err = gpgVerify(context.Background(), wrongData, sig, pubKey)
assert.Error(t, err)
}
@@ -284,12 +292,12 @@ func TestGPGVerifyBadPublicKey(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
sig, err := gpgSign(data, keyID)
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
// Try to verify with invalid public key - should fail
badPubKey := []byte("not a valid public key")
err = gpgVerify(data, sig, badPubKey)
err = gpgVerify(context.Background(), data, sig, badPubKey)
assert.Error(t, err)
}
@@ -312,7 +320,7 @@ func TestManifestSignatureVerification(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(&buf)
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest - signature should be verified during load
@@ -341,7 +349,7 @@ func TestManifestTamperedSignatureFails(t *testing.T) {
var buf bytes.Buffer
err = b.Build(&buf)
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Tamper with the signature by replacing some bytes
@@ -375,7 +383,7 @@ func TestBuilderWithoutSigning(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(&buf)
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are empty
@@ -390,3 +398,91 @@ func TestBuilderWithoutSigning(t *testing.T) {
assert.Empty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be empty when not signing")
}
// fakeGPGPath writes script as an executable named gpg into a temporary
// directory and returns a PATH value with that directory first.
func fakeGPGPath(t *testing.T, script string) string {
t.Helper()
binDir := t.TempDir()
//nolint:gosec // G306: the fake gpg has to be executable
require.NoError(t, os.WriteFile(filepath.Join(binDir, "gpg"),
[]byte(script), 0o700))
return binDir + string(os.PathListSeparator) + os.Getenv("PATH")
}
// TestGPGTimeoutKillsGPG puts a fake gpg that never finishes first on
// PATH and checks that a run past its deadline is killed and reported as
// a timeout of the named operation, instead of hanging.
func TestGPGTimeoutKillsGPG(t *testing.T) {
t.Setenv("PATH", fakeGPGPath(t, "#!/bin/sh\nexec sleep 10\n"))
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
require.ErrorIs(t, err, context.DeadlineExceeded)
assert.Contains(t, err.Error(), "gpg sign failed: gpg timed out")
}
// TestGPGCancelWhenChildHoldsOutput uses a fake gpg that runs sleep as a
// child instead of exec-ing it, the way a wrapper script around the real
// gpg might. Killing the fake gpg leaves sleep holding its stdout and
// stderr open; the call must still return once ctx ends instead of waiting
// for sleep to exit. The fake gpg writes the process ID of sleep to a named
// pipe; the test ends ctx only after reading it, so sleep is running by
// then, and kills sleep before returning.
func TestGPGCancelWhenChildHoldsOutput(t *testing.T) {
pidPipe := filepath.Join(t.TempDir(), "sleep.pid")
require.NoError(t, syscall.Mkfifo(pidPipe, 0o600))
// sleep outlasts the 10 s wait below, so a call that waits for it fails.
t.Setenv("PATH", fakeGPGPath(t,
"#!/bin/sh\nsleep 60 &\necho $! >'"+pidPipe+"'\nwait\n"))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
signErr := make(chan error, 1)
go func() {
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
signErr <- err
}()
pid, err := os.ReadFile(pidPipe) //nolint:gosec // G304: path inside t.TempDir()
require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(pid)))
require.NoError(t, err)
sleep, err := os.FindProcess(n)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, sleep.Kill()) })
cancel()
// The call should return about gpgWaitDelay (one second) after the
// cancel. 10 s is far above that and well under the 30 s test timeout,
// which would abort the whole package before the cleanup kills sleep.
select {
case err := <-signErr:
require.ErrorIs(t, err, context.Canceled)
case <-time.After(10 * time.Second):
t.Fatal("the call waited for the child holding gpg's output to exit")
}
}
// TestBuildPassesContextToSigning checks that a caller can cancel the gpg
// runs that sign a manifest through the context given to Build.
func TestBuildPassesContextToSigning(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{KeyID: "any"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
require.ErrorIs(t, b.Build(ctx, io.Discard), context.Canceled)
}
+1 -2
View File
@@ -283,8 +283,7 @@ func (s *Scanner) ToManifest(
}
// Build and write manifest
//nolint:contextcheck // Build's GPG signing exec is not cancellable by design
return builder.Build(w)
return builder.Build(ctx, w)
}
// configureBuilder constructs a manifest builder configured from the
+4 -31
View File
@@ -304,46 +304,19 @@ func TestScannerEnumerateFS(t *testing.T) {
func TestSendEnumerateStatusNonBlocking(t *testing.T) {
t.Parallel()
// Channel with no buffer - send should not block
// Nobody receives, so a blocking send would hang the test into its timeout.
ch := make(chan EnumerateStatus)
// This should not block
done := make(chan bool)
go func() {
sendEnumerateStatus(ch, EnumerateStatus{FilesFound: 1})
done <- true
}()
select {
case <-done:
// Success - did not block
case <-time.After(100 * time.Millisecond):
t.Fatal("sendEnumerateStatus blocked on full channel")
}
sendEnumerateStatus(ch, EnumerateStatus{FilesFound: 1})
}
func TestSendScanStatusNonBlocking(t *testing.T) {
t.Parallel()
// Channel with no buffer - send should not block
// Nobody receives, so a blocking send would hang the test into its timeout.
ch := make(chan ScanStatus)
done := make(chan bool)
go func() {
sendScanStatus(ch, ScanStatus{ScannedFiles: 1})
done <- true
}()
select {
case <-done:
// Success - did not block
case <-time.After(100 * time.Millisecond):
t.Fatal("sendScanStatus blocked on full channel")
}
sendScanStatus(ch, ScanStatus{ScannedFiles: 1})
}
func TestSendStatusNilChannel(t *testing.T) {
+9 -8
View File
@@ -2,6 +2,7 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -50,13 +51,13 @@ func newTimestampFromTime(t time.Time) *Timestamp {
}
}
func (m *manifest) generate() error {
func (m *manifest) generate(ctx context.Context) error {
if m.pbInner == nil {
return errInnerNotSet
}
if m.pbOuter == nil {
e := m.generateOuter()
e := m.generateOuter(ctx)
if e != nil {
return e
}
@@ -77,7 +78,7 @@ func (m *manifest) generate() error {
return nil
}
func (m *manifest) generateOuter() error {
func (m *manifest) generateOuter(ctx context.Context) error {
if m.pbInner == nil {
return errInternal
}
@@ -135,7 +136,7 @@ func (m *manifest) generateOuter() error {
// Sign the manifest if signing options are provided
if m.signingOptions != nil && m.signingOptions.KeyID != "" {
return m.signOuter()
return m.signOuter(ctx)
}
return nil
@@ -143,27 +144,27 @@ func (m *manifest) generateOuter() error {
// signOuter signs the outer message with the configured GPG key and
// embeds the signature, signer fingerprint, and public key.
func (m *manifest) signOuter() error {
func (m *manifest) signOuter(ctx context.Context) error {
sigString, err := m.signatureString()
if err != nil {
return fmt.Errorf("failed to generate signature string: %w", err)
}
sig, err := gpgSign([]byte(sigString), m.signingOptions.KeyID)
sig, err := gpgSign(ctx, []byte(sigString), m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to sign manifest: %w", err)
}
m.pbOuter.Signature = sig
fingerprint, err := gpgGetKeyFingerprint(m.signingOptions.KeyID)
fingerprint, err := gpgGetKeyFingerprint(ctx, m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to get key fingerprint: %w", err)
}
m.pbOuter.Signer = fingerprint
pubKey, err := gpgExportPublicKey(m.signingOptions.KeyID)
pubKey, err := gpgExportPublicKey(ctx, m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to export public key: %w", err)
}
+2
View File
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFGy[i\x04\xe5O\x82A\x1d\xf4\xb0\xe2z7:U\xee\xa3\xf9\xd6m\xacZ\x9b\xce\x1d\xd9/{@\x1d\xa5y[i\x04\xe5O\x82A\x1d\xf4\xb0\xe2z7:U\xee\xa3\xf9\xd6m\xacZ\x9b\xce\x1d\xd9/{@\x1d\xa5")
+2
View File
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 \xa3\xf7\x97\xa7\xf3\x87:\x90)\\ӊj\xb9\xf7\xfaTJ\x1b\xe7:\xee\xbe\"V\xe0:\x8d(Z\xd6%\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\fc(\xb5/\xfd\x04\x00\xb1\x02\x00\xa0\x06\x01\xaa\x06=\n\x05a.txt\x10\x01\x1a$\n\"\x12 ʗ\x81\x12\xca\x1b\xbd\xca\xfa\xc21\xb3\x9a#\xdcM\xa7\x86\xef\xf8\x14|Nr\xb9\x80w\x85\xaf\xeeH\xbb\xf2\x12\v\b\x80\x92\xb8Ø\xfe\xff\xff\xff\x01\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3s\xeeG\x80")
+2
View File
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 \xa3\xf7\x97\xa7\xf3\x87:\x90)\\ӊj\xb9\xf7\xfaTJ\x1b\xe7:\xee\xbe\"V\xe0:\x8d(Z\xd6%\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\fc(\xb5/\xfd\x04\x00\xb1\x02\x00\xa0\x06\x01\xaa\x06=\n\x05a.txt\x10\x01\x1a$\n\"\x12 ʗ\x81\x12\xca\x1b\xbd\xca\xfa\xc21\xb3\x9a#\xdcM\xa7\x86\xef\xf8\x14|Nr\xb9\x80w\x85\xaf\xeeH\xbb\xf2\x12\v\b\x80\x92\xb8Ø\xfe\xff\xff\xff\x01\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3s\xeeG\x80\xca\f\xe8\x03-----BEGIN PGP SIGNATURE-----\n\niQEzBAABCgAdFiEET1Yr+4Y/3GtRtO6IhypRF2zvI64FAmrBIXAACgkQhypRF2zv\nI67BQAf/QrpX2MjY15YGMGkjR5oIhnx/YV96aGYZyZThzb+l/R/N75iVFVkhX21d\nZhQqdCsORrodTPAXic2g2UGVXP9PhNMh7n6Wm3LsvQYjrRQGrQnqtCkut+3tUt8K\n7pt4OAnnwRSieaVImA1COmzxIrQQKNOs6UkgmAstGuPV0XZoeDiSG8TUYJ/vieCn\np5hC0FFXtzfw4NtkxSmkewE0xBxIwFCA/RfSHCGH3m5K+tRz41vMEgGbL1iEp6+V\nuBaoEc4hqCgEt+Af2pA8VHfqeu2vKiwggOpYpaILXZKVqH9+tWHL1EBv9t0vTsYE\n9D57euuR9+kOdngYNPieP1yn5dOSHg==\n=kvgL\n-----END PGP SIGNATURE-----\n\xd2\f(4F562BFB863FDC6B51B4EE88872A51176CEF23AE\xda\f\xb5\a-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQENBGrBIW8BCADESetN5EdxIe7Fafgxl99Yoo5cOexf7wJyYT0wfUYlRaxt3neR\nhir7LOfH4PZWWoDx7qghxCS4+vs7yGypl6JOm7jnJlhn4HneDa2zeIlgGW2TamyE\nua9KPWBQqkFOYmKPmzp+KnL6ncnBLR5mDkNKFyON812KVvteu6Dp/DNk4Meufe44\nWWr49LSFZa9gEbmRCoQGKby9F0H0yIi4FAc74VdQudy0+fMKcfkKjEvByMzlbBEK\n92Hq3sRFzWd3kvPliNjZTmlh5n5m9aBhMpoy3GkKy8gpDdFc6NLA9iAJe7oNMriR\nkVoa5EjQL1xCXAiAWTYA9NScFfU/574sCTxZABEBAAG0Hk1GRVIgVGVzdCBLZXkg\nPHRlc3RAbWZlci50ZXN0PokBTwQTAQoAORYhBE9WK/uGP9xrUbTuiIcqURds7yOu\nBQJqwSFvAxsvBAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRCHKlEXbO8jrjml\nCAC8wUK9wmvxq0+NZUpFyP+P29klLZYzBDaBrLPJFs0GjnG4kvfUAktWx0Ro80F7\ncjTJ4f44XjDj4glvSjbe2VaDnZl9FTfzUfG+xjD4462NgntQ4fHk/uG4F6d1ikWx\nkEoMpIn1PlSMas1jTQSGlxUr+zFwWuUbGq4n6hRxEnwLlwJlwQt/Aw1vPDYuPDE3\nOYDhJIAJyP+6e9W8ToaAG9byg/22KA1u1qxnNQqsx5Tped2VltAzdYub+yeCuNc8\nIUo5ILo/fQq3GM5sUEaHjPolv88WlDm3vcdbSbDoh5m2inDtg5zuUKJwu32UGu8l\nKoTjp4nxgQGy5WmeBzs4Hdm/\n=TCB8\n-----END PGP PUBLIC KEY BLOCK-----\n")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06!\xc2\x06 \x91\x90*\xa5>\fݐ \x87\xbeaL\xc1\x05?\x0eR\xc18\xa4eՕ\xa95\xb9KʺoZ\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\f-(\xb5/\xfd\x04\x00\x01\x01\x00\xa0\x06\x01\xaa\x06\a\n\x05a.txt\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3a[k'")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06\x1f\xc2\x06 \x91\x90*\xa5>\fݐ \x87\xbeaL\xc1\x05?\x0eR\xc18\xa4eՕ\xa95\xb9KʺoZ\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\f-(\xb5/\xfd\x04\x00\x01\x01\x00\xa0\x06\x01\xaa\x06\a\n\x05a.txt\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3a[k'")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 \xa3\xf7\x97\xa7\xf3\x87:\x90)\\ӊj\xb9\xf7\xfaTJ\x1b\xe7:\xee\xbe\"V\xe0:\x8d(Z\xd6%\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\fc(\xb5/\xfd\x04\x00\xb1\x02\x00\xa0\x06\x01\xaa\x06=\n\x05a.txt\x10\x01\x1a$\n\"\x12 ʗ\x81\x12\xca\x1b\xbd\xca\xfa\xc21\xb3\x9a#\xdcM\xa7\x86\xef\xf8\x14|Nr\xb9\x80w\x85\xaf\xeeH\xbb\xf2\x12\v\b\x80\x92\xb8Ø\xfe\xff\xff\xff\x01\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3s\xeeG")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 ")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAV")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 \xa3\xf7\x97\xa7\xf3\x87:\x90)\\ӊj\xb9\xf7\xfaTJ\x1b\xe7:\xee\xbe\"V\xe0:\x8d(Z\xd6%\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\fc(\xb5/\xfd\x04\x00\xb1\x02\x00\xa0\x06\x01")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFX\xa8\x06\x01\xb0\x06\x01\xb8\x06V\xc2\x06 \xa3\xf7\x97\xa7\xf3\x87:\x90)\\ӊj\xb9\xf7\xfaTJ\x1b\xe7:\xee\xbe\"V\xe0:\x8d(Z\xd6%\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\fc(\xb5/\xfd\x04\x00\xb1\x02\x00\xa0\x06\x01\xaa\x06=\n\x05a.txt\x10\x01\x1a$\n\"\x12 ʗ\x81\x12\xca\x1b\xbd\xca\xfa\xc21\xb3\x9a#\xdcM\xa7\x86\xef\xf8\x14|Nr\xb9\x80w\x85\xaf\xeeH\xbb\xf2\x12\v\b\x80\x92\xb8Ø\xfe\xff\xff\xff\x01\xb2\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3s\xeeG\x80")
File diff suppressed because one or more lines are too long
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06\x01\xc2\x06 \xd6aQ\xa4\x85\xc0+\xbb\xca\x11\x11\a<\x019\x97\xb3\xbb3\xd0 \xd5U\xfa!\xaeAf<N@\x9d\xca\x06\x10\x93\x85\vpu\x85\xe4\x04\xe4\x95\x1a=\xdc\x1f\x05\xa3\xba\f\x12(\xb5/\xfd\xc0\x00\x00\x00\x00\x00\x02\x00\x00\x00\v\x00\x00\x00")
@@ -0,0 +1,2 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06\x01\xc2\x06 \xc8g?\xa0\xc9\xc5]\xb57M\xe5O#\x04\xb2\x12\xc6)@=\xd2\xf3f/͉~\x10\x15\xfbkD\xca\x06\x10o\x1c*N;]L~\x9a\x8b\x1d.?@Qb\xba\f\x89\t(\xb5/\xfd\x00\x00\x01\x00\x00(\xb5/\xfd\x00\x01\x01\x00\x00(\xb5/\xfd\x00\x02\x01\x00\x00(\xb5/\xfd\x00\x03\x01\x00\x00(\xb5/\xfd\x00\x04\x01\x00\x00(\xb5/\xfd\x00\x05\x01\x00\x00(\xb5/\xfd\x00\x06\x01\x00\x00(\xb5/\xfd\x00\a\x01\x00\x00(\xb5/\xfd\x00\b\x01\x00\x00(\xb5/\xfd\x00\t\x01\x00\x00(\xb5/\xfd\x00\n\x01\x00\x00(\xb5/\xfd\x00\v\x01\x00\x00(\xb5/\xfd\x00\f\x01\x00\x00(\xb5/\xfd\x00\r\x01\x00\x00(\xb5/\xfd\x00\x0e\x01\x00\x00(\xb5/\xfd\x00\x0f\x01\x00\x00(\xb5/\xfd\x00\x10\x01\x00\x00(\xb5/\xfd\x00\x11\x01\x00\x00(\xb5/\xfd\x00\x12\x01\x00\x00(\xb5/\xfd\x00\x13\x01\x00\x00(\xb5/\xfd\x00\x14\x01\x00\x00(\xb5/\xfd\x00\x15\x01\x00\x00(\xb5/\xfd\x00\x16\x01\x00\x00(\xb5/\xfd\x00\x17\x01\x00\x00(\xb5/\xfd\x00\x18\x01\x00\x00(\xb5/\xfd\x00\x19\x01\x00\x00(\xb5/\xfd\x00\x1a\x01\x00\x00(\xb5/\xfd\x00\x1b\x01\x00\x00(\xb5/\xfd\x00\x1c\x01\x00\x00(\xb5/\xfd\x00\x1d\x01\x00\x00(\xb5/\xfd\x00\x1e\x01\x00\x00(\xb5/\xfd\x00\x1f\x01\x00\x00(\xb5/\xfd\x00 \x01\x00\x00(\xb5/\xfd\x00!\x01\x00\x00(\xb5/\xfd\x00\"\x01\x00\x00(\xb5/\xfd\x00#\x01\x00\x00(\xb5/\xfd\x00$\x01\x00\x00(\xb5/\xfd\x00%\x01\x00\x00(\xb5/\xfd\x00&\x01\x00\x00(\xb5/\xfd\x00'\x01\x00\x00(\xb5/\xfd\x00(\x01\x00\x00(\xb5/\xfd\x00)\x01\x00\x00(\xb5/\xfd\x00*\x01\x00\x00(\xb5/\xfd\x00+\x01\x00\x00(\xb5/\xfd\x00,\x01\x00\x00(\xb5/\xfd\x00-\x01\x00\x00(\xb5/\xfd\x00.\x01\x00\x00(\xb5/\xfd\x00/\x01\x00\x00(\xb5/\xfd\x000\x01\x00\x00(\xb5/\xfd\x001\x01\x00\x00(\xb5/\xfd\x002\x01\x00\x00(\xb5/\xfd\x003\x01\x00\x00(\xb5/\xfd\x004\x01\x00\x00(\xb5/\xfd\x005\x01\x00\x00(\xb5/\xfd\x006\x01\x00\x00(\xb5/\xfd\x007\x01\x00\x00(\xb5/\xfd\x008\x01\x00\x00(\xb5/\xfd\x009\x01\x00\x00(\xb5/\xfd\x00:\x01\x00\x00(\xb5/\xfd\x00;\x01\x00\x00(\xb5/\xfd\x00<\x01\x00\x00(\xb5/\xfd\x00=\x01\x00\x00(\xb5/\xfd\x00>\x01\x00\x00(\xb5/\xfd\x00?\x01\x00\x00(\xb5/\xfd\x00@\x01\x00\x00(\xb5/\xfd\x00A\x01\x00\x00(\xb5/\xfd\x00B\x01\x00\x00(\xb5/\xfd\x00C\x01\x00\x00(\xb5/\xfd\x00D\x01\x00\x00(\xb5/\xfd\x00E\x01\x00\x00(\xb5/\xfd\x00F\x01\x00\x00(\xb5/\xfd\x00G\x01\x00\x00(\xb5/\xfd\x00H\x01\x00\x00(\xb5/\xfd\x00I\x01\x00\x00(\xb5/\xfd\x00J\x01\x00\x00(\xb5/\xfd\x00K\x01\x00\x00(\xb5/\xfd\x00L\x01\x00\x00(\xb5/\xfd\x00M\x01\x00\x00(\xb5/\xfd\x00N\x01\x00\x00(\xb5/\xfd\x00O\x01\x00\x00(\xb5/\xfd\x00P\x01\x00\x00(\xb5/\xfd\x00Q\x01\x00\x00(\xb5/\xfd\x00R\x01\x00\x00(\xb5/\xfd\x00S\x01\x00\x00(\xb5/\xfd\x00T\x01\x00\x00(\xb5/\xfd\x00U\x01\x00\x00(\xb5/\xfd\x00V\x01\x00\x00(\xb5/\xfd\x00W\x01\x00\x00(\xb5/\xfd\x00X\x01\x00\x00(\xb5/\xfd\x00Y\x01\x00\x00(\xb5/\xfd\x00Z\x01\x00\x00(\xb5/\xfd\x00[\x01\x00\x00(\xb5/\xfd\x00\\\x01\x00\x00(\xb5/\xfd\x00]\x01\x00\x00(\xb5/\xfd\x00^\x01\x00\x00(\xb5/\xfd\x00_\x01\x00\x00(\xb5/\xfd\x00`\x01\x00\x00(\xb5/\xfd\x00a\x01\x00\x00(\xb5/\xfd\x00b\x01\x00\x00(\xb5/\xfd\x00c\x01\x00\x00(\xb5/\xfd\x00d\x01\x00\x00(\xb5/\xfd\x00e\x01\x00\x00(\xb5/\xfd\x00f\x01\x00\x00(\xb5/\xfd\x00g\x01\x00\x00(\xb5/\xfd\x00h\x01\x00\x00(\xb5/\xfd\x00i\x01\x00\x00(\xb5/\xfd\x00j\x01\x00\x00(\xb5/\xfd\x00k\x01\x00\x00(\xb5/\xfd\x00l\x01\x00\x00(\xb5/\xfd\x00m\x01\x00\x00(\xb5/\xfd\x00n\x01\x00\x00(\xb5/\xfd\x00o\x01\x00\x00(\xb5/\xfd\x00p\x01\x00\x00(\xb5/\xfd\x00q\x01\x00\x00(\xb5/\xfd\x00r\x01\x00\x00(\xb5/\xfd\x00s\x01\x00\x00(\xb5/\xfd\x00t\x01\x00\x00(\xb5/\xfd\x00u\x01\x00\x00(\xb5/\xfd\x00v\x01\x00\x00(\xb5/\xfd\x00w\x01\x00\x00(\xb5/\xfd\x00x\x01\x00\x00(\xb5/\xfd\x00y\x01\x00\x00(\xb5/\xfd\x00z\x01\x00\x00(\xb5/\xfd\x00{\x01\x00\x00(\xb5/\xfd\x00|\x01\x00\x00(\xb5/\xfd\x00}\x01\x00\x00(\xb5/\xfd\x00~\x01\x00\x00(\xb5/\xfd\x00\x7f\x01\x00\x00(\xb5/\xfd\x00\x80\x01\x00\x00")
File diff suppressed because one or more lines are too long
+10
View File
@@ -0,0 +1,10 @@
{
"name": "mfer",
"version": "0.1.0",
"private": true,
"description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.",
"license": "WTFPL",
"devDependencies": {
"prettier": "3.9.6"
}
}
+8 -9
View File
@@ -130,18 +130,17 @@ main() {
if missing make; then pkg_install gnumake make make make; fi
# ---- JS / docs repos ----
# ensure_node
# ensure_yarn
# install_js_deps
# This is a Go repo, but node and yarn are required anyway: prettier
# formats the Markdown and JSON, and script/fmt-check verifies it.
# The version is pinned by package.json/yarn.lock, whose integrity
# hashes --frozen-lockfile enforces.
ensure_node
ensure_yarn
install_js_deps
# ---- Go repos ----
if missing go; then pkg_install go golang go go; fi
# golangci-lint: packaged in nix, brew, and apk. On apt there is no
# package: download a specific release archive from GitHub and
# verify its hash (verify_sha256), never curl | sh.
if missing golangci-lint; then
pkg_install golangci-lint golangci-lint golangci-lint golangci-lint
fi
# No golangci-lint: script/lint runs it in Docker only.
go mod download
# ---- Python repos ----
+15 -5
View File
@@ -1,14 +1,24 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check
# (via make check), so a successful build implies all checks pass.
# Generic: needs no adaptation. The Gitea workflow runs this on push.
# script/cibuild: run the CI build; the Gitea workflow runs this on push.
# It builds the image with the same command as script/docker. --no-cache
# because the checks the final stage depends on are RUN steps, and a
# cached one is a check that did not run.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build .
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
+11 -2
View File
@@ -1,7 +1,8 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# Generic: needs no adaptation.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -9,7 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build -t "$("$SCRIPT_DIR/projectname")" .
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
+4 -5
View File
@@ -2,7 +2,8 @@
# script/fmt: format all files (writes).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
@@ -18,10 +19,8 @@ main() {
cd "$ROOT"
ensure_pb
gofumpt -l -w mfer internal cmd
golangci-lint run --fix
# prettier is best-effort, as in the old Makefile (- prefix)
prettier -w *.json || true
prettier -w *.md || true
# Markdown and JSON, over the same file set script/fmt-check verifies.
"$SCRIPT_DIR/prettier" --write
}
main "$@"
+5 -19
View File
@@ -1,28 +1,14 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
# script/fmt, but fails instead of writing: Go via script/fmt-check-go,
# Markdown and JSON via script/prettier.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
# committed, so this is normally a no-op).
ensure_pb() {
if [ ! -f mfer/mf.pb.go ] ||
[ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then
(cd mfer && go generate .)
fi
}
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
cd "$ROOT"
ensure_pb
if [ -n "$(gofmt -l .)" ]; then
echo "gofmt: files need formatting:" >&2
gofmt -l . >&2
exit 1
fi
"$SCRIPT_DIR/fmt-check-go"
"$SCRIPT_DIR/prettier" --check
}
main "$@"
+29
View File
@@ -0,0 +1,29 @@
#!/bin/sh
# script/fmt-check-go: check Go formatting (read-only). Split out from
# script/fmt-check so the Docker lint stage, whose image has no node and
# therefore no prettier, can run the Go half on its own.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
# committed, so this is normally a no-op).
ensure_pb() {
if [ ! -f mfer/mf.pb.go ] ||
[ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then
(cd mfer && go generate .)
fi
}
main() {
cd "$ROOT"
ensure_pb
if [ -n "$(gofmt -l .)" ]; then
echo "gofmt: files need formatting:" >&2
gofmt -l . >&2
exit 1
fi
}
main "$@"
Executable
+17
View File
@@ -0,0 +1,17 @@
#!/bin/sh
# script/fuzz: fuzz the manifest parser for one minute. Run by hand only:
# script/test already runs the committed seed corpus as ordinary tests,
# and CI never fuzzes. An input that fails is written to
# mfer/testdata/fuzz/FuzzNewManifestFromReader/; once the parser is fixed,
# commit it there as a regression seed.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go test -run '^$' -fuzz '^FuzzNewManifestFromReader$' \
-fuzztime 1m -parallel 2 ./mfer
}
main "$@"
+11 -8
View File
@@ -1,17 +1,20 @@
#!/bin/sh
# script/lint: run the linter.
# script/lint: run golangci-lint, in Docker only. Builds the lint stage of
# the Dockerfile, whose build runs the linter, so a successful build is a
# clean lint. --no-cache because a cached build runs no linter. The image
# is removed afterwards, whatever the outcome.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
golangci-lint run
if [ -n "$(gofmt -l .)" ]; then
echo "gofmt: files need formatting:" >&2
gofmt -l . >&2
exit 1
fi
# Tagged per run, so concurrent runs never remove each other's image.
image="$("$SCRIPT_DIR/projectname")-lint:$$"
# A failed build leaves no image, so there is nothing to remove then.
trap 'docker image rm "$image" >/dev/null 2>&1 || true' EXIT INT TERM
docker build --no-cache --target lint -t "$image" .
}
main "$@"
+69
View File
@@ -0,0 +1,69 @@
#!/bin/sh
# script/prettier: run prettier over this repo's canonical file set.
#
# Takes exactly one mode argument, --write or --check, and applies the
# same patterns in both modes. script/fmt and script/fmt-check both go
# through here, so the set of files that get formatted and the set that
# get verified cannot drift apart.
#
# Failures are never swallowed: a missing prettier is an error, not a
# silent skip. A formatter that quietly does nothing is worse than one
# that fails loudly.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
usage() {
echo "usage: script/prettier --write|--check" >&2
exit 2
}
# Prefer the version pinned by package.json/yarn.lock so that CI and
# developer machines format identically. Fall back to a prettier on PATH,
# but say so, because a different version formats differently.
find_prettier() {
if [ -x "$ROOT/node_modules/.bin/prettier" ]; then
printf '%s\n' "$ROOT/node_modules/.bin/prettier"
return 0
fi
if command -v prettier >/dev/null 2>&1; then
echo "prettier: node_modules/.bin/prettier is absent; using the" \
"prettier on PATH, which may be a different version than the" \
"one pinned in package.json. Run script/bootstrap to install" \
"the pinned version." >&2
command -v prettier
return 0
fi
return 1
}
main() {
[ "$#" -eq 1 ] || usage
case "$1" in
--write | --check) mode="$1" ;;
*) usage ;;
esac
cd "$ROOT"
if ! prettier_bin="$(find_prettier)"; then
echo "prettier: not found." >&2
echo " Install it with: script/bootstrap" >&2
echo " (installs the version pinned in package.json/yarn.lock)" >&2
exit 1
fi
# Markdown and JSON, repo-wide rather than root-only, so files in
# subdirectories (docs/, once it exists) are covered too. Exclusions
# live in .prettierignore; REPO_POLICIES.md is excluded there because
# it is a verbatim copy of an upstream document.
#
# --no-error-on-unmatched-pattern is deliberately NOT used: both
# patterns always match at least one tracked file (README.md,
# package.json), so an empty match means the glob broke, and prettier
# erroring out is exactly what we want rather than a vacuous pass.
"$prettier_bin" "$mode" "**/*.md"
"$prettier_bin" "$mode" "**/*.json"
}
main "$@"
+6 -1
View File
@@ -17,7 +17,12 @@ ensure_pb() {
main() {
cd "$ROOT"
ensure_pb
go test -v --timeout 10s ./...
go test -timeout 30s -race -cover ./... ||
{
echo "--- Rerunning with -v for details ---"
go test -timeout 30s -race -v ./...
exit 1
}
}
main "$@"
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.9.6:
version "3.9.6"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.9.6.tgz#b3ea5146515d40fc53f18aa63f74dfab1e10dbf6"
integrity sha512-OpN0zzVdiaiAhxpuuj5efpIS4sY9j7bY6uR5mnj5yPzGkdkjNKSJeUThPb60Jw29QuAZgA4o+/iB49kFiaBX6g==