sneak/mfer
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions 31 Releases Wiki Activity

Compare commits

base: sneak/mfer:ccd018aef005406b80d5cbafd94505f0f382d88e
Branches Tags
sneak/mfer:main sneak/mfer:next sneak/mfer:issue-116-gomodguard
..
compare: sneak/mfer:next
Branches Tags
sneak/mfer:next sneak/mfer:issue-116-gomodguard sneak/mfer:main
18 Commits
ccd018aef0 .. next
Author SHA1 Message Date
clawbot 0fbb3da0a5 Sign and verify manifests in Go with OpenPGP instead of running gpg (closes #181)
check / check (push) Waiting to run
Details 
mfer ran the gpg binary to sign, export keys and verify, so it failed
wherever gpg is missing. It now uses github.com/ProtonMail/go-crypto/openpgp.
--sign-key and MFER_SIGN_KEY name a file holding one version 4 OpenPGP
secret key; a protected key's passphrase comes from
MFER_SIGN_KEY_PASSPHRASE or a terminal prompt. gen and freshen check that
the key can sign before they read any file. Verification keeps the rules
of the --require-signature fix: one primary key in the embedded block,
counted from its packets, exactly one signature, made by that key or a
subkey, and signer equal to its fingerprint. The embedded block may hold
no DSA key and no secret key, and an armored field must be one
well-formed block.

Model: opus-5-5
 2026-10-08 09:42:45 +02:00
clawbot c23367c216 List a file given to Scanner.EnumeratePath by its name (closes #182)
check / check (push) Waiting to run
Details 
EnumeratePath treated its argument as a directory: given a file, it
listed it under an empty path and ToManifest stopped with "path cannot
be empty". It now hands its one path to EnumeratePaths, which lists a
directory's files by their paths under it and a file by its name, as
EnumerateFile does. EnumeratePath's own resolution of a directory named
through a symlink goes with it, since EnumeratePaths does the same.

Model: opus-5-5
 2026-10-08 07:08:40 +02:00
clawbot a5f218e42a List a directory named through a symlink in gen and freshen (closes #185)
check / check (push) Waiting to run
Details 
The walk in gen and freshen does not follow a symlink at its top. A
directory given as a symlink, or a working directory whose path names
one, was listed as empty and gen exited 0, while freshen removed every
entry from the manifest. Each walk now resolves its starting directory
with filepath.EvalSymlinks first, as check's search for extra files
does, and starts at the path as named if that fails. Symlinks inside the
tree are still skipped unless --follow-symlinks is given.

Model: opus-5-5
 2026-10-08 05:59:02 +02:00
clawbot 6229c4eca0 Write gen DIR's manifest to DIR/index.mf (closes #178)
check / check (push) Waiting to run
Details 
Without --output, gen given one directory now writes index.mf in it,
and given one file writes index.mf beside it; with no path or several,
it still writes index.mf in the current directory. An --output given
with an empty value is refused. What gen lists depends only on its
arguments and the tree, never on where the manifest is written, so gen
DIR followed by check DIR passes.

Scanner.EnumeratePaths lists a file argument by its name, as
EnumerateFile does. Before, it listed the file under an empty path and
gen stopped with "path cannot be empty".

The --output help text and the README's Tool Examples state the default.

Model: opus-5-5
 2026-10-08 03:08:40 +02:00
clawbot e35cd4a045 Resolve check and freshen paths against the manifest's directory (closes #177)
check / check (push) Waiting to run
Details 
Without --base, check and freshen now look for a manifest's files in the
directory that holds it, the file named or the one found in a directory
argument, instead of the current directory. So `mfer check /media/drive`
checks a drive against its own index.mf from anywhere. check of a manifest
given by URL still uses the current directory, and --base still overrides,
even when it names the current directory. The --base help text of both
commands and the README's Tool Examples state the default; the README gains
a freshen entry for this, which also names the hidden files and symlinks
freshen leaves out by default and the flags that include them.

Model: opus-5-5
 2026-10-07 17:28:38 +02:00
clawbot c0b099cc48 Make error message wording consistent (closes #165)
check / check (push) Waiting to run
Details 
Error messages in mfer/ and internal/cli/ are lowercase except names and
acronyms, carry no "failed to" or command-name prefix, and each wrap names
only the operation and thing the wrapped error does not already name, so a
stacked message names what failed once. Wraps around errors that already
name their operation and path (os and afero path errors, url.Error, the
builder's path errors, the gpg helpers' own errors) are dropped. gpg's
stderr is appended to a gpg failure, and to the error for a signing key gpg
did not report, only when gpg wrote some. errHTTPStatus reads "unexpected
HTTP status"; both inner-not-set sentinels read "inner message not set".
No sentinel, errors.Is result or exit status changes.

Model: opus-5-5
 2026-10-07 17:25:41 +02:00
clawbot dce5e050c3 Link docs/FORMAT.md as the format specification in the README (closes #166)
check / check (push) Waiting to run
Details 
The README's opening paragraph called mfer/mf.proto the format specification
and linked to it. It now links docs/FORMAT.md, which is the specification, and
names mfer/mf.proto as the protobuf schema that document refers to for field
numbers and types. The link is relative, as the README's link to
REPO_POLICIES.md is, because docs/FORMAT.md is not on main yet.

Model: opus-5-5
 2026-10-07 15:59:10 +02:00
clawbot 4fe1ff2fe1 Refuse a path listed twice in a manifest (closes #170)
check / check (push) Waiting to run
Details 
gen given arguments whose files share a path, such as gen a b with a.txt
in both, or gen . ., now fails while listing the files, before hashing
any, naming the path and both files. Builder.AddFile and
Builder.AddFileWithHash refuse a path already added. Loading refuses a
manifest that lists a path twice, compared byte for byte; fetch keeps
its own letter-case check. The Path Rules in docs/FORMAT.md say each
path appears at most once. The decode-size test listed one path 1000
times; each entry now has its own path of the same length.

Model: opus-5-5
 2026-10-07 15:28:57 +02:00
clawbot 01ff67a38e Refuse a manifest whose inner message version is not one (closes #169)
check / check (push) Waiting to run
Details 
Loading a manifest checked only the outer message's version, so an
inner message of version 0 or a later version loaded as if it were
version one, although docs/FORMAT.md requires VERSION_ONE in both.
deserializeInner now refuses any other inner version with the same
error as an unknown outer version. Two existing tests built inner
messages with no version and expected them to load; they now write
version one.

Model: opus-5-5
 2026-10-07 15:25:47 +02:00
clawbot f663f4242d Limit how much fetch and check read for a manifest or a file (closes #168)
check / check (push) Canceled after 0s
Details 
NewManifestFromReader reads at most one byte past MaxManifestSize, a new
constant of 258 MiB: the 256 MiB decompressed limit grown by zstd's worst
case of 1/256, plus 1 MiB for the signature, the signing key and the
other outer fields. It refuses a larger manifest. fetch, and check given
a URL, stop downloading a manifest one byte past the same size and report
it as too large; tests lower that size to keep their memory small. fetch
stops reading a file one byte past its listed size, so a longer body ends
in the size mismatch at once instead of filling the disk. docs/FORMAT.md
states the limit and gives the decompressed limit as 256 MiB, the size
the code uses.

Model: opus-5-5
 2026-10-07 14:25:45 +02:00
clawbot 0762a728d4 Compare --require-signature with the key that signed (closes #167)
check / check (push) Canceled after 0s
Details 
check and fetch --require-signature compared the required fingerprint
with the first key in the manifest's embedded public key block, while
gpg accepted a good signature by any key in that block.

Loading a signed manifest now refuses one whose embedded block holds
more than one primary key, counted as gpg reads the block, or whose
signer field is not the primary key fingerprint gpg reports for the
signature. --require-signature compares with the signer field, which
loading has checked. Signing names and embeds the key gpg reports it
signed with, so a key ID matching several keys still writes a manifest
that loads. docs/FORMAT.md states what a verifier checks.

Model: opus-5-5
 2026-10-07 13:59:17 +02:00
clawbot 2a174e3ba2 Raise Go to the latest release, update dependencies, use the standard library uuid, add a vulnerability check (closes #102)
check / check (push) Failing after 3s
Details 
Go 1.27.1 in go.mod and in the Dockerfile's test and build images.
Every module go.mod requires is at its current release; protoc-gen-go
follows protobuf to v1.36.12 and mf.pb.go is regenerated. The standard
library uuid package replaces github.com/google/uuid; FromBytes could
only fail on a length validateUUID already checks, so that call and its
unreachable error are gone. make vulncheck runs govulncheck v1.8.0,
installed with go install at its release commit, in a vulncheck stage
of the Dockerfile; script/check does not run it. The newer go directive
switches on lint checks for strings.SplitSeq and t.Chdir, now used. A
new test pins the bytes of a seeded manifest written by an mfer built
before this change.

Model: opus-5-5
 2026-10-06 20:26:15 +02:00
clawbot 2a270b40c5 Record file mode in the manifest, 0000 unless asked (closes #161)
check / check (push) Failing after 4s
Details 
MFFilePath gains mode (field 304): a file's permission bits, 0777 at
most, or 0000, meaning none recorded. gen and freshen record real modes
only with --include-permissions (ScannerOptions.IncludePermissions); the
builder keeps only mode.Perm(), so setuid, setgid and sticky are never
written. list -l and export show the mode in octal. check reports
MODE_MISMATCH for a recorded mode other than 0000 the file lacks. fetch
refuses a manifest with a mode above 0777 before requesting any file,
sets only the permission bits of each recorded mode on the files it
writes, and downloads again a present file whose mode differs. The decoding-cost bound counts a file
entry at 176 bytes, up from 160.

Model: opus-5-5
 2026-10-06 11:43:18 +02:00
clawbot ce66f7c1c1 Re-vendor the canonical files from sneak/prompts dd4027b (closes #159, closes #116)
check / check (push) Failing after 7s
Details 
Fetches .dockerignore, .editorconfig, the CI workflow, .gitignore,
.golangci.yml, .prettierignore, .prettierrc and REPO_POLICIES.md byte
for byte from sneak/prompts dd4027b; this repo's own entries follow the
canonical text in .dockerignore, .gitignore and .editorconfig. The new
.golangci.yml disables gomodguard. The Dockerfile gets a lint phase on
golangci-lint v2.14.0 and a test phase on the Debian Go image; the
build stage depends on both and stamps the version as the policy shows.
script/test and script/lint build only their phase, and script/cibuild
bootstraps and runs script/check first. bin/tools goes: script/bootstrap
installs gofumpt and protoc-gen-go into bin/ with go install pinned to a
commit.

Model: opus-5-5
 2026-10-06 04:43:18 +02:00
clawbot 343431dd30 Drop atime from the format spec and pin the file entry fields (closes #158)
check / check (push) Successful in 2m16s
Details 
atime left mf.proto earlier and nothing reads or writes it, but
docs/FORMAT.md still narrated its removal and listed it among the
determinism rules. The spec now describes the file entry by its fields
only.

A new test compares the MFFilePath message descriptor, by name and
number, with a list copied from the spec's field table. It does not read
the spec, so changing a field means changing the proto, the spec and
that list together.

Model: opus-5-5
 2026-10-06 03:26:16 +02:00
clawbot 99b3e0e202 fetch refuses a manifest whose paths differ only in letter case (closes #154)
check / check (push) Failing after 2s
Details 
On a case-insensitive filesystem such paths are one name. Of two files,
one replaced the other and fetch exited 0. For a file and a directory
another file is in, fetch stopped partway with a non-zero exit, leaving
a partial tree. For two spellings of one directory, both files landed
in one directory, one under a spelling the manifest does not list, and
check reported that file as not in the manifest.

The existing name-clash check now also records each listed file and
each directory one is in. A listed file at a name already taken, or a
directory spelled differently from one already there, is refused on
every filesystem, before the destination is created. The message names
both paths.

Model: opus-5-5
 2026-10-06 01:43:33 +02:00
clawbot ce024baaed Move the CLI to urfave/cli v3 (closes #110)
check / check (push) Failing after 3s
Details 
Ported per the library's v2-to-v3 migration guide: the app is a root
cli.Command, actions take a context and the command, and flag
environment variables become value sources. -v, -q and the version flag
are local so, as before, only the commands defining them accept them.
Every command stops reading flags at its first argument, as v2 did.
ErrWriter is stdout so usage errors print with their help. The action's
context reaches the manifest download in check, export and list.
testify rises to v1.12.1, which v3 requires; the urfave_cli_no_docs
build tag, which v3 lacks, is dropped. v3 accepts a flag given under two
names, so -v --verbose gives debug output, and the test pinning the
refusal becomes a TestVerboseCount case.

Model: opus-5-5
 2026-10-04 22:02:12 +02:00
clawbot ab72692439 Pin the remaining developer tool installs (closes #68)
check / check (push) Failing after 3s
Details 
gofumpt and protoc-gen-go are tools of a separate Go module in bin/tools,
so `go tool` builds them from source checked against bin/tools/go.sum and
mfer's own module gains no dependencies. script/bootstrap downloads that
module, and unpacks protoc 33.4 into bin/protoc from its release archive
after checking the sha256 it holds for the platform. script/generate runs
that protoc with the pinned plugin, so mfer/mf.go and its go:generate line
go. script/prettier runs only the node_modules prettier, fails when it
differs from the package.json pin, finds the node bootstrap installed
through nvm by the version in .nvmrc, and runs prettier once. Comment and
package.json fixes.

Model: opus-5-5
 2026-10-04 20:48:51 +02:00
66 changed files with 4680 additions and 2125 deletions
Expand all files Collapse all files
.dockerignore
+19 -1
View File
@@ -17,7 +17,17 @@
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
.git/config
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
@@ -41,7 +51,9 @@
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
@@ -61,3 +73,9 @@
# This repo's own host-built binary (make build).
/bin/mfer
# The tools script/bootstrap installs for script/gofumpt and
# script/generate.
/bin/gofumpt
/bin/protoc
/bin/protoc-gen-go
.editorconfig
+3
View File
@@ -10,3 +10,6 @@ insert_final_newline = true
[Makefile]
indent_style = tab
[*.go]
indent_style = tab
.gitea/workflows/check.yml
+1 -1
View File
@@ -4,6 +4,6 @@ jobs:
check:
runs-on: ubuntu-latest
steps:
# actions/checkout v4.2.2, 2026-03-16
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
- run: script/cibuild
.gitignore
+51 -16
View File
@@ -1,28 +1,63 @@
/bin/mfer
/tmp
/node_modules/
# Generated manifest files
/index.mf
# Secrets
.env
.env.*
*.key
*.pem
# OS files
# OS
.DS_Store
Thumbs.db
# Editor files
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Go build artifacts
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# Go build and test artifacts.
*.log
*.out
*.test
# This repo's own binary (make build), and the tools script/bootstrap
# installs into bin/.
/bin/mfer
/bin/gofumpt
/bin/protoc/
/bin/protoc-gen-go
# A manifest generated at the repo root, and local scratch.
/index.mf
/tmp
.golangci.yml
+67 -2
View File
@@ -10,14 +10,21 @@ run:
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
@@ -28,6 +35,64 @@ linters:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
.nvmrc
+1
View File
@@ -0,0 +1 @@
22.17.0
.prettierignore
+1 -13
View File
@@ -1,14 +1,2 @@
# REPO_POLICIES.md is a verbatim copy of an authoritative upstream
# document (sneak/prompts). Local tooling must never rewrite it: any
# reformatting is silent drift from the source of truth.
REPO_POLICIES.md
# User-owned configuration, copied verbatim from upstream. Not matched by
# the current prettier file set, but listed so widening that set can never
# start rewriting it.
.golangci.yml
# Dependencies and build output
node_modules/
vendor/
bin/
yarn.lock
Dockerfile
+53 -53
View File
@@ -1,74 +1,74 @@
# Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
# Lint phase. The linter is invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
# golangci/golangci-lint:v2.14.0, 2026-09-24
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below. The image has no gofumpt
# either; script/gofumpt builds the version it pins with `go run`.
RUN script/gofumpt --check
# The linter directly, not `make lint`: script/lint builds this stage, and
# there is no docker inside this build.
RUN golangci-lint run --config .golangci.yml ./...
# Markdown/JSON format stage — prettier needs node, which the Go images
# do not have. node:22.17.0-bookworm-slim (2026-08-09); ships node
# 22.17.0 and yarn 1.22.22, the versions script/bootstrap pins.
FROM node@sha256:b04ce4ae4e95b522112c2e5c52f781471a5cbc3b594527bcddedee9bc48c03a0 AS mdfmt
WORKDIR /src
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY . .
# No make in this image; call the script entrypoint directly.
RUN script/prettier --check
# Build stage — tests and compilation
# golang:1.23 (2026-03-14)
FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder
# Force BuildKit to run the lint and mdfmt stages by creating stage dependencies
COPY --from=lint /src/go.sum /dev/null
COPY --from=mdfmt /src/go.sum /dev/null
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.27.1, 2026-10-06
FROM golang@sha256:1e93e00a31255c07e9a34c4207f3006e1501730c5323697cee7dfb827fdae44c AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Vulnerability check, built only by script/vulncheck (make vulncheck).
# No stage depends on it, so the image build does not run it.
# golang:1.27.1, 2026-10-06
FROM golang@sha256:1e93e00a31255c07e9a34c4207f3006e1501730c5323697cee7dfb827fdae44c AS vulncheck
# govulncheck v1.8.0, 2026-10-06
RUN go install golang.org/x/vuln/cmd/govulncheck@709015412431dd2b5b28a53c06c70bc02d49074c
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN govulncheck ./...
# Build stage. Nothing is wanted from the lint or test phase; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed. The Debian Go image ships git, which the
# version step below needs.
# golang:1.27.1, 2026-10-06
FROM golang@sha256:1e93e00a31255c07e9a34c4207f3006e1501730c5323697cee7dfb827fdae44c AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
# A build context sent as a tar archive, as upaas sends it, keeps its files'
# owners, and git refuses to read a checkout owned by another user.
RUN git config --system --add safe.directory /src
# The revision `mfer version` prints, stamped into main.Gitrev: the VERSION
# build argument when one is given (script/docker passes one), otherwise
# `git describe --tags --always` of the .git the build context carries: the
# tag on a tagged commit, tag-N-gHASH on a commit after one, the short commit
# when no tag is reachable. git ships in this base image. A context that
# carries .git and still yields no version fails the build.
# The revision `mfer version` prints, stamped into main.Gitrev: the
# VERSION build arg when one is given, otherwise `git describe --tags
# --always` on the .git in the build context. With .git present, a
# version that is still empty, dev or unknown fails the build: git is
# missing or could not read the checkout.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "no version could be derived although the build context carries .git" >&2; \
exit 1; \
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
cd cmd/mfer && \
CGO_ENABLED=0 go build -ldflags "-X main.Gitrev=$version" -o /mfer .
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Gitrev=${VERSION}" \
-o /mfer ./cmd/mfer/
# Fail unless /mfer is statically linked: scratch has no C library to run it.
RUN ldd /mfer 2>&1 | grep -q 'not a dynamic executable'
# Runtime stage, and the last one.
FROM scratch
# scratch has no CA certificates; fetch needs them to verify HTTPS servers.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
Makefile
+4 -1
View File
@@ -1,4 +1,4 @@
.PHONY: bootstrap setup test lint fmt fmt-check check docker hooks build generate fuzz
.PHONY: bootstrap setup test lint fmt fmt-check check docker hooks build generate fuzz vulncheck
# Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
@@ -39,3 +39,6 @@ generate:
fuzz:
@script/fuzz
vulncheck:
@script/vulncheck
README.md
+90 -40
View File
@@ -9,8 +9,9 @@ downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
which is included in the [project repository](https://git.eeqj.de/sneak/mfer).
[in the format specification](docs/FORMAT.md), which refers to the protobuf
schema `mfer/mf.proto` for exact field numbers and types. Both are included in
the [project repository](https://git.eeqj.de/sneak/mfer).
The current version is pre-1.0 and while the repo was published in 2022, there
has not yet been any versioned release. [SemVer](https://semver.org) will be
@@ -23,7 +24,7 @@ javascript library is planned.
# Getting Started
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
`mfer` builds from source with Go 1.27.1 or later. The generated protobuf code
is committed, so no `protoc` toolchain is required:
```sh
@@ -55,9 +56,9 @@ for a single command's options.
# Build Status
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
the formatting, lint and test steps in the `Dockerfile` run on every build. The
`main` branch must always be green.
CI runs `script/cibuild`, which runs `script/bootstrap` and `script/check`, then
builds the Docker image with `--no-cache`, so the lint and test phases in the
`Dockerfile` run on every build. The `main` branch must always be green.
# Entrypoints
@@ -67,52 +68,62 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. We
provide:
- `script/bootstrap` — install all dependencies (Go, Go module download, and
node/yarn plus the prettier version pinned in `package.json`/`yarn.lock`),
idempotently; golangci-lint is not installed, it runs only in Docker
- `script/bootstrap` — install all dependencies, idempotently: Go and the
modules of `go.mod`; node (the version `.nvmrc` names, through nvm when there
is no node on `PATH`) and yarn, plus the prettier version pinned in
`package.json`/`yarn.lock`; `gofumpt` v0.12.0 and `protoc-gen-go` v1.36.12,
installed into `bin/` with `go install`, each pinned to a commit; and `protoc`
33.4, unpacked into `bin/protoc` from its release archive once the archive
matches the sha256 the script holds for this platform. Each of those three is
installed again whenever the one in `bin/` reports another version.
golangci-lint is not installed, it runs only in Docker
- `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (`mfer`); used by other scripts
such as `script/docker`
- `script/test` — run the test suite (`go test`); one test fails when
`mfer/mf.proto` no longer matches the hash `script/generate` recorded
- `script/test` — run the test suite in Docker: builds only the `test` stage of
the `Dockerfile`, whose build runs `go test -race`, uncached so it runs every
time; one test fails when `mfer/mf.proto` no longer matches the hash
`script/generate` recorded
- `script/build` (`make build`) — build the `mfer` binary into `bin/mfer`,
stamped with the revision `mfer version` prints: the output of
`git describe --tags --always --dirty`, as `script/docker` passes it
- `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from
`mfer/mf.proto` and record the hash of that `mfer/mf.proto` in
`mfer/mf.proto.sha256`; the only thing that regenerates the committed
`mfer/mf.pb.go`. It needs the exact versions that wrote the committed file,
and refuses to run with any other: `protoc` 33.4 (unpack
`protoc-33.4-<platform>.zip` from
[its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4)
and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11
(`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which
installs it in `$(go env GOPATH)/bin`; `script/generate` adds that directory
to `PATH`)
`mfer/mf.pb.go`. It runs the `protoc` that `script/bootstrap` unpacks into
`bin/protoc` and the `protoc-gen-go` it installs into `bin/`, refusing any
version of either but the pinned one
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
as ordinary tests
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
every time, then removes the image
the `Dockerfile`, whose build runs the linter, uncached so it runs every time
- `script/vulncheck` (`make vulncheck`) — run `govulncheck` in Docker: builds
only the `vulncheck` stage of the `Dockerfile`, uncached, which reports known
vulnerabilities in the code `mfer` calls, from the Go vulnerability database.
`script/check` does not run it, so an advisory published later never turns the
gate red
- `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and
`script/prettier --write`
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
given mode, `--write` or `--check`, at the one version it pins (built on
demand by `go run`, so nothing installs it); `script/fmt`, `script/fmt-check`
and the Docker lint stage all go through it, so they cannot disagree about Go
formatting
given mode, `--write` or `--check`, with the `bin/gofumpt` that
`script/bootstrap` installs, refusing any version but the pinned one;
`script/fmt` and `script/fmt-check` both go through it, so they cannot
disagree about Go formatting
- `script/prettier` — run prettier over the repository's canonical file set
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
`--check`; the single definition of that file set, so `script/fmt` and
`script/fmt-check` cannot disagree about it
`--check`, with the prettier version `yarn.lock` pins, run by the node on
`PATH` or else the one `script/bootstrap` installed through nvm; the single
definition of that file set, so `script/fmt` and `script/fmt-check` cannot
disagree about it
- `script/fmt-check` — check formatting without writing:
`script/gofumpt --check` plus `script/prettier --check`
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/docker` — build the Docker image tagged with the project name
- `script/cibuild` — CI entrypoint: builds the image with the same command as
`script/docker`, uncached, so the checks in the Dockerfile run every time
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
then builds the image with the same command as `script/docker`, uncached, so
the lint and test phases in the `Dockerfile` run every time
- `script/precommit` — pre-commit checks: `go mod tidy` verification, then
`script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
@@ -246,8 +257,9 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- Should the manifest signature format be GnuPG signatures, or those from
OpenBSD's signify (of which there is a good
[golang implementation](https://github.com/frankbraun/gosignify))? Still open,
as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).
[golang implementation](https://github.com/frankbraun/gosignify))? Settled
under question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82):
OpenPGP signatures, which mfer makes and checks itself without running `gpg`.
- Should the on-disk serialization format be proto3 or json? Settled: it is
proto3, see `docs/FORMAT.md` and `mfer/mf.proto`.
@@ -256,24 +268,62 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- `mfer gen` / `mfer gen .`
- recurses under current directory and writes out an `index.mf`
- records every file's mode as `0000` unless given `--include-permissions`,
which records each file's permission bits (`0777` at most)
- `mfer gen /media/drive`
- writes `/media/drive/index.mf`, listing each file by its path under
`/media/drive`, so `mfer check /media/drive` verifies it. Given a file,
gen writes `index.mf` beside it and lists the file by its name; given
several paths, it writes `index.mf` in the current directory
- `--output` names another file to write instead. What gen lists depends
only on the paths it is given and the files under them, so with the same
`--seed` and an unchanged tree it writes the same bytes wherever the
manifest goes. The file it writes to is never listed
- `mfer check` / `mfer check .`
- verifies checksums of all files in manifest, displaying error and exiting
nonzero if any files are missing or corrupted
nonzero if any files are missing or corrupted, or have permission bits
other than the mode the manifest records, unless that is `0000`
- looks for those files under the base directory: the one `--base` names, or
else the directory holding the manifest, or the current directory for a
manifest given by URL. So `mfer check /media/drive` checks a drive against
the `index.mf` at its root, from any directory
- warns about each file under the base directory that the manifest does not
list, hidden files included; with `--no-extra-files` each one is a failure
instead
- `mfer freshen` / `mfer freshen .`
- rewrites `index.mf` to list the files now under the directory holding it,
or under the one `--base` names, hashing only the files that are new or
changed
- leaves out hidden files unless given `--include-dotfiles`, and symlinks
unless given `--follow-symlinks`, which lists each symlink to a file under
its own name with the contents of the file it points to
- `mfer gen --sign-key key.asc` / `mfer freshen --sign-key key.asc`
- signs the manifest with the OpenPGP secret key in `key.asc`, armored or
binary, as `gpg --export-secret-keys` writes it; `MFER_SIGN_KEY` names the
file too. mfer signs it itself and does not need `gpg`. A file holding
more than one key is refused, and a key held only on a smartcard cannot
sign. The key must be a version 4 key, whose 40-character fingerprint is
what `--require-signature` takes, and not a DSA key. A key that cannot
sign, because it has expired or been revoked or its passphrase is wrong,
stops `gen` and `freshen` before they read any file
- takes a protected key's passphrase from `MFER_SIGN_KEY_PASSPHRASE`, or
else asks for it at the terminal
- `mfer fetch https://example.com/stuff/`
- fetches `/stuff/index.mf` and downloads all files listed in manifest into
the current directory, or the one given with `--dest`, and assures
cryptographic integrity of downloaded files. A file already there with the
size and hash the manifest lists is skipped. Once every file is in place,
the manifest is saved there as `index.mf`, so `mfer check` can verify the
tree later. Each file is downloaded to a temp file beside it, such as
`.a.txt.tmp` for `a.txt`, then moved into place. A manifest is refused
before any file is downloaded if it lists a file where fetch writes
another: at the temp file of a listed file, or at `index.mf` or
`.index.mf.tmp` at the top of the tree. Names are compared in any letter
case.
size, hash and recorded mode the manifest lists is skipped. Once every
file is in place, the manifest is saved there as `index.mf`, so
`mfer check` can verify the tree later. Each file is downloaded to a temp
file beside it, such as `.a.txt.tmp` for `a.txt`, given the mode the
manifest records unless that is `0000`, then moved into place. A manifest
is refused before any file is downloaded if it records a mode above
`0777`, or lists a file where fetch writes another: at another listed file
or a directory one is in, at the temp file of a listed file, or at
`index.mf` or `.index.mf.tmp` at the top of the tree. Names are compared
in any letter case, on every filesystem, since on a case-insensitive one
`A.txt` and `a.txt` are one file; a directory two listed files are in must
be spelled alike in both.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
- as above, but first refuses a manifest not signed by the key with that
fingerprint, as `mfer check --require-signature` does, before downloading
REPO_POLICIES.md
+355 -84
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-07-06
last_modified: 2026-10-04
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -60,17 +60,28 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -89,87 +100,198 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
The standard pattern for a Go repo Dockerfile is:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
RUN golangci-lint run --config .golangci.yml ./...
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage
# Runtime stage, and the last one
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -189,14 +311,21 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile
test:
@@ -209,11 +338,26 @@ style conventions are in separate documents:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
@go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
@@ -239,10 +383,84 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
@@ -258,9 +476,56 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -374,12 +639,14 @@ style conventions are in separate documents:
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
@@ -406,3 +673,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
docs/FORMAT.md
+47 -14
View File
@@ -6,8 +6,11 @@ Version 1.0
An `.mf` file is a binary manifest that describes a directory tree of files,
including their paths, sizes, and cryptographic checksums. It supports optional
GPG signatures for integrity verification and optional timestamps for metadata
preservation.
OpenPGP signatures for integrity verification and optional timestamps and file
permissions for metadata preservation.
Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
reserved or kept for later use.
## File Structure
@@ -33,9 +36,9 @@ The outer message contains:
| `sha256` | 104 | bytes | SHA-256 hash of the **compressed** `innerMessage` (corruption detection) |
| `uuid` | 105 | bytes | Random v4 UUID; must match the inner message UUID |
| `innerMessage` | 199 | bytes | Zstd-compressed serialized `MFFile` message |
| `signature` | 201 | bytes (optional) | GPG signature (ASCII-armored or binary) |
| `signer` | 202 | bytes (optional) | Full GPG key ID of the signer |
| `signingPubKey` | 203 | bytes (optional) | Full GPG signing public key |
| `signature` | 201 | bytes (optional) | OpenPGP detached signature (ASCII-armored or binary) |
| `signer` | 202 | bytes (optional) | Fingerprint of the signing key |
| `signingPubKey` | 203 | bytes (optional) | Full OpenPGP public key of the signing key (ASCII-armored or binary) |
### SHA-256 Hash
@@ -47,11 +50,14 @@ allows verifying data integrity before decompression.
The `innerMessage` field is compressed with
[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must
enforce a decompression size limit to prevent decompression bombs. The reference
implementation limits decompressed size to 256 MB. It writes zstd frames with a
implementation limits decompressed size to 256 MiB. It writes zstd frames with a
window of at most 8 MiB, the largest window the zstd format recommends decoders
support, and refuses frames that ask for a larger one. It also refuses an inner
message whose file entries, hashes, timestamps and MIME types, counted at 160,
112, 64 and 16 bytes each, add up to more than 8 times its size.
message whose file entries, hashes, timestamps and MIME types, counted at 176,
112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
manifest file larger than 258 MiB without reading the rest of it: zstd's worst
case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
for the signature, the signing key and the other outer fields.
## Inner Message (`MFFile`)
@@ -77,9 +83,21 @@ Each file entry contains:
| `mimeType` | 301 | string (optional) | MIME type |
| `mtime` | 302 | Timestamp (optional) | Modification time |
| `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) |
| `mode` | 304 | uint32 | Permission bits (see File Mode) |
Field 304 (`atime`) has been removed from the specification. Access time is
volatile and non-deterministic; it is not useful for integrity verification.
## File Mode
`mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
Writers record `0000` unless whoever creates the manifest asks for permissions.
`0000`, the proto3 default, means no mode was recorded: readers never check or
apply it.
The reference implementation records modes when `gen` or `freshen` is given
`--include-permissions`. `check` fails a file whose permission bits differ from
a recorded mode other than `0000`. `fetch` sets a recorded mode other than
`0000` on each file it writes, and refuses a manifest that records a mode above
`0777` before it requests any file.
## Path Rules
@@ -91,9 +109,12 @@ All `path` values must satisfy these invariants:
- **No parent traversal**: no `..` path segments
- **No empty segments**: no `//` sequences
- **No trailing slash**: paths refer to files, not directories
- **Listed once**: each path appears at most once in a manifest, compared byte
for byte, so `A.txt` and `a.txt` are two paths
Implementations must validate these invariants when reading and writing
manifests. Paths that violate these rules must be rejected.
manifests. Paths that violate these rules must be rejected, and a reader must
reject a manifest that lists a path more than once.
## Hash Format (`MFFileChecksum`)
@@ -121,8 +142,20 @@ Where:
- `<SHA256>` is the hex-encoded SHA-256 hash from the outer message (covering
compressed data)
Components are separated by hyphens. The signature is produced by GPG over this
canonical string and stored in the `signature` field of the outer message.
Components are separated by hyphens. The signature is an OpenPGP detached
signature over this canonical string, stored in the `signature` field of the
outer message. The signing key's public key goes in `signingPubKey` and its
fingerprint, in hex, in `signer`.
A verifier accepts a signed manifest only if `signingPubKey` holds exactly one
primary key, `signature` is one good signature over the canonical string made by
that key (or one of its subkeys), and `signer` is that key's fingerprint. The
reference implementation refuses to load a manifest that fails these checks;
`check` and `fetch` given `--require-signature` then compare the required
fingerprint with `signer`. It also refuses a manifest whose `signingPubKey`
holds a DSA key or subkey, or any secret key or subkey, since checking the
self-signatures of a DSA key or the numbers of a secret key can take hours when
those numbers are very large.
## Deterministic Serialization
@@ -130,7 +163,7 @@ By default, manifests are generated deterministically:
- File entries are sorted by `path` in **lexicographic byte order**
- `createdAt` is omitted unless explicitly requested
- `atime` is never included (field removed from schema)
- `mode` is `0000` unless explicitly requested
This ensures that two independent runs over the same directory tree produce
byte-identical `.mf` files (assuming file contents and metadata have not
go.mod
+17 -15
View File
@@ -1,29 +1,31 @@
module sneak.berlin/go/mfer
go 1.23
go 1.27.1
require (
github.com/ProtonMail/go-crypto v1.5.2
github.com/creack/pty v1.1.25-0.20260601142114-9246436fffe8
github.com/davecgh/go-spew v1.1.1
github.com/dustin/go-humanize v1.0.1
github.com/google/uuid v1.1.2
github.com/klauspost/compress v1.18.2
github.com/dustin/go-humanize v1.1.0
github.com/klauspost/compress v1.20.1
github.com/multiformats/go-multihash v0.2.3
github.com/spf13/afero v1.8.0
github.com/spf13/afero v1.15.0
github.com/stretchr/testify v1.12.1
github.com/urfave/cli/v3 v3.14.0
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
google.golang.org/protobuf v1.28.1
golang.org/x/term v0.46.0
google.golang.org/protobuf v1.36.12
)
require (
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-varint v0.0.6 // indirect
github.com/cloudflare/circl v1.6.3 // indirect
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
github.com/minio/sha256-simd v1.0.1 // indirect
github.com/mr-tron/base58 v1.3.0 // indirect
github.com/multiformats/go-varint v0.1.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
golang.org/x/sys v0.1.0 // indirect
golang.org/x/text v0.3.6 // indirect
lukechampine.com/blake3 v1.1.6 // indirect
golang.org/x/crypto v0.57.0 // indirect
golang.org/x/sys v0.48.0 // indirect
golang.org/x/text v0.42.0 // indirect
lukechampine.com/blake3 v1.4.1 // indirect
)
go.sum
+34 -463
View File
@@ -1,475 +1,46 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.44.3/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI=
cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk=
cloud.google.com/go v0.75.0/go.mod h1:VGuuCn7PG0dwsd5XPVm2Mm3wlh3EL55/79EKB6hlPTY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/ProtonMail/go-crypto v1.5.2 h1:cucYnvqcY7UOXVD//mSyjeaPY0SSN3v5cDkYPxumINk=
github.com/ProtonMail/go-crypto v1.5.2/go.mod h1:/RaSu30DaKO4RY+XdV/ACcCcZkGr7AhUIduq5sjzzCo=
github.com/cloudflare/circl v1.6.3 h1:9GPOhQGF9MCYUeXyMYlqTR6a5gTrgR/fBLXvUgtVcg8=
github.com/cloudflare/circl v1.6.3/go.mod h1:2eXP6Qfat4O/Yhh8BznvKnJ+uzEoTQ6jVKJRn81BiS4=
github.com/creack/pty v1.1.25-0.20260601142114-9246436fffe8 h1:CY3gjC7naqYGLMiywvj3suPfa1i0p/QEr7o8ujxL/2M=
github.com/creack/pty v1.1.25-0.20260601142114-9246436fffe8/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
github.com/dustin/go-humanize v1.1.0 h1:dbKTrvD0klcbBV/h4AWJdMuZogJACoMlvWIWZ5b2xWg=
github.com/dustin/go-humanize v1.1.0/go.mod h1:hc1CvRkJMsgxqjmjMQF3QNRAZBwY8AXBAzKYoSX9sFI=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8=
github.com/mr-tron/base58 v1.3.0 h1:K6Y13R2h+dku0wOqKtecgRnBUBPrZzLZy5aIj8lCcJI=
github.com/mr-tron/base58 v1.3.0/go.mod h1:2BuubE67DCSWwVfx37JWNG8emOC0sHEU4/HpcYgCLX8=
github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U=
github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM=
github.com/multiformats/go-varint v0.0.6 h1:gk85QWKxh3TazbLxED/NlDVv8+q+ReFJk7Y2W/KhfNY=
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/multiformats/go-varint v0.1.0 h1:i2wqFp4sdl3IcIxfAonHQV9qU5OsZ4Ts9IOoETFs5dI=
github.com/multiformats/go-varint v0.1.0/go.mod h1:5KVAVXegtfmNQQm/lCY+ATvDzvJJhSkUlGQV9wgObdI=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.8.0 h1:5MmtuhAgYeU6qpa7w7bP0dv6MBYuup0vekhSpSkoq60=
github.com/spf13/afero v1.8.0/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfAqwo=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/urfave/cli/v3 v3.14.0 h1:a8414NQlHJs0c/iBsulKLzlES0n/lEAskbL2LKpU4/s=
github.com/urfave/cli/v3 v3.14.0/go.mod h1:vXn6HxPNccJSzQr2QvwVncOKrgYGIHU0HY5h8B2nQj4=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e h1:T8NU3HyQ8ClP4SEE+KbFlg6n0NhuTsN4MyznaarGsZM=
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
golang.org/x/tools v0.0.0-20201110124207-079ba7bd75cd/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201201161351-ac6f37ff4c2a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210108195828-e2f9c7f1fc8e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/api v0.35.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg=
google.golang.org/api v0.36.0/go.mod h1:+z5ficQTmoYpPn8LCUNVpK5I7hwkpjbcgqA7I34qYtE=
google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjRCQ8=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201109203340-2640f1f9cdfb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210108203827-ffc7fda8c3d7/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8=
google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
lukechampine.com/blake3 v1.1.6 h1:H3cROdztr7RCfoaTpGZFQsrqvweFLrqS73j7L7cmR5c=
lukechampine.com/blake3 v1.1.6/go.mod h1:tkKEOtDkNtklkXtLNEOGNq5tcV90tJiA1vAA12R78LA=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI=
golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
lukechampine.com/blake3 v1.4.1 h1:I3Smz7gso8w4/TunLKec6K2fn+kyKtDxr/xcQEN84Wg=
lukechampine.com/blake3 v1.4.1/go.mod h1:QFosUxmjB8mnrWFSNwKmvxHpfY72bmD2tQ0kBMM3kwo=
internal/cli/check.go
+31 -29
View File
@@ -21,8 +21,8 @@ import (
"sneak.berlin/go/mfer/mfer"
)
// fingerprintHexLen is the length of a full GPG key fingerprint in hex
// characters.
// fingerprintHexLen is the length in hex characters of the fingerprint of
// an OpenPGP version 4 key, the only version mfer signs with.
const fingerprintHexLen = 40
var (
@@ -108,28 +108,33 @@ func (mfa *CLIApp) fetchManifestToTemp(
if tmpErr != nil {
_ = rc.Close()
return "", fmt.Errorf("failed to create temp file: %w", tmpErr)
return "", tmpErr
}
tmpPath := tmpFile.Name()
_, cpErr := io.Copy(tmpFile, rc)
// Copying stops one byte past mfa.maxManifestSize, which is enough to
// tell that the manifest is too large.
written, cpErr := io.Copy(tmpFile, io.LimitReader(rc, mfa.maxManifestSize+1))
_ = rc.Close()
_ = tmpFile.Close()
if cpErr == nil && written > mfa.maxManifestSize {
cpErr = fmt.Errorf("%w of %d bytes", errManifestTooLarge, mfa.maxManifestSize)
}
if cpErr != nil {
_ = mfa.Fs.Remove(tmpPath)
return "", fmt.Errorf("failed to download manifest: %w", cpErr)
return "", fmt.Errorf("download manifest: %w", cpErr)
}
return tmpPath, nil
}
// verifyRequiredSigner enforces the --require-signature fingerprint
// against the manifest's embedded signing key.
func verifyRequiredSigner(
ctx context.Context, chk *mfer.Checker, requiredSigner string,
) error {
// against the key that made the manifest's signature.
func verifyRequiredSigner(chk *mfer.Checker, requiredSigner string) error {
// Validate fingerprint format: must be exactly 40 hex characters
if len(requiredSigner) != fingerprintHexLen {
return fmt.Errorf("%w, got %d", errInvalidFingerprint, len(requiredSigner))
@@ -137,7 +142,7 @@ func verifyRequiredSigner(
_, err := hex.DecodeString(requiredSigner)
if err != nil {
return fmt.Errorf("invalid fingerprint: must be valid hex: %w", err)
return fmt.Errorf("invalid fingerprint: %w", err)
}
if !chk.IsSigned() {
@@ -145,22 +150,17 @@ func verifyRequiredSigner(
errManifestNotSigned, requiredSigner)
}
// Extract fingerprint from the embedded public key (not from the
// signer field). This validates the key is importable and gets its
// actual fingerprint.
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(ctx)
if err != nil {
return fmt.Errorf(
"failed to extract fingerprint from embedded signing key: %w", err)
}
// Loading the manifest checked that the signer is the fingerprint of
// the key that made the signature.
signer := string(chk.Signer())
// Compare fingerprints - must be exact match (case-insensitive)
if !strings.EqualFold(embeddedFP, requiredSigner) {
if !strings.EqualFold(signer, requiredSigner) {
return fmt.Errorf("embedded signing key fingerprint %s %w %s",
embeddedFP, errSignerMismatch, requiredSigner)
signer, errSignerMismatch, requiredSigner)
}
log.Infof("manifest signature verified (signer: %s)", embeddedFP)
log.Infof("manifest signature verified (signer: %s)", signer)
return nil
}
@@ -232,7 +232,7 @@ func findExtraFiles(
err := chk.FindExtraFiles(ctx, extraResults)
if err != nil {
return fmt.Errorf("failed to check for extra files: %w", err)
return fmt.Errorf("find extra files: %w", err)
}
<-extraDone
@@ -275,7 +275,7 @@ func runCheck(
progressWg.Wait()
if err != nil {
return 0, fmt.Errorf("check failed: %w", err)
return 0, fmt.Errorf("check files: %w", err)
}
// Wait for results processing to complete
@@ -296,14 +296,18 @@ func (mfa *CLIApp) checkManifestOperation(
manifestPath, err := mfa.resolveManifestArg(cmd)
if err != nil {
return fmt.Errorf("check: %w", err)
return err
}
// Done before a URL is swapped for the temp file it is downloaded to,
// whose directory is not the base.
basePath := resolveBasePath(cmd, manifestPath)
// URL manifests need to be downloaded to a temp file for the checker
if isHTTPURL(manifestPath) {
tmpPath, tmpErr := mfa.fetchManifestToTemp(ctx, manifestPath)
if tmpErr != nil {
return fmt.Errorf("check: %w", tmpErr)
return tmpErr
}
defer func() { _ = mfa.Fs.Remove(tmpPath) }()
@@ -311,26 +315,24 @@ func (mfa *CLIApp) checkManifestOperation(
manifestPath = tmpPath
}
basePath := cmd.String("base")
showProgress := cmd.Bool("progress")
log.Infof("checking manifest %s with base %s", manifestPath, basePath)
// Create checker
//nolint:contextcheck // mfer loads a manifest without a context
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath,
BasePath: basePath,
Fs: mfa.Fs,
})
if err != nil {
return fmt.Errorf("failed to load manifest: %w", err)
return fmt.Errorf("load manifest: %w", err)
}
// Check signature requirement
requiredSigner := cmd.String(flagRequireSignature)
if requiredSigner != "" {
err = verifyRequiredSigner(ctx, chk, requiredSigner)
err = verifyRequiredSigner(chk, requiredSigner)
if err != nil {
return err
}
internal/cli/entry.go
+2
View File
@@ -5,6 +5,7 @@ import (
"os"
"github.com/spf13/afero"
"sneak.berlin/go/mfer/mfer"
)
// NoColor disables colored output when set. Automatically true if the
@@ -60,6 +61,7 @@ func RunWithOptions(opts *RunOptions) int {
version: opts.Version,
gitrev: opts.Gitrev,
exitCode: 0,
maxManifestSize: mfer.MaxManifestSize,
Stdin: opts.Stdin,
Stdout: opts.Stdout,
Stderr: opts.Stderr,
internal/cli/entry_test.go
+510
View File
@@ -3,10 +3,12 @@ package cli
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"math/rand"
"net/http/httptest"
"os"
"path/filepath"
"slices"
@@ -353,6 +355,77 @@ func TestGenerateCommand(t *testing.T) {
assert.True(t, exists)
}
// TestGenerateRefusesTwoFilesAtOnePath runs gen on arguments whose files
// would share a path in the manifest: two directories that each hold a.txt,
// and one directory given twice. gen must fail while it lists the files,
// before it hashes any, naming the path, and write no manifest.
func TestGenerateRefusesTwoFilesAtOnePath(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
first, second string
}{
{"two directories", testDir, "/other"},
{"one directory twice", testDir, testDir},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
require.NoError(t, fs.MkdirAll("/other", 0o755))
writeTestFile(t, fs, "/testdir/a.txt", "first")
writeTestFile(t, fs, "/other/a.txt", "second")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "-o", testOutput, tc.first, tc.second,
}, fs)
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
`enumerate files: duplicate path "a.txt": `+
tc.first+"/a.txt and "+tc.second+"/a.txt")
exists, err := afero.Exists(fs, testOutput)
require.NoError(t, err)
assert.False(t, exists)
})
}
}
// TestGenerateSeededManifestBytes pins the exact bytes `gen --seed` writes
// for a fixed tree, so that a Go or dependency update that changes what
// mfer writes fails here. testdata/seeded.mf was written by an mfer built
// before such an update. The tree has enough files that zstd compresses
// the manifest instead of storing it as it is.
func TestGenerateSeededManifestBytes(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
start := time.Date(2025, 6, 1, 0, 0, 0, 0, time.UTC)
for i := range 200 {
path := fmt.Sprintf("/testdir/d%d/f%03d.txt", i%10, i)
mtime := start.Add(time.Duration(i) * time.Second)
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o755))
writeTestFile(t, fs, path, fmt.Sprintf("file %d\n", i))
require.NoError(t, fs.Chtimes(path, mtime, mtime))
}
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "--seed", "mfer", "-o", testOutput, testDir,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
got, err := afero.ReadFile(fs, testOutput)
require.NoError(t, err)
want, err := os.ReadFile("testdata/seeded.mf")
require.NoError(t, err)
assert.Equal(t, want, got)
}
func TestGenerateAndCheckCommand(t *testing.T) {
t.Parallel()
@@ -374,6 +447,98 @@ func TestGenerateAndCheckCommand(t *testing.T) {
assert.Equal(t, 0, exitCode, "check failed: %s", testStderr(t, opts))
}
// TestGenerateRecordsModeOnlyWhenAsked runs gen with and without
// --include-permissions: without it every mode is recorded as 0000, with
// it each file's permission bits. list -l and export show the recorded
// modes.
func TestGenerateRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, "/testdir/notes.txt", "hello world")
writeTestFile(t, fs, "/testdir/run.sh", "#!/bin/sh\n")
require.NoError(t, fs.Chmod("/testdir/run.sh", 0o755))
for _, tc := range []struct {
flags []string
want map[string]string // recorded mode by path
}{
{nil, map[string]string{"notes.txt": "0000", "run.sh": "0000"}},
{
[]string{"--" + flagIncludePermissions},
map[string]string{"notes.txt": "0644", "run.sh": "0755"},
},
} {
opts := testOpts(slices.Concat(
[]string{testApp, cmdGenerate, "-q", "-f", "-o", testOutput}, tc.flags,
[]string{testDir},
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
opts = testOpts([]string{testApp, cmdList, "-l", testOutput}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
listed := map[string]string{}
out := strings.TrimSuffix(testStdout(t, opts), "\n")
for line := range strings.SplitSeq(out, "\n") {
fields := strings.Split(line, "\t") // mode, size, mtime, path
require.Len(t, fields, 4, line)
listed[fields[3]] = fields[0]
}
assert.Equal(t, tc.want, listed, "list -l %v", tc.flags)
opts = testOpts([]string{testApp, cmdExport, testOutput}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
var entries []ExportEntry
require.NoError(t, json.Unmarshal([]byte(testStdout(t, opts)), &entries))
exported := map[string]string{}
for _, e := range entries {
exported[e.Path] = e.Mode
}
assert.Equal(t, tc.want, exported, "export %v", tc.flags)
}
}
// TestCheckComparesRecordedMode changes a file's mode after gen: check
// must fail on it when gen recorded the file's mode, and pass when gen
// recorded 0000.
func TestCheckComparesRecordedMode(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
flags []string
exitCode int
}{
{nil, 0},
{[]string{"--" + flagIncludePermissions}, 1},
} {
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello world")
opts := testOpts(slices.Concat(
[]string{testApp, cmdGenerate, "-q", "-o", testMF}, tc.flags,
[]string{testDir},
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
require.NoError(t, fs.Chmod(testFile1, 0o600))
opts = testOpts([]string{testApp, cmdCheck, testFlagBase, testDir, testMF}, fs)
assert.Equal(t, tc.exitCode, runCLI(opts), "%v: %s", tc.flags, testStderr(t, opts))
if tc.exitCode != 0 {
assert.Contains(t, testStderr(t, opts), "MODE_MISMATCH: file1.txt")
}
}
}
// sharedWriter appends to a buffer shared with other sharedWriters, so
// output written to stdout and stderr is kept in the order it was written.
// Each write first waits for delay.
@@ -487,6 +652,32 @@ func runCheckAfterRewrite(t *testing.T, rewritten, msg string) {
assert.Equal(t, 1, exitCode, msg)
}
// TestCheckRequireSignatureRefusesOtherSigningKey runs check
// --require-signature on a manifest signed by another key whose embedded
// public key block also holds the required key. check must refuse it.
func TestCheckRequireSignatureRefusesOtherSigningKey(t *testing.T) {
t.Parallel()
content := []byte("signed file")
manifest, required := manifestSignedByAnotherKey(t,
map[string][]byte{testFileTxt: content})
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
filepath.Join(testDir, testFileTxt), content, 0o644))
require.NoError(t, afero.WriteFile(fs, testManifest, manifest, 0o644))
opts := testOpts([]string{
testApp, cmdCheck, "-q", testFlagBase, testDir,
"--" + flagRequireSignature, required, testManifest,
}, fs)
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
"load manifest: "+
"embedded public key block must hold exactly one key, found 2")
}
func TestCheckCommandWithCorruptedFile(t *testing.T) {
t.Parallel()
@@ -788,6 +979,90 @@ func TestCheckNeverReportsManifest(t *testing.T) {
}
}
// The directory setupManifestInSubdir makes and the manifest it writes there,
// relative to the working directory it sets.
const (
testSubdir = "sub"
testSubdirManifest = testSubdir + "/" + defaultManifestName
)
// setupManifestInSubdir makes a temp dir holding file.txt and sub/b.txt,
// where sub/index.mf is the manifest gen writes for sub, and makes it the
// working directory, so a test calling it cannot run in parallel. It returns
// the temp dir.
func setupManifestInSubdir(t *testing.T) string {
t.Helper()
root := t.TempDir()
sub := filepath.Join(root, testSubdir)
fs := afero.NewOsFs()
require.NoError(t, fs.MkdirAll(sub, 0o750))
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "not in the manifest")
writeTestFile(t, fs, filepath.Join(sub, "b.txt"), "in the manifest")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "-o", filepath.Join(sub, defaultManifestName), sub,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
t.Chdir(root)
return root
}
// TestCheckResolvesEntriesAgainstManifestDirectory runs check from the
// directory above sub, on the manifest in sub. Without --base, the
// manifest's entries are looked for in sub, whether check is given the
// manifest or sub, and the files above sub are not reported. --base names
// the directory to look in instead, the current one included.
//
//nolint:paralleltest // changes the process-global working directory
func TestCheckResolvesEntriesAgainstManifestDirectory(t *testing.T) {
root := setupManifestInSubdir(t)
for _, tc := range []struct {
args []string
exitCode int
failure string // a line check must print, if any
}{
{[]string{testSubdir}, 0, ""},
{[]string{testSubdirManifest}, 0, ""},
{[]string{filepath.Join(root, testSubdir)}, 0, ""},
{[]string{testFlagBase, testSubdir, testSubdirManifest}, 0, ""},
{[]string{testFlagBase, ".", testSubdirManifest}, 1, "MISSING: b.txt"},
} {
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
opts := testOpts(slices.Concat(
[]string{testApp, cmdCheck, testFlagNoExtra}, tc.args,
), afero.NewOsFs())
assert.Equal(t, tc.exitCode, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Contains(t, testStderr(t, opts), tc.failure)
})
}
}
// TestCheckURLManifestResolvesEntriesAgainstCurrentDirectory runs check on
// a manifest given by URL, from a directory holding the file it lists: the
// file is looked for there.
//
//nolint:paralleltest // changes the process-global working directory
func TestCheckURLManifestResolvesEntriesAgainstCurrentDirectory(t *testing.T) {
files := map[string][]byte{testFileTxt: []byte("hello")}
server := httptest.NewServer(fetchTestHandler(manifestOf(t, files), files))
defer server.Close()
cwd := chdirTemp(t)
require.NoError(t,
os.WriteFile(filepath.Join(cwd, testFileTxt), files[testFileTxt], 0o600))
opts := testOpts([]string{
testApp, cmdCheck, testFlagNoExtra, server.URL + "/" + defaultManifestName,
}, afero.NewOsFs())
assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
}
// unlistableDirFs is a filesystem on which one directory cannot be listed.
type unlistableDirFs struct {
afero.Fs
@@ -1013,6 +1288,225 @@ func TestGenerateLeavesLeftoverTempFileOutOfListing(t *testing.T) {
assert.Equal(t, []string{testFileTxt}, manifestPaths(t, fs, output))
}
// writeTestTree writes file.txt and sub/nested.txt under dir.
func writeTestTree(t *testing.T, fs afero.Fs, dir string) {
t.Helper()
require.NoError(t, fs.MkdirAll(filepath.Join(dir, testSubdir), 0o750))
writeTestFile(t, fs, filepath.Join(dir, testFileTxt), "hello")
writeTestFile(t, fs, filepath.Join(dir, testSubdir, "nested.txt"), "in sub")
}
// TestGenerateDefaultOutput runs gen without --output on one directory or
// one file: it writes index.mf in that directory, or beside that file,
// listing each file by its path under the directory index.mf is in, and
// check given that directory passes.
func TestGenerateDefaultOutput(t *testing.T) {
t.Parallel()
// Paths are relative to a temp dir holding file.txt and sub/nested.txt.
for name, tc := range map[string]struct {
input, output string
listed []string
}{
"directory": {
".", defaultManifestName, []string{testFileTxt, "sub/nested.txt"},
},
"subdirectory": {testSubdir, testSubdirManifest, []string{"nested.txt"}},
"file": {testFileTxt, defaultManifestName, []string{testFileTxt}},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
root := t.TempDir()
fs := afero.NewOsFs()
writeTestTree(t, fs, root)
opts := testOpts([]string{
testApp, cmdGenerate, "-q", filepath.Join(root, tc.input),
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
output := filepath.Join(root, tc.output)
assert.ElementsMatch(t, tc.listed, manifestPaths(t, fs, output))
opts = testOpts([]string{
testApp, cmdCheck, "-q", filepath.Dir(output),
}, fs)
assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
})
}
}
// TestGenerateSeveralPathsDefaultOutput runs gen without --output on two
// directories: it writes index.mf in the current directory, listing both
// directories' files, and writes no index.mf in either directory.
//
//nolint:paralleltest // changes the process-global working directory
func TestGenerateSeveralPathsDefaultOutput(t *testing.T) {
root := t.TempDir()
fs := afero.NewOsFs()
dirs := []string{"first", "second"}
for _, dir := range dirs {
require.NoError(t, fs.MkdirAll(filepath.Join(root, dir), 0o750))
writeTestFile(t, fs, filepath.Join(root, dir, dir+".txt"), dir)
}
t.Chdir(root)
opts := testOpts(append([]string{testApp, cmdGenerate, "-q"}, dirs...), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, []string{"first.txt", "second.txt"},
manifestPaths(t, fs, filepath.Join(root, defaultManifestName)))
for _, dir := range dirs {
exists, err := afero.Exists(fs, filepath.Join(root, dir, defaultManifestName))
require.NoError(t, err)
assert.False(t, exists, "index.mf written in %s", dir)
}
}
// testLink is the name of the symlink to a tree that the
// DirectoryNamedThroughSymlink tests make in a temp dir.
const testLink = "link"
// TestGenerateDirectoryNamedThroughSymlink runs gen on a directory named
// through a symlink, given as the argument or as the working directory: the
// manifest lists the files in the directory the symlink points to, and
// leaves out a symlink inside it, as gen does without --follow-symlinks.
//
//nolint:paralleltest // changes the process-global working directory
func TestGenerateDirectoryNamedThroughSymlink(t *testing.T) {
// Paths are relative to a temp dir holding data and link, a symlink to
// data.
for name, tc := range map[string]struct {
workDir string
args []string
}{
"argument": {".", []string{testLink}},
"working directory": {testLink, nil},
} {
t.Run(name, func(t *testing.T) {
root := t.TempDir()
data := filepath.Join(root, "data")
fs := afero.NewOsFs()
writeTestTree(t, fs, data)
require.NoError(t,
os.Symlink(testFileTxt, filepath.Join(data, "alias.txt")))
require.NoError(t, os.Symlink(data, filepath.Join(root, testLink)))
// t.Chdir sets PWD to the path it is given, as a shell does, and
// os.Getwd returns PWD when it names the working directory.
t.Chdir(filepath.Join(root, tc.workDir))
opts := testOpts(slices.Concat(
[]string{testApp, cmdGenerate, "-q"}, tc.args,
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, []string{testFileTxt, "sub/nested.txt"},
manifestPaths(t, fs, filepath.Join(data, defaultManifestName)))
})
}
}
// TestGenerateBytesDoNotDependOnOutput runs gen --seed on one tree, each
// time writing to another file, over a file already there and beside an
// earlier run's temp file: neither is listed, and what is listed depends
// only on the tree, so every manifest has the same bytes.
func TestGenerateBytesDoNotDependOnOutput(t *testing.T) {
t.Parallel()
root := t.TempDir()
tree := filepath.Join(root, "tree")
defaultOutput := filepath.Join(tree, defaultManifestName)
fs := afero.NewOsFs()
writeTestTree(t, fs, tree)
var first []byte
for _, output := range []string{
defaultOutput,
filepath.Join(tree, "listing.mf"),
filepath.Join(tree, testSubdir, "listing.mf"),
filepath.Join(root, "outside.mf"),
} {
writeTestFile(t, fs, output, "previous manifest")
writeTestFile(t, fs, manifestTempPath(output), "part of a manifest")
args := []string{testApp, cmdGenerate, "-q", "-f", "--seed", "mfer"}
if output != defaultOutput {
args = append(args, "-o", output)
}
args = append(args, tree)
opts := testOpts(args, fs)
require.Equal(t, 0, runCLI(opts),
"output %s, stderr: %s", output, testStderr(t, opts))
got, err := afero.ReadFile(fs, output)
require.NoError(t, err)
require.NoError(t, fs.Remove(output))
if first == nil {
first = got
}
assert.Equal(t, first, got, "output %s", output)
}
}
// TestGenerateRefusesExistingDefaultOutput runs gen without --output or
// --force on a directory already holding index.mf: gen fails, naming that
// file, and leaves it as it was.
func TestGenerateRefusesExistingDefaultOutput(t *testing.T) {
t.Parallel()
output := filepath.Join(testDir, defaultManifestName)
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello")
writeTestFile(t, fs, output, "previous manifest")
opts := testOpts([]string{testApp, cmdGenerate, "-q", testDir}, fs)
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
"output file "+output+" already exists (use --force to overwrite)")
content, err := afero.ReadFile(fs, output)
require.NoError(t, err)
assert.Equal(t, "previous manifest", string(content))
}
// TestGenerateRefusesEmptyOutput runs gen with --force and an --output
// given an empty value, as an unset shell variable gives it, on a
// directory already holding index.mf: gen fails and leaves that file as it
// was.
func TestGenerateRefusesEmptyOutput(t *testing.T) {
t.Parallel()
output := filepath.Join(testDir, defaultManifestName)
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello")
writeTestFile(t, fs, output, "previous manifest")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-f", "-o", "", testDir}, fs)
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts), errEmptyOutput.Error())
content, err := afero.ReadFile(fs, output)
require.NoError(t, err)
assert.Equal(t, "previous manifest", string(content))
}
func TestGenerateAtomicWriteUsesTemp(t *testing.T) {
t.Parallel()
@@ -1170,6 +1664,22 @@ func TestGenerateValidatesInputPaths(t *testing.T) {
})
}
// TestFlagAfterArgumentIsArgument asserts that flags are read only before a
// command's first argument: after it, -v is a path, not the verbose flag.
func TestFlagAfterArgumentIsArgument(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "content")
opts := testOpts([]string{testApp, cmdGenerate, testDir, "-v"}, fs)
exitCode := runCLI(opts)
assert.Equal(t, 1, exitCode)
assert.Contains(t, testStderr(t, opts), "path does not exist: -v")
}
func TestCheckDetectsManifestCorruption(t *testing.T) {
t.Parallel()
internal/cli/errmsg_test.go
+203 -50
View File
@@ -4,17 +4,23 @@ package cli
import (
"bytes"
"context"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"github.com/ProtonMail/go-crypto/openpgp"
"github.com/ProtonMail/go-crypto/openpgp/armor"
"github.com/ProtonMail/go-crypto/openpgp/packet"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v3"
"google.golang.org/protobuf/proto"
"sneak.berlin/go/mfer/mfer"
)
@@ -86,8 +92,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
t.Run("invalid fingerprint length", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), "12345678")
err := verifyRequiredSigner(unsignedChecker(t), "12345678")
require.ErrorIs(t, err, errInvalidFingerprint)
assert.EqualError(t, err,
"invalid fingerprint: must be exactly 40 hex characters, got 8")
@@ -96,8 +101,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
t.Run("manifest not signed", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), msgFpA)
err := verifyRequiredSigner(unsignedChecker(t), msgFpA)
require.ErrorIs(t, err, errManifestNotSigned)
assert.EqualError(t, err,
"manifest is not signed, but signature from "+msgFpA+" is required")
@@ -105,64 +109,66 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
}
// TestSignerMismatchMessage drives verifyRequiredSigner against a real signed
// manifest. The embedded fingerprint is whatever the generated key produced,
// so it is read back from the checker and substituted into the expected
// string; the required signer is a fixed value that cannot match it. Requires
// gpg and is skipped where it is absent, as the other signing tests are.
//
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
// manifest. The signing key's fingerprint is whatever the generated key
// produced, so it is read back from the checker and substituted into the
// expected string; the required signer is a fixed value that cannot match
// it.
func TestSignerMismatchMessage(t *testing.T) {
t.Parallel()
chk := signedChecker(t,
signedManifest(t, map[string][]byte{"f.txt": []byte("signed file")}))
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
require.NoError(t, err)
err = verifyRequiredSigner(context.Background(), chk, msgFpB)
err := verifyRequiredSigner(chk, msgFpB)
require.ErrorIs(t, err, errSignerMismatch)
assert.EqualError(t, err,
"embedded signing key fingerprint "+embeddedFP+
"embedded signing key fingerprint "+string(chk.Signer())+
" does not match required "+msgFpB)
}
// signedManifest returns a manifest of files signed by a throwaway GPG key
// generated in a temporary GNUPGHOME, which it leaves set for the rest of
// the test.
// testSecretKey returns a new OpenPGP key with its secret key, armored, as
// gpg --export-secret-keys --armor writes it, and the key's fingerprint.
// The key is protected by passphrase unless that is nil. config sets how
// the key is made; without one it is an Ed25519 key, which is quick to
// make.
func testSecretKey(
t *testing.T, passphrase []byte, config *packet.Config,
) ([]byte, string) {
t.Helper()
if config == nil {
config = &packet.Config{Algorithm: packet.PubKeyAlgoEdDSA}
}
key, err := openpgp.NewEntity("MFER Test Key", "", "test@mfer.test", config)
require.NoError(t, err)
if passphrase != nil {
require.NoError(t, key.EncryptPrivateKeys(passphrase, nil))
}
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
require.NoError(t, err)
require.NoError(t, key.SerializePrivateWithoutSigning(w, nil))
require.NoError(t, w.Close())
return buf.Bytes(), strings.ToUpper(hex.EncodeToString(key.PrimaryKey.Fingerprint))
}
// signedManifest returns a manifest of files signed by a new OpenPGP key.
func signedManifest(t *testing.T, files map[string][]byte) []byte {
t.Helper()
_, err := exec.LookPath("gpg")
if err != nil {
t.Skip("gpg not installed, skipping signing test")
}
gpgHome := t.TempDir()
params := "%no-protection\n" +
"Key-Type: RSA\nKey-Length: 2048\n" +
"Name-Real: MFER Test Key\nName-Email: test@mfer.test\n" +
"Expire-Date: 0\n%commit\n"
paramsFile := filepath.Join(gpgHome, "key-params")
require.NoError(t, os.WriteFile(paramsFile, []byte(params), 0o600))
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
cmd := exec.CommandContext(context.Background(), "gpg",
"--batch", "--gen-key", paramsFile)
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
out, err := cmd.CombinedOutput()
if err != nil {
t.Skipf("failed to generate test GPG key: %v: %s", err, out)
}
t.Setenv("GNUPGHOME", gpgHome)
secretKey, _ := testSecretKey(t, nil, nil)
b := mfer.NewBuilder()
b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
b.SetSigningOptions(&mfer.SigningOptions{SecretKey: secretKey})
for path, content := range files {
_, err = b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)),
mfer.ModTime{}, bytes.NewReader(content), nil)
_, err := b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)),
mfer.ModTime{}, 0, bytes.NewReader(content), nil)
require.NoError(t, err)
}
@@ -191,6 +197,52 @@ func signedChecker(t *testing.T, manifest []byte) *mfer.Checker {
return chk
}
// manifestSignedByAnotherKey returns a manifest of files and the
// fingerprint of a new key, the required key, that did not sign it.
// The manifest is signed by a second new key; its embedded public key
// block holds the required key followed by the second key, and its signer
// field names the required key.
func manifestSignedByAnotherKey(
t *testing.T, files map[string][]byte,
) ([]byte, string) {
t.Helper()
required := new(mfer.MFFileOuter)
require.NoError(t, proto.Unmarshal(
signedManifest(t, files)[len(mfer.MAGIC):], required))
outer := new(mfer.MFFileOuter)
require.NoError(t, proto.Unmarshal(
signedManifest(t, files)[len(mfer.MAGIC):], outer))
// One armored block holding both keys, as gpg --export --armor writes
// two keys.
var block bytes.Buffer
w, err := armor.Encode(&block, openpgp.PublicKeyType, nil)
require.NoError(t, err)
for _, key := range [][]byte{
required.GetSigningPubKey(), outer.GetSigningPubKey(),
} {
decoded, err := armor.Decode(bytes.NewReader(key))
require.NoError(t, err)
_, err = io.Copy(w, decoded.Body)
require.NoError(t, err)
}
require.NoError(t, w.Close())
outer.SigningPubKey = block.Bytes()
outer.Signer = required.GetSigner()
data, err := proto.Marshal(outer)
require.NoError(t, err)
return append([]byte(mfer.MAGIC), data...), string(required.GetSigner())
}
func TestPathDoesNotExistMessage(t *testing.T) {
t.Parallel()
@@ -277,7 +329,7 @@ func TestManifestLoaderHTTPStatusMessage(t *testing.T) {
_, err := mfa.openManifestReader(context.Background(), server.URL+"/foo.mf")
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err,
"failed to fetch "+server.URL+"/foo.mf: HTTP 404")
"download manifest "+server.URL+"/foo.mf: unexpected HTTP status 404")
}
func TestFetchManifestHTTPStatusMessage(t *testing.T) {
@@ -299,7 +351,7 @@ func TestFetchManifestHTTPStatusMessage(t *testing.T) {
return cmd.Run(context.Background(), []string{cmdFetch, server.URL})
})
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "failed to fetch manifest: HTTP 404")
assert.EqualError(t, err, "download manifest: unexpected HTTP status 404")
}
func TestFetchFileHTTPStatusMessage(t *testing.T) {
@@ -317,7 +369,108 @@ func TestFetchFileHTTPStatusMessage(t *testing.T) {
&mfer.MFFilePath{}, nil)
})
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "HTTP 500")
assert.EqualError(t, err, "unexpected HTTP status 500")
}
// TestCheckCorruptManifestMessage runs check on a file that is not a
// manifest.
func TestCheckCorruptManifestMessage(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/bad.mf", []byte("not a manifest"), 0o644))
mfa := &CLIApp{Fs: fs}
cmd := mfa.checkCommand()
cmd.Action = mfa.checkManifestOperation
// checkManifestOperation logs to the process-global logger.
err := runLocked(func() error {
return cmd.Run(context.Background(), []string{cmdCheck, "/bad.mf"})
})
assert.EqualError(t, err, "load manifest: invalid file format")
}
// TestListMissingManifestMessage runs list on a manifest file that does not
// exist.
func TestListMissingManifestMessage(t *testing.T) {
t.Parallel()
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
cmd := mfa.listCommand()
// listManifestOperation sets the process-global log level.
err := runLocked(func() error {
return cmd.Run(context.Background(), []string{cmdList, "/nope.mf"})
})
require.ErrorIs(t, err, os.ErrNotExist)
assert.EqualError(t, err, "open /nope.mf: file does not exist")
}
// TestFetchHashMismatchMessage runs fetch against a server that sends a
// listed file with other content of the same size.
func TestFetchHashMismatchMessage(t *testing.T) {
t.Parallel()
manifest := builtManifest(t, map[string][]byte{testFileTxt: []byte("listed")})
server := httptest.NewServer(fetchTestHandler(manifest,
map[string][]byte{testFileTxt: []byte("served")}))
defer server.Close()
mfa := &CLIApp{Fs: afero.NewMemMapFs(), maxManifestSize: mfer.MaxManifestSize}
cmd := mfa.fetchCommand()
cmd.Action = mfa.fetchManifestOperation
// fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error {
return cmd.Run(context.Background(),
[]string{cmdFetch, "--" + flagDest, t.TempDir(), server.URL})
})
require.ErrorIs(t, err, errHashMismatch)
assert.EqualError(t, err, "download "+testFileTxt+": hash mismatch")
}
// TestFreshenBackslashPathMessage runs freshen on a tree that has gained a
// file whose name holds a backslash, which a manifest path may not contain.
func TestFreshenBackslashPathMessage(t *testing.T) {
t.Parallel()
fs := afero.NewOsFs()
root, manifestPath := setupFreshenDir(t, fs,
map[string]string{testFileTxt: "content"})
writeTestFile(t, fs, filepath.Join(root, `a\b.txt`), "new")
mfa := &CLIApp{Fs: fs}
cmd := mfa.freshenCommand()
cmd.Action = mfa.freshenManifestOperation
// freshenManifestOperation logs to the process-global logger.
err := runLocked(func() error {
return cmd.Run(context.Background(),
[]string{cmdFreshen, testFlagBase, root, manifestPath})
})
assert.EqualError(t, err,
`path "a\\b.txt" contains backslash; use forward slashes only`)
}
// TestFreshenReadErrorMessage has freshen hash a directory as though it
// were a file, so reading it fails.
func TestFreshenReadErrorMessage(t *testing.T) {
t.Parallel()
root := t.TempDir()
require.NoError(t, os.Mkdir(filepath.Join(root, "sub"), 0o750))
hasher := &freshenHasher{
fs: afero.NewOsFs(),
absBase: root,
builder: mfer.NewBuilder(),
}
err := hasher.processEntry(&freshenEntry{path: "sub", needsHash: true})
assert.EqualError(t, err,
"read "+filepath.Join(root, "sub")+": is a directory")
}
func TestURLRequiredMessage(t *testing.T) {
internal/cli/export.go
+6 -5
View File
@@ -18,6 +18,7 @@ type ExportEntry struct {
Hashes []string `json:"hashes"`
Mtime *string `json:"mtime,omitempty"`
Ctime *string `json:"ctime,omitempty"`
Mode string `json:"mode"` // octal, "0000" when none was recorded
}
func (mfa *CLIApp) exportManifestOperation(
@@ -25,20 +26,19 @@ func (mfa *CLIApp) exportManifestOperation(
) error {
pathOrURL, err := mfa.resolveManifestArg(cmd)
if err != nil {
return fmt.Errorf("export: %w", err)
return err
}
rc, err := mfa.openManifestReader(ctx, pathOrURL)
if err != nil {
return fmt.Errorf("export: %w", err)
return err
}
defer func() { _ = rc.Close() }()
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(rc)
if err != nil {
return fmt.Errorf("export: failed to parse manifest: %w", err)
return fmt.Errorf("parse manifest: %w", err)
}
files := manifest.Files()
@@ -49,6 +49,7 @@ func (mfa *CLIApp) exportManifestOperation(
Path: f.GetPath(),
Size: f.GetSize(),
Hashes: make([]string, 0, len(f.GetHashes())),
Mode: fmt.Sprintf("%04o", f.GetMode()),
}
for _, h := range f.GetHashes() {
@@ -74,7 +75,7 @@ func (mfa *CLIApp) exportManifestOperation(
err = enc.Encode(entries)
if err != nil {
return fmt.Errorf("export: failed to encode JSON: %w", err)
return fmt.Errorf("encode JSON: %w", err)
}
return nil
internal/cli/export_test.go
+1 -1
View File
@@ -132,7 +132,7 @@ func TestListFromHTTPURL(t *testing.T) {
exitCode := runCLI(&RunOptions{
Appname: testApp,
Args: []string{testApp, "list", server.URL + "/index.mf"},
Args: []string{testApp, cmdList, server.URL + "/index.mf"},
Stdin: &bytes.Buffer{},
Stdout: &stdout,
Stderr: &stderr,
internal/cli/fetch.go
+98 -53
View File
@@ -95,6 +95,9 @@ var (
// writes another file.
errNameClash = errors.New(
"manifest lists a file where fetch writes another file")
// errModeOutOfRange indicates a manifest that lists a mode with more
// than the permission bits, such as setuid.
errModeOutOfRange = errors.New("manifest lists a mode outside 0777")
)
// DownloadProgress reports the progress of a single file download.
@@ -236,7 +239,7 @@ func reportDownloadProgress(progress <-chan DownloadProgress, done chan<- struct
func manifestBaseURL(manifestURL string) (*url.URL, error) {
parsed, err := url.Parse(manifestURL)
if err != nil {
return nil, fmt.Errorf("fetch: invalid manifest URL: %w", err)
return nil, fmt.Errorf("invalid manifest URL: %w", err)
}
// JoinPath cleans the path it builds, so ".." drops the manifest's
@@ -265,7 +268,7 @@ func downloadManifestFiles(
// Sanitize the path to prevent path traversal attacks
localPath, err := sanitizePath(f.GetPath())
if err != nil {
return 0, 0, fmt.Errorf("invalid path in manifest: %w", err)
return 0, 0, fmt.Errorf("invalid file entry: %w", err)
}
if alreadyPresent(dest, localPath, f) {
@@ -281,7 +284,7 @@ func downloadManifestFiles(
err = downloadFile(ctx, client, fileURL, dest, localPath, f, progress)
if err != nil {
return 0, 0, fmt.Errorf("failed to download %s: %w", f.GetPath(), err)
return 0, 0, fmt.Errorf("download %s: %w", f.GetPath(), err)
}
downloaded++
@@ -292,11 +295,11 @@ func downloadManifestFiles(
}
// alreadyPresent reports whether localPath under dest is a regular file
// with the size and one of the hashes the manifest lists for entry. It
// hashes the whole file, since a matching size alone would accept a
// corrupted or partly written one. A file it cannot read, or reaches only
// through a symlink, is not present: fetch downloads it, and the download
// reports the problem.
// with the size, the recorded mode if any, and one of the hashes the
// manifest lists for entry. It hashes the whole file, since a matching
// size alone would accept a corrupted or partly written one. A file it
// cannot read, or reaches only through a symlink, is not present: fetch
// downloads it, and the download reports the problem.
func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
if checkNoSymlinks(dest, localPath) != nil {
return false
@@ -309,6 +312,12 @@ func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
return false
}
// A recorded mode of 0 means none was recorded.
if entry.GetMode() != 0 &&
info.Mode().Perm() != os.FileMode(entry.GetMode()).Perm() {
return false
}
// G304: localPath is a relative path that sanitizePath keeps inside
// dest as text, and checkNoSymlinks just found no symlink in it.
f, err := os.Open(path) //nolint:gosec // G304: see comment above
@@ -352,7 +361,7 @@ func (mfa *CLIApp) fetchManifestOperation(
firstDelay: firstRetryDelay,
}
manifestData, files, err := fetchManifest(ctx, cmd, client, manifestURL)
manifestData, files, err := mfa.fetchManifest(ctx, cmd, client, manifestURL)
if err != nil {
return err
}
@@ -367,7 +376,7 @@ func (mfa *CLIApp) fetchManifestOperation(
err = os.MkdirAll(dest, dirPerms)
if err != nil {
return fmt.Errorf("failed to create destination directory %s: %w", dest, err)
return err
}
// Create progress channel and start progress reporter goroutine
@@ -394,7 +403,7 @@ func (mfa *CLIApp) fetchManifestOperation(
// "mfer check" can verify the tree later.
err = saveManifest(dest, manifestData)
if err != nil {
return fmt.Errorf("failed to save manifest: %w", err)
return fmt.Errorf("save manifest: %w", err)
}
// Print summary
@@ -414,40 +423,46 @@ func (mfa *CLIApp) fetchManifestOperation(
// fetchManifest downloads the manifest at manifestURL and parses it,
// enforcing --require-signature if it is given and refusing a manifest
// that lists a file where fetch writes another. It returns the manifest as
// downloaded, to be saved once the files are in place, and the files it
// lists.
func fetchManifest(
// that lists a file where fetch writes another or a mode outside 0777. It
// returns the manifest as downloaded, to be saved once the files are in
// place, and the files it lists.
func (mfa *CLIApp) fetchManifest(
ctx context.Context, cmd *cli.Command, client retryingClient, manifestURL string,
) ([]byte, []*mfer.MFFilePath, error) {
log.Infof("fetching manifest from %s", manifestURL)
// Read the whole manifest before parsing it, so that a connection
// lost partway through is retried rather than reported as a bad
// manifest.
// manifest. Reading stops one byte past mfa.maxManifestSize, which is
// enough to tell that the manifest is too large.
var manifestData []byte
err := client.get(ctx, manifestURL, func(resp *http.Response) error {
var readErr error
manifestData, readErr = io.ReadAll(resp.Body)
manifestData, readErr = io.ReadAll(
io.LimitReader(resp.Body, mfa.maxManifestSize+1))
return readErr
})
if err != nil {
return nil, nil, fmt.Errorf("failed to fetch manifest: %w", err)
return nil, nil, fmt.Errorf("download manifest: %w", err)
}
if int64(len(manifestData)) > mfa.maxManifestSize {
return nil, nil, fmt.Errorf("download manifest: %w of %d bytes",
errManifestTooLarge, mfa.maxManifestSize)
}
// Parse manifest
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData))
if err != nil {
return nil, nil, fmt.Errorf("failed to parse manifest: %w", err)
return nil, nil, fmt.Errorf("parse manifest: %w", err)
}
requiredSigner := cmd.String(flagRequireSignature)
if requiredSigner != "" {
err = verifyFetchedSigner(ctx, manifestData, requiredSigner)
err = verifyFetchedSigner(manifestData, requiredSigner)
if err != nil {
return nil, nil, err
}
@@ -460,23 +475,34 @@ func fetchManifest(
return nil, nil, err
}
// fetch sets each recorded mode on the file it writes, so a mode above
// 0777, which could carry setuid, setgid or sticky bits, is refused
// before any file is requested.
for _, f := range files {
if f.GetMode() > uint32(os.ModePerm) {
return nil, nil, fmt.Errorf("%w: %s (%#o)",
errModeOutOfRange, f.GetPath(), f.GetMode())
}
}
log.Infof("manifest contains %d files", len(files))
return manifestData, files, nil
}
// checkNoNameClash returns an error if files lists a file, or a directory
// a file is in, under a name where fetch writes another file: the temp
// file it downloads a listed file to, or, at the top of the tree, the
// saved manifest or its temp file. fetch would remove or replace what is
// a file is in, under a name where fetch writes another file: another
// listed file, a directory another listed file is in, the temp file it
// downloads a listed file to, or, at the top of the tree, the saved
// manifest or its temp file. fetch would remove or replace what is
// listed there, or fail partway, leaving a tree check rejects. Names are
// compared ignoring case, since on a case-insensitive filesystem INDEX.MF
// and index.mf are one file.
// compared ignoring case, on every filesystem, since on a
// case-insensitive one A.txt and a.txt are one file.
func checkNoNameClash(files []*mfer.MFFilePath) error {
sep := string(filepath.Separator)
// written maps each name fetch writes, other than the listed files
// themselves, in lower case, to the file it writes there.
// written maps each name fetch writes, in lower case, to the file or
// directory it writes there.
written := map[string]string{
defaultManifestName: "the saved manifest",
tempPathFor(defaultManifestName): "the saved manifest's temp file",
@@ -488,14 +514,26 @@ func checkNoNameClash(files []*mfer.MFFilePath) error {
}
for _, f := range files {
// Look up each directory on the file's path, then the file itself.
parts := strings.Split(strings.ToLower(filepath.Clean(f.GetPath())), sep)
// Look up and add each directory on the file's path, then the
// file itself. Only the same directory, spelled alike, may
// already be there.
parts := strings.Split(filepath.Clean(f.GetPath()), sep)
last := len(parts) - 1
for i := range parts {
what, ok := written[strings.Join(parts[:i+1], sep)]
if ok {
return fmt.Errorf("%w: %s (%s)", errNameClash, f.GetPath(), what)
name := strings.Join(parts[:i+1], sep)
what := "the directory " + name
if i == last {
what = "the file " + f.GetPath()
}
other, ok := written[strings.ToLower(name)]
if ok && (i == last || other != what) {
return fmt.Errorf("%w: %s (%s)", errNameClash, f.GetPath(), other)
}
written[strings.ToLower(name)] = what
}
}
@@ -506,9 +544,7 @@ func checkNoNameClash(files []*mfer.MFFilePath) error {
// exactly as check does. verifyRequiredSigner takes a Checker, which loads
// its manifest from a file, so the manifest is handed to it as a file in
// memory.
func verifyFetchedSigner(
ctx context.Context, manifestData []byte, requiredSigner string,
) error {
func verifyFetchedSigner(manifestData []byte, requiredSigner string) error {
memFs := afero.NewMemMapFs()
manifestPath := "/" + defaultManifestName
@@ -517,17 +553,16 @@ func verifyFetchedSigner(
return err
}
//nolint:contextcheck // mfer loads a manifest without a context
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath,
BasePath: "/",
Fs: memFs,
})
if err != nil {
return fmt.Errorf("failed to load manifest: %w", err)
return fmt.Errorf("load manifest: %w", err)
}
return verifyRequiredSigner(ctx, chk, requiredSigner)
return verifyRequiredSigner(chk, requiredSigner)
}
// saveManifest writes the fetched manifest into dest under the default
@@ -607,7 +642,7 @@ func sanitizePath(p string) (string, error) {
func checkNoSymlinks(dest, p string) error {
current := dest
for _, part := range strings.Split(p, string(filepath.Separator)) {
for part := range strings.SplitSeq(p, string(filepath.Separator)) {
current = filepath.Join(current, part)
info, err := os.Lstat(current)
@@ -616,7 +651,7 @@ func checkNoSymlinks(dest, p string) error {
}
if err != nil {
return fmt.Errorf("failed to check %s for a symlink: %w", current, err)
return err
}
if info.Mode()&os.ModeSymlink != 0 {
@@ -733,7 +768,7 @@ func tempPathFor(localPath string) string {
func verifyDownloadedHash(digest []byte, entry *mfer.MFFilePath) error {
computed, err := multihash.Encode(digest, multihash.SHA2_256)
if err != nil {
return fmt.Errorf("failed to encode hash: %w", err)
return fmt.Errorf("encode hash: %w", err)
}
for _, hash := range entry.GetHashes() {
@@ -760,7 +795,7 @@ func downloadFile(
// so every entry point to downloadFile gets the same treatment.
localPath, err := sanitizePath(localPath)
if err != nil {
return fmt.Errorf("invalid path: %w", err)
return fmt.Errorf("invalid file entry: %w", err)
}
// Create parent directories if needed
@@ -775,7 +810,7 @@ func downloadFile(
err = os.MkdirAll(dir, dirPerms)
if err != nil {
return fmt.Errorf("failed to create directory %s: %w", dir, err)
return err
}
}
@@ -809,7 +844,7 @@ func createTempFile(dest, tmpPath string) (*os.File, error) {
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
path, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
if err != nil {
return nil, fmt.Errorf("failed to create temp file: %w", err)
return nil, err
}
return out, nil
@@ -822,12 +857,7 @@ func moveIntoPlace(dest, tmpPath, localPath string) error {
return err
}
err = os.Rename(filepath.Join(dest, tmpPath), filepath.Join(dest, localPath))
if err != nil {
return fmt.Errorf("failed to rename temp file: %w", err)
}
return nil
return os.Rename(filepath.Join(dest, tmpPath), filepath.Join(dest, localPath))
}
// saveResponse writes resp's body to tmpPath, verifies it against entry,
@@ -853,6 +883,19 @@ func saveResponse(
return err
}
// A recorded mode of 0 means none was recorded. Of any other, only the
// permission bits are set, whatever the umask, so setuid, setgid and
// sticky never are, whichever caller passed the entry.
if entry.GetMode() != 0 {
err = out.Chmod(os.FileMode(entry.GetMode()).Perm())
if err != nil {
_ = out.Close()
_ = os.Remove(filepath.Join(dest, tmpPath))
return err
}
}
// Set up hash computation
h := sha256.New()
@@ -865,8 +908,10 @@ func saveResponse(
progress: progress,
}
// Copy content while hashing and reporting progress
written, copyErr := io.Copy(pw, resp.Body)
// Copy content while hashing and reporting progress. One byte past
// the listed size is enough for finishDownload to report a size
// mismatch.
written, copyErr := io.Copy(pw, io.LimitReader(resp.Body, expectedSize+1))
// Close file before checking errors (to flush writes)
closeErr := out.Close()
internal/cli/fetch_test.go
+358 -22
View File
@@ -4,6 +4,7 @@ package cli
import (
"bytes"
"context"
"crypto/sha256"
"fmt"
"io"
"maps"
@@ -18,10 +19,15 @@ import (
"sync/atomic"
"testing"
"time"
"uuid"
"github.com/klauspost/compress/zstd"
"github.com/multiformats/go-multihash"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v3"
"google.golang.org/protobuf/proto"
"sneak.berlin/go/mfer/mfer"
)
@@ -181,12 +187,7 @@ func chdirTemp(t *testing.T) string {
t.Helper()
destDir := t.TempDir()
origDir, err := os.Getwd()
require.NoError(t, err)
require.NoError(t, os.Chdir(destDir))
t.Cleanup(func() { _ = os.Chdir(origDir) })
t.Chdir(destDir)
return destDir
}
@@ -441,6 +442,76 @@ func TestFetchSizeMismatch(t *testing.T) {
"temp file should be cleaned up on size mismatch")
}
// zeros is an io.Reader of zero bytes without end.
type zeros struct{}
func (zeros) Read(p []byte) (int, error) {
clear(p)
return len(p), nil
}
// TestFetchStopsReadingFilePastListedSize serves a body that never ends
// for a file listed at 16 bytes. fetch must stop reading one byte past
// the listed size, report the size mismatch and remove its temp file.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchStopsReadingFilePastListedSize(t *testing.T) {
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.Copy(w, zeros{})
}))
defer server.Close()
chdirTemp(t)
err := downloadFile(context.Background(), testClient(),
server.URL+"/"+testFileTxt, ".", testFileTxt,
&mfer.MFFilePath{Path: testFileTxt, Size: 16}, nil)
require.ErrorIs(t, err, errSizeMismatch)
require.EqualError(t, err, "size mismatch: expected 16 bytes, got 17")
assert.NoFileExists(t, tempPathFor(testFileTxt))
}
// TestManifestDownloadStopsAtLimit serves a manifest that never ends to
// fetch and to check, with the most they download of a manifest lowered
// to 64 KiB. Each must stop reading at that limit, fail with an error
// naming it and leave no temp file.
func TestManifestDownloadStopsAtLimit(t *testing.T) {
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = io.Copy(w, zeros{})
}))
defer server.Close()
tmpDir := t.TempDir()
t.Setenv("TMPDIR", tmpDir)
mfa := &CLIApp{Fs: afero.NewOsFs(), maxManifestSize: 64 << 10}
fetch := mfa.fetchCommand()
fetch.Action = mfa.fetchManifestOperation
check := mfa.checkCommand()
check.Action = mfa.checkManifestOperation
for _, cmd := range []*urfcli.Command{fetch, check} {
// Both operations log to the process-global logger.
err := runLocked(func() error {
return cmd.Run(context.Background(),
[]string{cmd.Name, server.URL + "/index.mf"})
})
require.ErrorIs(t, err, errManifestTooLarge, cmd.Name)
require.EqualError(t, err,
"download manifest: file exceeds maximum allowed size of 65536 bytes",
cmd.Name)
}
leftover, err := os.ReadDir(tmpDir)
require.NoError(t, err)
assert.Empty(t, leftover)
}
//nolint:paralleltest // changes the process-global working directory
func TestFetchProgress(t *testing.T) {
// Create source filesystem with a larger test file
@@ -535,27 +606,27 @@ func TestFetchRefusesSymlinks(t *testing.T) {
}{
{
"parent directory", "sub/deeper/file.txt", "sub", ".",
"failed to download sub/deeper/file.txt",
"download sub/deeper/file.txt",
},
{
"directory inside a plain directory", "docs/data/passwd", "docs/data", ".",
"failed to download docs/data/passwd",
"download docs/data/passwd",
},
{
"temp file", testFileTxt, ".file.txt.tmp", newFile,
"failed to download " + testFileTxt,
"download " + testFileTxt,
},
{
"file", testFileTxt, testFileTxt, newFile,
"failed to download " + testFileTxt,
"download " + testFileTxt,
},
{
"manifest temp file", testFileTxt, tempPathFor(defaultManifestName), newFile,
"failed to save manifest",
"save manifest",
},
{
"manifest", testFileTxt, defaultManifestName, newFile,
"failed to save manifest",
"save manifest",
},
}
@@ -622,7 +693,7 @@ func TestFetchDoesNotSkipThroughSymlink(t *testing.T) {
}, afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
"failed to download sub/"+testFileTxt+": symlink in path not allowed: "+link)
"download sub/"+testFileTxt+": symlink in path not allowed: "+link)
assert.Equal(t, map[string][]byte{testFileTxt: content}, filesUnder(t, outside))
}
@@ -692,7 +763,8 @@ func TestGetRetriesTransientStatusesOnly(t *testing.T) {
err := getNothing(testClient(), server.URL, 10*time.Second)
require.ErrorIs(t, err, errHTTPStatus)
require.EqualError(t, err, fmt.Sprintf("HTTP %d", tt.status))
require.EqualError(t, err,
fmt.Sprintf("unexpected HTTP status %d", tt.status))
assert.Equal(t, tt.requests, requests.Load())
})
}
@@ -1117,25 +1189,28 @@ func TestFetchIntoDest(t *testing.T) {
// TestFetchRequireSignature runs fetch with --require-signature. A
// manifest that is unsigned, or signed by another key, must stop fetch
// with check's message before it downloads or writes anything; the
// required key lets it through. The signed cases need gpg and are skipped
// without it, as the other signing tests are.
//
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
// required key lets it through. A manifest signed by another key whose
// embedded public key block also holds the required key must stop fetch
// too.
func TestFetchRequireSignature(t *testing.T) {
t.Parallel()
files := map[string][]byte{testFileTxt: []byte("signed file")}
t.Run("unsigned", func(t *testing.T) {
t.Parallel()
assertFetchRefused(t, manifestOf(t, files), files,
"manifest is not signed, but signature from "+msgFpA+" is required",
"--"+flagRequireSignature, msgFpA)
})
t.Run("signed", func(t *testing.T) {
t.Parallel()
manifest := signedManifest(t, files)
signer, err := signedChecker(t, manifest).
ExtractEmbeddedSigningKeyFP(context.Background())
require.NoError(t, err)
signer := string(signedChecker(t, manifest).Signer())
assertFetchRefused(t, manifest, files,
"embedded signing key fingerprint "+signer+" does not match required "+msgFpB,
@@ -1153,6 +1228,17 @@ func TestFetchRequireSignature(t *testing.T) {
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
assert.Equal(t, files[testFileTxt], filesUnder(t, dest)[testFileTxt])
})
t.Run("signed by another key embedded after the required one", func(t *testing.T) {
t.Parallel()
manifest, required := manifestSignedByAnotherKey(t, files)
assertFetchRefused(t, manifest, files,
"parse manifest: "+
"embedded public key block must hold exactly one key, found 2",
"--"+flagRequireSignature, required)
})
}
// TestFetchRefusesListedManifestName fetches manifests that list, at the
@@ -1236,6 +1322,256 @@ func TestFetchRefusesListedTempName(t *testing.T) {
}
}
// TestFetchRefusesNamesEqualIgnoringCase fetches manifests that list two
// paths that are one name on a case-insensitive filesystem. Fetched
// there, of two such files, at the top of the tree or in a directory,
// one replaces the other and fetch exits 0. A file and a directory
// another file is in stop fetch partway with a non-zero exit, leaving a
// partial tree. Two spellings of one directory put both files in one
// directory, one of them under a spelling the manifest does not list,
// and check reports that file as not in the manifest. So fetch must
// refuse each on every filesystem before it creates the destination or
// requests any file. A file and a directory with the same name, and a
// file listed twice, are refused the same way. A manifest whose names
// differ in more than letter case is fetched in full.
func TestFetchRefusesNamesEqualIgnoringCase(t *testing.T) {
t.Parallel()
for _, tc := range []struct{ first, second, message string }{
{"X.txt", "x.txt", "x.txt (the file X.txt)"},
{"sub/X.txt", "sub/x.txt", "sub/x.txt (the file sub/X.txt)"},
{"Dir", "dir/x", "dir/x (the file Dir)"},
{"Dir/a.txt", "dir/b.txt", "dir/b.txt (the directory Dir)"},
{"dir", "dir/x", "dir/x (the file dir)"},
{"./x.txt", "x.txt", "x.txt (the file ./x.txt)"},
} {
t.Run(tc.first+" and "+tc.second, func(t *testing.T) {
t.Parallel()
files := map[string][]byte{
tc.first: []byte("the first file"),
tc.second: []byte("the second file"),
}
assertFetchRefused(t, builtManifest(t, files), files,
"manifest lists a file where fetch writes another file: "+tc.message)
})
}
t.Run("names that differ in more than letter case", func(t *testing.T) {
t.Parallel()
files := map[string][]byte{
"A.txt": []byte("at the top"),
"B.txt": []byte("also at the top"),
"dir/a.txt": []byte("in a directory"),
"dir/b.txt": []byte("in the same directory"),
}
manifest := builtManifest(t, files)
server := httptest.NewServer(fetchTestHandler(manifest, files))
defer server.Close()
dest := t.TempDir()
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
want := maps.Clone(files)
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
})
}
// TestFetchSetsRecordedMode fetches a tree whose manifest records the
// modes 0640 and 0755. Each file must get its mode whatever the umask, and
// check must pass on the result. After one file's mode is changed, a
// second fetch must download that file again, and only it, to restore its
// mode.
func TestFetchSetsRecordedMode(t *testing.T) {
t.Parallel()
files := map[string][]byte{
testFileTxt: []byte("a file"),
"tool.sh": []byte("#!/bin/sh\n"),
}
modes := map[string]os.FileMode{testFileTxt: 0o640, "tool.sh": 0o755}
sourceFs := afero.NewMemMapFs()
for p, content := range files {
require.NoError(t, afero.WriteFile(sourceFs, "/"+p, content, modes[p]))
}
scanner := mfer.NewScannerWithOptions(&mfer.ScannerOptions{
Fs: sourceFs,
IncludePermissions: true,
})
require.NoError(t, scanner.EnumerateFS(sourceFs, "/", nil))
var manifest bytes.Buffer
require.NoError(t, scanner.ToManifest(context.Background(), &manifest, nil))
tree := fetchTestHandler(manifest.Bytes(), files)
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
dest := t.TempDir()
fetch := []string{testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL}
opts := testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
for p, mode := range modes {
info, err := os.Stat(filepath.Join(dest, p))
require.NoError(t, err)
assert.Equal(t, mode, info.Mode().Perm(), p)
}
opts = testOpts([]string{
testApp, cmdCheck, "-q", testFlagBase, dest,
filepath.Join(dest, defaultManifestName),
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
require.NoError(t, os.Chmod(filepath.Join(dest, testFileTxt), 0o600))
mu.Lock()
requested = nil
mu.Unlock()
opts = testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
info, err := os.Stat(filepath.Join(dest, testFileTxt))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o640), info.Mode().Perm())
mu.Lock()
defer mu.Unlock()
assert.ElementsMatch(t,
[]string{"/" + defaultManifestName, "/" + testFileTxt}, requested)
}
// TestFetchRefusesModeOutsidePermissionBits fetches manifests that record
// a mode with the setuid bit, once as Unix writes it (04755) and once as
// Go keeps it in os.FileMode. fetch must refuse both before it creates the
// destination or requests any file.
func TestFetchRefusesModeOutsidePermissionBits(t *testing.T) {
t.Parallel()
files := map[string][]byte{testFileTxt: []byte("a file")}
assertFetchRefused(t, manifestWithMode(t, testFileTxt, files[testFileTxt], 0o4755),
files, "manifest lists a mode outside 0777: file.txt (04755)")
assertFetchRefused(t,
manifestWithMode(t, testFileTxt, files[testFileTxt],
uint32(os.ModeSetuid|0o755)),
files, "manifest lists a mode outside 0777: file.txt (040000755)")
}
// TestDownloadFileSetsOnlyPermissionBits downloads a file whose entry
// records mode 0755 together with setuid, setgid or sticky, as Go keeps
// them in os.FileMode. fetchManifest would refuse such an entry; called
// directly, downloadFile must still set only the permission bits.
func TestDownloadFileSetsOnlyPermissionBits(t *testing.T) {
t.Parallel()
content := []byte("#!/bin/sh\n")
digest := sha256.Sum256(content)
hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
require.NoError(t, err)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write(content)
}))
defer server.Close()
for _, special := range []os.FileMode{os.ModeSetuid, os.ModeSetgid, os.ModeSticky} {
entry := &mfer.MFFilePath{
Path: testFileTxt,
Size: int64(len(content)),
Hashes: []*mfer.MFFileChecksum{{MultiHash: hash}},
Mode: uint32(special | 0o755),
}
dest := t.TempDir()
err := downloadFile(context.Background(), testClient(),
server.URL+"/"+testFileTxt, dest, testFileTxt, entry, nil)
require.NoError(t, err, special)
info, err := os.Stat(filepath.Join(dest, testFileTxt))
require.NoError(t, err)
assert.Equal(t, os.FileMode(0o755), info.Mode(), special)
}
}
// manifestWithMode returns a manifest listing one file, path, with content
// and the given recorded mode. It is assembled by hand, since the builder
// never records a mode outside 0777.
func manifestWithMode(t *testing.T, path string, content []byte, mode uint32) []byte {
t.Helper()
digest := sha256.Sum256(content)
hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
require.NoError(t, err)
id := uuid.NewV4()
inner, err := proto.Marshal(&mfer.MFFile{
Version: mfer.MFFile_VERSION_ONE,
Files: []*mfer.MFFilePath{{
Path: path,
Size: int64(len(content)),
Hashes: []*mfer.MFFileChecksum{{MultiHash: hash}},
Mode: mode,
}},
Uuid: id[:],
})
require.NoError(t, err)
encoder, err := zstd.NewWriter(nil)
require.NoError(t, err)
compressed := encoder.EncodeAll(inner, nil)
require.NoError(t, encoder.Close())
sum := sha256.Sum256(compressed)
outer, err := proto.Marshal(&mfer.MFFileOuter{
Version: mfer.MFFileOuter_VERSION_ONE,
CompressionType: mfer.MFFileOuter_COMPRESSION_ZSTD,
Size: int64(len(inner)),
Sha256: sum[:],
Uuid: id[:],
InnerMessage: compressed,
})
require.NoError(t, err)
return append([]byte(mfer.MAGIC), outer...)
}
// builtManifest returns a manifest of files, built directly rather than
// scanned, since a scan lists no hidden files and never a path starting
// with "./".
@@ -1246,7 +1582,7 @@ func builtManifest(t *testing.T, files map[string][]byte) []byte {
for p, content := range files {
_, err := builder.AddFile(mfer.RelFilePath(p), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
mfer.ModTime(time.Now()), 0, bytes.NewReader(content), nil)
require.NoError(t, err)
}
internal/cli/freshen.go
+67 -52
View File
@@ -48,6 +48,7 @@ type freshenEntry struct {
path string
size int64
mtime time.Time
mode fs.FileMode // mode to record, 0 for none
needsHash bool // true if new or changed
existing *mfer.MFFilePath // existing manifest entry if unchanged
}
@@ -60,6 +61,7 @@ type freshenScanner struct {
excluded []fs.FileInfo // files left out of the listing
includeDotfiles bool
followSymlinks bool
includePermissions bool
showProgress bool
existingByPath map[string]*mfer.MFFilePath
@@ -93,6 +95,12 @@ func (s *freshenScanner) resolveSymlink(path string) (fs.FileInfo, bool) {
// recordEntry classifies a scanned file as changed, unchanged, or added
// relative to the existing manifest.
func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
// A mode of 0 records 0000, which means none was recorded.
var mode fs.FileMode
if s.includePermissions {
mode = info.Mode().Perm()
}
existing, inManifest := s.existingByPath[relPath]
if !inManifest {
s.added++
@@ -102,16 +110,17 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath,
size: info.Size(),
mtime: info.ModTime(),
mode: mode,
needsHash: true,
})
return
}
// Check if changed (size or mtime). An entry with no recorded mtime
// cannot be compared, so it counts as changed and gets re-hashed;
// silently treating the absent mtime as the Unix epoch would classify
// every such entry as changed without saying why.
// Check if changed (size, mtime, or the mode to record). An entry
// with no recorded mtime cannot be compared, so it counts as changed
// and gets re-hashed; silently treating the absent mtime as the Unix
// epoch would classify every such entry as changed without saying why.
existingMtime, haveMtime := entryMtime(existing)
if !haveMtime {
log.Debugf("%s: manifest entry has no mtime, treating as changed",
@@ -119,7 +128,8 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
}
if !haveMtime || existing.GetSize() != info.Size() ||
!existingMtime.Equal(info.ModTime()) {
!existingMtime.Equal(info.ModTime()) ||
fs.FileMode(existing.GetMode()) != mode {
s.changed++
log.Verbosef("M %s", relPath)
@@ -127,6 +137,7 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath,
size: info.Size(),
mtime: info.ModTime(),
mode: mode,
needsHash: true,
})
} else {
@@ -136,6 +147,7 @@ func (s *freshenScanner) recordEntry(relPath string, info fs.FileInfo) {
path: relPath,
size: info.Size(),
mtime: info.ModTime(),
mode: mode,
needsHash: false,
existing: existing,
})
@@ -153,8 +165,7 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
// Get relative path
relPath, err := filepath.Rel(s.absBase, path)
if err != nil {
return fmt.Errorf(
"freshen: failed to compute relative path for %s: %w", path, err)
return err
}
// Handle dotfiles
@@ -268,13 +279,8 @@ func (h *freshenHasher) reportProgress(n int64) {
// processEntry hashes the entry if needed and adds it to the builder.
func (h *freshenHasher) processEntry(e *freshenEntry) error {
if !e.needsHash {
// Use existing entry
err := addExistingToBuilder(h.builder, e.existing)
if err != nil {
return fmt.Errorf("failed to add %s: %w", e.path, err)
}
return nil
// Use existing entry; the error names the entry
return addExistingToBuilder(h.builder, e.existing)
}
// Need to read and hash the file
@@ -282,26 +288,21 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
f, err := h.fs.Open(absPath)
if err != nil {
return fmt.Errorf("failed to open %s: %w", e.path, err)
return err
}
hash, bytesRead, err := hashFile(f, h.reportProgress)
_ = f.Close()
if err != nil {
return fmt.Errorf("failed to hash %s: %w", e.path, err)
return err
}
h.hashedBytes += bytesRead
h.hashedFiles++
// Add to builder with computed hash
err = addFileToBuilder(h.builder, e.path, e.size, e.mtime, hash)
if err != nil {
return fmt.Errorf("failed to add %s: %w", e.path, err)
}
return nil
// Add to builder with computed hash; a refused path is named in the error
return addFileToBuilder(h.builder, e.path, e.size, e.mtime, e.mode, hash)
}
// writeFreshenedManifest writes the manifest atomically (write to a
@@ -313,7 +314,7 @@ func writeFreshenedManifest(
outFile, err := afs.Create(tmpPath)
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
return err
}
err = builder.Build(ctx, outFile)
@@ -322,7 +323,7 @@ func writeFreshenedManifest(
if err != nil {
_ = afs.Remove(tmpPath)
return fmt.Errorf("failed to write manifest: %w", err)
return fmt.Errorf("build manifest: %w", err)
}
// Rename temp to final
@@ -330,7 +331,7 @@ func writeFreshenedManifest(
if err != nil {
_ = afs.Remove(tmpPath)
return fmt.Errorf("failed to rename manifest: %w", err)
return err
}
return nil
@@ -338,7 +339,7 @@ func writeFreshenedManifest(
// newFreshenBuilder constructs the manifest builder configured from CLI
// flags.
func newFreshenBuilder(cmd *cli.Command) *mfer.Builder {
func (mfa *CLIApp) newFreshenBuilder(cmd *cli.Command) (*mfer.Builder, error) {
builder := mfer.NewBuilder()
if cmd.Bool("include-timestamps") {
builder.SetIncludeTimestamps(true)
@@ -346,13 +347,15 @@ func newFreshenBuilder(cmd *cli.Command) *mfer.Builder {
// Set up signing options if sign-key is provided
if signKey := cmd.String("sign-key"); signKey != "" {
builder.SetSigningOptions(&mfer.SigningOptions{
KeyID: mfer.GPGKeyID(signKey),
})
log.Infof("signing manifest with GPG key: %s", signKey)
signing, err := mfa.signingOptions(signKey)
if err != nil {
return nil, err
}
return builder
builder.SetSigningOptions(signing)
}
return builder, nil
}
// freshenScan runs the scan phase against the loaded manifest entries
@@ -378,24 +381,33 @@ func (mfa *CLIApp) freshenScan(
}
}
// The walk does not follow a symlink at its top, so a base directory
// named through one is resolved first. If that fails, the base is
// walked as named and the walk reports the problem.
resolved, err := filepath.EvalSymlinks(absBase)
if err == nil {
absBase = resolved
}
scanner := &freshenScanner{
fs: mfa.Fs,
absBase: absBase,
excluded: excluded,
includeDotfiles: cmd.Bool("include-dotfiles"),
followSymlinks: cmd.Bool("follow-symlinks"),
includePermissions: cmd.Bool(flagIncludePermissions),
showProgress: showProgress,
existingByPath: existingByPath,
}
err := afero.Walk(mfa.Fs, absBase, scanner.walk)
err = afero.Walk(mfa.Fs, absBase, scanner.walk)
if showProgress {
log.ProgressDone()
}
if err != nil {
return nil, 0, fmt.Errorf("failed to scan filesystem: %w", err)
return nil, 0, fmt.Errorf("scan filesystem: %w", err)
}
// Remaining entries in existingByPath are removed files
@@ -431,7 +443,7 @@ func hashTotals(entries []*freshenEntry) (int64, int64) {
}
// runFreshenHash processes every entry through the hasher, aborting if
// the context is canceled.
// the context is canceled, and ends the hasher's progress line.
func runFreshenHash(
ctx context.Context, hasher *freshenHasher, entries []*freshenEntry,
) error {
@@ -448,6 +460,10 @@ func runFreshenHash(
}
}
if hasher.showProgress && hasher.filesToHash > 0 {
log.ProgressDone()
}
return nil
}
@@ -464,7 +480,7 @@ func (mfa *CLIApp) loadExistingEntries(
Fs: mfa.Fs,
})
if err != nil {
return nil, fmt.Errorf("failed to load manifest: %w", err)
return nil, fmt.Errorf("load manifest: %w", err)
}
existingFiles := manifest.Files()
@@ -484,24 +500,27 @@ func (mfa *CLIApp) freshenManifestOperation(
) error {
log.Debug("freshenManifestOperation()")
basePath := cmd.String("base")
showProgress := cmd.Bool("progress")
// Find manifest file
manifestPath, err := mfa.resolveFreshenManifestPath(cmd)
if err != nil {
return fmt.Errorf("freshen: %w", err)
return err
}
builder, err := mfa.newFreshenBuilder(cmd)
if err != nil {
return err
}
//nolint:contextcheck // mfer loads a manifest without a context
existingByPath, err := mfa.loadExistingEntries(manifestPath)
if err != nil {
return err
}
absBase, err := filepath.Abs(basePath)
absBase, err := filepath.Abs(resolveBasePath(cmd, manifestPath))
if err != nil {
return fmt.Errorf("freshen: invalid base path: %w", err)
return fmt.Errorf("invalid base path: %w", err)
}
// Phase 1: Scan filesystem
@@ -527,7 +546,7 @@ func (mfa *CLIApp) freshenManifestOperation(
totalHashBytes: totalHashBytes,
filesToHash: filesToHash,
startHash: time.Now(),
builder: newFreshenBuilder(cmd),
builder: builder,
}
err = runFreshenHash(ctx, hasher, scanner.entries)
@@ -535,10 +554,6 @@ func (mfa *CLIApp) freshenManifestOperation(
return err
}
if showProgress && filesToHash > 0 {
log.ProgressDone()
}
// Print summary
log.Infof("freshen complete: %d unchanged, %d changed, %d added, %d removed",
scanner.unchanged, scanner.changed, scanner.added, removed)
@@ -593,9 +608,7 @@ func hashFile(r io.Reader, progress func(int64)) ([]byte, int64, error) {
break
}
// Returned unwrapped: the caller renders this as
// "failed to hash <path>: <err>" and adding a second layer here
// would change that message.
// Returned unwrapped: a read error already names the file.
if err != nil {
return nil, total, err
}
@@ -611,10 +624,12 @@ func hashFile(r io.Reader, progress func(int64)) ([]byte, int64, error) {
// addFileToBuilder adds a new file entry to the builder
func addFileToBuilder(
b *mfer.Builder, path string, size int64, mtime time.Time, hash []byte,
b *mfer.Builder, path string, size int64, mtime time.Time, mode fs.FileMode,
hash []byte,
) error {
return b.AddFileWithHash(
mfer.RelFilePath(path), mfer.FileSize(size), mfer.ModTime(mtime), hash)
mfer.RelFilePath(path), mfer.FileSize(size), mfer.ModTime(mtime), mode,
hash)
}
// addExistingToBuilder adds an existing manifest entry to the builder.
@@ -634,7 +649,7 @@ func addExistingToBuilder(b *mfer.Builder, entry *mfer.MFFilePath) error {
err := b.AddFileWithHash(mfer.RelFilePath(entry.GetPath()),
mfer.FileSize(entry.GetSize()), mfer.ModTime(mtime),
entry.GetHashes()[0].GetMultiHash())
fs.FileMode(entry.GetMode()), entry.GetHashes()[0].GetMultiHash())
if err != nil {
return fmt.Errorf(
"manifest entry %s: %w (regenerate the manifest with mfer generate)",
internal/cli/freshen_test.go
+115
View File
@@ -7,6 +7,7 @@ import (
"os"
"path/filepath"
"slices"
"strings"
"testing"
"time"
@@ -99,6 +100,42 @@ func TestFreshenUnchanged(t *testing.T) {
}
}
// TestFreshenRecordsModeOnlyWhenAsked changes only the modes of a tree
// after gen made its manifest, which recorded every mode as 0000. freshen
// --include-permissions must record each file's permission bits, and a
// later freshen without it must record 0000 again, although no file's
// content or mtime changed.
func TestFreshenRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewOsFs()
root, manifestPath := setupFreshenDir(t, fs,
map[string]string{testFileTxt: "a file", testDirFile: "a file in dir"})
require.NoError(t, fs.Chmod(filepath.Join(root, testFileTxt), 0o640))
require.NoError(t, fs.Chmod(filepath.Join(root, testDirFile), 0o755))
recordedModes := func() map[string]uint32 {
modes := map[string]uint32{}
for _, f := range manifestFiles(t, fs, manifestPath) {
modes[f.GetPath()] = f.GetMode()
}
return modes
}
opts := testOpts([]string{
testApp, cmdFreshen, "-q", "--" + flagIncludePermissions,
testFlagBase, root, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Equal(t, map[string]uint32{testFileTxt: 0o640, testDirFile: 0o755},
recordedModes())
runFreshen(t, fs, root, manifestPath)
assert.Equal(t, map[string]uint32{testFileTxt: 0, testDirFile: 0},
recordedModes())
}
// assertManifestLists asserts that the manifest at manifestPath lists
// exactly the files in want, each with the size and SHA-256 hash of its
// content in want and the mtime of the file of that name under root.
@@ -264,6 +301,84 @@ func TestFreshenLeavesLeftoverTempFileOutOfListing(t *testing.T) {
manifestPaths(t, fs, manifestPath))
}
// TestFreshenResolvesEntriesAgainstManifestDirectory adds sub/c.txt, then
// runs freshen from the directory above sub, on the manifest in sub.
// Without --base, the manifest then lists the files in sub, whether freshen
// is given the manifest or sub. --base names the directory to list instead,
// the current one included.
//
//nolint:paralleltest // changes the process-global working directory
func TestFreshenResolvesEntriesAgainstManifestDirectory(t *testing.T) {
for _, tc := range []struct {
args []string
want []string // the paths the manifest lists afterwards
}{
{[]string{testSubdir}, []string{"b.txt", "c.txt"}},
{[]string{testSubdirManifest}, []string{"b.txt", "c.txt"}},
{
[]string{testFlagBase, ".", testSubdirManifest},
[]string{testFileTxt, "sub/b.txt", "sub/c.txt"},
},
} {
t.Run(strings.Join(tc.args, " "), func(t *testing.T) {
fs := afero.NewOsFs()
root := setupManifestInSubdir(t)
writeTestFile(t, fs, filepath.Join(root, testSubdir, "c.txt"), "added")
opts := testOpts(slices.Concat(
[]string{testApp, cmdFreshen, "-q"}, tc.args,
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, tc.want, manifestPaths(t, fs,
filepath.Join(root, testSubdirManifest)))
})
}
}
// TestFreshenDirectoryNamedThroughSymlink adds a file to a tree after gen
// made its manifest, then freshens it with the tree named through a symlink,
// given as the argument or as the working directory: the manifest lists the
// files in the tree, and leaves out a symlink inside it, as freshen does
// without --follow-symlinks.
//
//nolint:paralleltest // changes the process-global working directory
func TestFreshenDirectoryNamedThroughSymlink(t *testing.T) {
// Paths are relative to a temp dir holding link, a symlink to the tree.
for name, tc := range map[string]struct {
workDir string
args []string
}{
"argument": {".", []string{testLink}},
"working directory": {testLink, nil},
} {
t.Run(name, func(t *testing.T) {
fs := afero.NewOsFs()
tree, manifestPath := setupFreshenDir(t, fs,
map[string]string{testFileTxt: "in the tree"})
writeTestFile(t, fs, filepath.Join(tree, "later.txt"), "added later")
require.NoError(t,
os.Symlink(testFileTxt, filepath.Join(tree, "alias.txt")))
root := t.TempDir()
require.NoError(t, os.Symlink(tree, filepath.Join(root, testLink)))
// t.Chdir sets PWD to the path it is given, as a shell does, and
// os.Getwd returns PWD when it names the working directory.
t.Chdir(filepath.Join(root, tc.workDir))
opts := testOpts(slices.Concat(
[]string{testApp, cmdFreshen, "-q"}, tc.args,
), fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assertManifestLists(t, fs, tree, manifestPath, map[string]string{
testFileTxt: "in the tree", "later.txt": "added later",
})
})
}
}
// TestFreshenRecordEntryMtimePresence pins the behavior of recordEntry
// with respect to MFFilePath.Mtime, which is a message pointer with
// proto3 field presence and may legitimately be absent.
internal/cli/gen.go
+94 -38
View File
@@ -4,6 +4,7 @@ import (
"context"
"errors"
"fmt"
"io"
"os"
"os/signal"
"path/filepath"
@@ -26,6 +27,8 @@ var (
// rendered message stays exactly as mfer has always printed it.
errOutputExists = errors.New(
"already exists (use --force to overwrite)")
// errEmptyOutput indicates --output given with an empty value.
errEmptyOutput = errors.New("--output must not be empty")
)
// reportEnumProgress renders enumeration progress until the channel
@@ -74,7 +77,7 @@ func (mfa *CLIApp) collectInputPaths(args cli.Args) ([]string, error) {
ap, err := filepath.Abs(inputPath)
if err != nil {
return nil, fmt.Errorf("generate: invalid path %q: %w", inputPath, err)
return nil, fmt.Errorf("invalid path %q: %w", inputPath, err)
}
// Validate path exists before adding to list
if exists, _ := afero.Exists(mfa.Fs, ap); !exists {
@@ -88,13 +91,45 @@ func (mfa *CLIApp) collectInputPaths(args cli.Args) ([]string, error) {
return paths, nil
}
// buildScannerOptions constructs scanner options from the CLI flags.
func (mfa *CLIApp) buildScannerOptions(cmd *cli.Command) *mfer.ScannerOptions {
// outputPath returns the file gen writes the manifest to: the one --output
// names, or else index.mf in the directory the only argument names, or
// beside the file it names, or else in the current directory. An --output
// given with an empty value is refused.
func (mfa *CLIApp) outputPath(cmd *cli.Command) (string, error) {
if cmd.IsSet("output") {
output := cmd.String("output")
if output == "" {
return "", errEmptyOutput
}
return output, nil
}
if cmd.Args().Len() != 1 {
return defaultManifestName, nil
}
arg := cmd.Args().First()
// A path that does not exist is refused when it is enumerated.
info, err := mfa.Fs.Stat(arg)
if err == nil && !info.IsDir() {
return filepath.Join(filepath.Dir(arg), defaultManifestName), nil
}
return filepath.Join(arg, defaultManifestName), nil
}
// buildScannerOptions constructs scanner options from the CLI flags and
// the path the manifest is written to.
func (mfa *CLIApp) buildScannerOptions(
cmd *cli.Command, output string,
) (*mfer.ScannerOptions, error) {
opts := &mfer.ScannerOptions{
IncludeDotfiles: cmd.Bool("include-dotfiles"),
FollowSymLinks: cmd.Bool("follow-symlinks"),
IncludeTimestamps: cmd.Bool("include-timestamps"),
IncludePermissions: cmd.Bool(flagIncludePermissions),
Fs: mfa.Fs,
// Neither a manifest being replaced nor a temp file left by an
// interrupted run belongs in the new manifest.
@@ -110,13 +145,15 @@ func (mfa *CLIApp) buildScannerOptions(cmd *cli.Command) *mfer.ScannerOptions {
// Set up signing options if sign-key is provided
if signKey := cmd.String("sign-key"); signKey != "" {
opts.SigningOptions = &mfer.SigningOptions{
KeyID: mfer.GPGKeyID(signKey),
}
log.Infof("signing manifest with GPG key: %s", signKey)
signing, err := mfa.signingOptions(signKey)
if err != nil {
return nil, err
}
return opts
opts.SigningOptions = signing
}
return opts, nil
}
// enumerateInputs runs the enumeration phase over the argument paths,
@@ -128,8 +165,7 @@ func (mfa *CLIApp) enumerateInputs(
// Default to current directory
err := s.EnumeratePath(".", enumProgress)
if err != nil {
return fmt.Errorf(
"generate: failed to enumerate current directory: %w", err)
return fmt.Errorf("enumerate current directory: %w", err)
}
return nil
@@ -143,7 +179,7 @@ func (mfa *CLIApp) enumerateInputs(
err = s.EnumeratePaths(enumProgress, paths...)
if err != nil {
return fmt.Errorf("generate: failed to enumerate paths: %w", err)
return fmt.Errorf("enumerate files: %w", err)
}
return nil
@@ -202,23 +238,59 @@ func (mfa *CLIApp) runEnumeratePhase(cmd *cli.Command, s *mfer.Scanner) error {
return nil
}
// runScanPhase reads the enumerated files and writes the manifest to out,
// with optional progress reporting.
func (mfa *CLIApp) runScanPhase(
ctx context.Context, cmd *cli.Command, s *mfer.Scanner, out io.Writer,
) error {
var (
scanProgress chan mfer.ScanStatus
scanWg sync.WaitGroup
)
if cmd.Bool("progress") {
scanProgress = make(chan mfer.ScanStatus, 1)
scanWg.Add(1)
go reportScanProgress(scanProgress, &scanWg)
}
err := s.ToManifest(ctx, out, scanProgress)
scanWg.Wait()
if err != nil {
return fmt.Errorf("generate manifest: %w", err)
}
return nil
}
func (mfa *CLIApp) generateManifestOperation(
ctx context.Context, cmd *cli.Command,
) error {
log.Debug("generateManifestOperation()")
s := mfer.NewScannerWithOptions(mfa.buildScannerOptions(cmd))
// Phase 1: Enumeration - collect paths and stat files
err := mfa.runEnumeratePhase(cmd, s)
outputPath, err := mfa.outputPath(cmd)
if err != nil {
return err
}
showProgress := cmd.Bool("progress")
opts, err := mfa.buildScannerOptions(cmd, outputPath)
if err != nil {
return err
}
s := mfer.NewScannerWithOptions(opts)
// Phase 1: Enumeration - collect paths and stat files
err = mfa.runEnumeratePhase(cmd, s)
if err != nil {
return err
}
// Check if output file exists
outputPath := cmd.String("output")
if exists, _ := afero.Exists(mfa.Fs, outputPath); exists && !cmd.Bool("force") {
return fmt.Errorf("output file %s %w", outputPath, errOutputExists)
}
@@ -228,7 +300,7 @@ func (mfa *CLIApp) generateManifestOperation(
outFile, err := mfa.Fs.Create(tmpPath)
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
return err
}
// Set up signal handler to clean up temp file on Ctrl-C
@@ -249,37 +321,21 @@ func (mfa *CLIApp) generateManifestOperation(
}()
// Phase 2: Scan - read file contents and generate manifest
var (
scanProgress chan mfer.ScanStatus
scanWg sync.WaitGroup
)
if showProgress {
scanProgress = make(chan mfer.ScanStatus, 1)
scanWg.Add(1)
go reportScanProgress(scanProgress, &scanWg)
}
err = s.ToManifest(ctx, outFile, scanProgress)
scanWg.Wait()
err = mfa.runScanPhase(ctx, cmd, s, outFile)
if err != nil {
return fmt.Errorf("failed to generate manifest: %w", err)
return err
}
// Close file before rename to ensure all data is flushed
err = outFile.Close()
if err != nil {
return fmt.Errorf("failed to close temp file: %w", err)
return err
}
// Atomic rename
err = mfa.Fs.Rename(tmpPath, outputPath)
if err != nil {
return fmt.Errorf("failed to rename temp file: %w", err)
return err
}
success = true
internal/cli/list.go
+5 -6
View File
@@ -19,20 +19,19 @@ func (mfa *CLIApp) listManifestOperation(ctx context.Context, cmd *cli.Command)
pathOrURL, err := mfa.resolveManifestArg(cmd)
if err != nil {
return fmt.Errorf("list: %w", err)
return err
}
rc, err := mfa.openManifestReader(ctx, pathOrURL)
if err != nil {
return fmt.Errorf("list: %w", err)
return err
}
defer func() { _ = rc.Close() }()
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(rc)
if err != nil {
return fmt.Errorf("list: failed to parse manifest: %w", err)
return fmt.Errorf("parse manifest: %w", err)
}
files := manifest.Files()
@@ -52,8 +51,8 @@ func (mfa *CLIApp) listManifestOperation(ctx context.Context, cmd *cli.Command)
mtimeStr = mtime.Format(time.RFC3339)
}
_, _ = fmt.Fprintf(mfa.Stdout, "%d\t%s\t%s%s",
f.GetSize(), mtimeStr, f.GetPath(), lineEnd)
_, _ = fmt.Fprintf(mfa.Stdout, "%04o\t%d\t%s\t%s%s",
f.GetMode(), f.GetSize(), mtimeStr, f.GetPath(), lineEnd)
} else {
_, _ = fmt.Fprintf(mfa.Stdout, "%s%s", f.GetPath(), lineEnd)
}
internal/cli/manifest_loader.go
+27 -10
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"io"
"net/http"
"path/filepath"
"strings"
"time"
@@ -15,13 +16,13 @@ import (
// manifestFetchTimeout bounds HTTP requests made to fetch a manifest.
const manifestFetchTimeout = 30 * time.Second
// errHTTPStatus indicates an HTTP response with a non-OK status code.
//
// Its text is the literal "HTTP" prefix of the rendered "HTTP <code>"
// message that mfer has always printed, so that wrapping it does not
// change any user-visible output. Match it with errors.Is; do not read
// its message.
var errHTTPStatus = errors.New("HTTP")
// errHTTPStatus indicates an HTTP response with a non-OK status code. It is
// followed by the code, as in "unexpected HTTP status 404".
var errHTTPStatus = errors.New("unexpected HTTP status")
// errManifestTooLarge indicates a manifest download that passed
// CLIApp.maxManifestSize.
var errManifestTooLarge = errors.New("file exceeds maximum allowed size")
// isHTTPURL returns true if the string starts with http:// or https://.
func isHTTPURL(s string) bool {
@@ -36,20 +37,22 @@ func (mfa *CLIApp) openManifestReader(
if isHTTPURL(pathOrURL) {
client := &http.Client{Timeout: manifestFetchTimeout}
// The *url.Error that NewRequestWithContext and Do return names
// the URL.
req, err := http.NewRequestWithContext(ctx, http.MethodGet, pathOrURL, nil)
if err != nil {
return nil, fmt.Errorf("failed to fetch %s: %w", pathOrURL, err)
return nil, fmt.Errorf("download manifest: %w", err)
}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("failed to fetch %s: %w", pathOrURL, err)
return nil, fmt.Errorf("download manifest: %w", err)
}
if resp.StatusCode != http.StatusOK {
_ = resp.Body.Close()
return nil, fmt.Errorf("failed to fetch %s: %w %d",
return nil, fmt.Errorf("download manifest %s: %w %d",
pathOrURL, errHTTPStatus, resp.StatusCode)
}
@@ -84,3 +87,17 @@ func (mfa *CLIApp) resolveManifestArg(cmd *cli.Command) (string, error) {
return findManifest(mfa.Fs, ".")
}
// resolveBasePath returns the directory a manifest's paths are resolved
// against: the one --base names, or else the directory holding the manifest,
// or the current directory for a manifest URL.
func resolveBasePath(cmd *cli.Command, manifestPath string) string {
switch {
case cmd.IsSet(flagBase):
return cmd.String(flagBase)
case isHTTPURL(manifestPath):
return "."
default:
return filepath.Dir(manifestPath)
}
}
internal/cli/mfer.go
+53 -17
View File
@@ -21,12 +21,15 @@ const (
cmdFreshen = "freshen"
cmdExport = "export"
cmdFetch = "fetch"
cmdList = "list"
cmdVersion = "version"
flagBase = "base"
flagProgress = "progress"
flagTimeout = "timeout"
flagDest = "dest"
flagRequireSignature = "require-signature"
flagIncludePermissions = "include-permissions"
manifestArgsUsage = "[manifest file]"
@@ -57,6 +60,10 @@ type CLIApp struct {
exitCode int
app *cli.Command
// maxManifestSize is the most of a manifest that fetch, and check
// given a URL, download: mfer.MaxManifestSize, which tests lower.
maxManifestSize int64
Stdin io.Reader // Standard input stream
Stdout io.Writer // Standard output stream for normal output
Stderr io.Writer // Standard error stream for diagnostics
@@ -147,23 +154,43 @@ func commonFlags() []cli.Flag {
}
}
// stopOnFirstArg returns the StopOnNthArg setting every command uses: flags
// are read only before the command's first argument, and everything after it
// is an argument, as with urfave/cli v2. So `mfer gen d -v` names a path "-v".
func stopOnFirstArg() *int {
n := 1
return &n
}
// requireSignatureFlag returns the --require-signature flag taken by the
// check and fetch subcommands.
func requireSignatureFlag() *cli.StringFlag {
return &cli.StringFlag{
Name: flagRequireSignature,
Aliases: []string{"S"},
Usage: "Require manifest to be signed by the specified GPG key ID",
Usage: "Require manifest to be signed by the OpenPGP key with this fingerprint",
Sources: cli.EnvVars("MFER_REQUIRE_SIGNATURE"),
}
}
// includePermissionsFlag returns the --include-permissions flag taken by the
// generate and freshen subcommands.
func includePermissionsFlag() *cli.BoolFlag {
return &cli.BoolFlag{
Name: flagIncludePermissions,
Usage: "Record each file's permission bits in manifest " +
"(recorded as 0000 by default)",
}
}
func (mfa *CLIApp) generateCommand() *cli.Command {
return &cli.Command{
Name: cmdGenerate,
Aliases: []string{"gen"},
Usage: "Generate manifest file",
ArgsUsage: "[path ...]",
StopOnNthArg: stopOnFirstArg(),
Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner()
@@ -184,9 +211,10 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
},
&cli.StringFlag{
Name: "output",
Value: defaultManifestName,
Aliases: []string{"o"},
Usage: "Specify output filename",
Usage: "File to write the manifest to (default: index.mf in " +
"the directory given, or beside the file given; with no " +
"path or several, index.mf in the current directory)",
},
&cli.BoolFlag{
Name: "force",
@@ -201,7 +229,7 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
&cli.StringFlag{
Name: "sign-key",
Aliases: []string{"s"},
Usage: "GPG key ID to sign the manifest with",
Usage: "OpenPGP secret key file to sign the manifest with",
Sources: cli.EnvVars("MFER_SIGN_KEY"),
},
&cli.StringFlag{
@@ -214,6 +242,7 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
Usage: "Include createdAt timestamp in manifest " +
"(omitted by default for determinism)",
},
includePermissionsFlag(),
),
}
}
@@ -223,6 +252,7 @@ func (mfa *CLIApp) checkCommand() *cli.Command {
Name: cmdCheck,
Usage: "Validate files using manifest file",
ArgsUsage: manifestArgsUsage,
StopOnNthArg: stopOnFirstArg(),
Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner()
@@ -231,10 +261,11 @@ func (mfa *CLIApp) checkCommand() *cli.Command {
},
Flags: append(commonFlags(),
&cli.StringFlag{
Name: "base",
Name: flagBase,
Aliases: []string{"b"},
Value: ".",
Usage: "Base directory for resolving relative paths from manifest",
Usage: "Base directory for resolving relative paths from manifest " +
"(by default the directory holding the manifest, or the " +
"current directory for a manifest URL)",
},
&cli.BoolFlag{
Name: flagProgress,
@@ -255,6 +286,7 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
Name: cmdFreshen,
Usage: "Update manifest with changed, new, and removed files",
ArgsUsage: manifestArgsUsage,
StopOnNthArg: stopOnFirstArg(),
Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner()
@@ -263,10 +295,10 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
},
Flags: append(commonFlags(),
&cli.StringFlag{
Name: "base",
Name: flagBase,
Aliases: []string{"b"},
Value: ".",
Usage: "Base directory for resolving relative paths",
Usage: "Base directory for resolving relative paths " +
"(by default the directory holding the manifest)",
},
&cli.BoolFlag{
Name: "follow-symlinks",
@@ -287,7 +319,7 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
&cli.StringFlag{
Name: "sign-key",
Aliases: []string{"s"},
Usage: "GPG key ID to sign the manifest with",
Usage: "OpenPGP secret key file to sign the manifest with",
Sources: cli.EnvVars("MFER_SIGN_KEY"),
},
&cli.BoolFlag{
@@ -295,6 +327,7 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
Usage: "Include createdAt timestamp in manifest " +
"(omitted by default for determinism)",
},
includePermissionsFlag(),
),
}
}
@@ -304,6 +337,7 @@ func (mfa *CLIApp) exportCommand() *cli.Command {
Name: cmdExport,
Usage: "Export manifest contents as JSON",
ArgsUsage: "[manifest file or URL]",
StopOnNthArg: stopOnFirstArg(),
Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
@@ -316,6 +350,7 @@ func (mfa *CLIApp) versionCommand() *cli.Command {
return &cli.Command{
Name: cmdVersion,
Usage: "Show version",
StopOnNthArg: stopOnFirstArg(),
Action: func(context.Context, *cli.Command) error {
mfa.printVersion()
@@ -326,16 +361,17 @@ func (mfa *CLIApp) versionCommand() *cli.Command {
func (mfa *CLIApp) listCommand() *cli.Command {
return &cli.Command{
Name: "list",
Name: cmdList,
Aliases: []string{"ls"},
Usage: "List files in manifest",
ArgsUsage: manifestArgsUsage,
StopOnNthArg: stopOnFirstArg(),
Action: mfa.listManifestOperation,
Flags: []cli.Flag{
&cli.BoolFlag{
Name: "long",
Aliases: []string{"l"},
Usage: "Show size and mtime",
Usage: "Show mode, size and mtime",
},
&cli.BoolFlag{
Name: "print0",
@@ -350,6 +386,7 @@ func (mfa *CLIApp) fetchCommand() *cli.Command {
Name: cmdFetch,
Usage: "fetch manifest and referenced files",
ArgsUsage: "URL",
StopOnNthArg: stopOnFirstArg(),
Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner()
@@ -406,12 +443,11 @@ func (mfa *CLIApp) run(args []string) {
Version: mfa.VersionString(),
EnableShellCompletion: true,
Writer: mfa.Stdout,
// urfave/cli writes only its "Incorrect Usage" line to ErrWriter. It
// goes to stdout with the help printed after it; run logs the error
// itself to stderr. Tests rely on the logger being the only writer to
// a run's stderr.
// v3 writes its "Incorrect Usage" line to ErrWriter; v2 wrote it to
// stdout, before the help, so it stays on stdout.
ErrWriter: mfa.Stdout,
Flags: commonFlags(),
StopOnNthArg: stopOnFirstArg(),
Action: func(_ context.Context, cmd *cli.Command) error {
if cmd.Args().Len() > 0 {
return fmt.Errorf("%w %q", errUnknownCommand, cmd.Args().First())
internal/cli/signing.go
+86
View File
@@ -0,0 +1,86 @@
package cli
import (
"errors"
"fmt"
"os"
"github.com/spf13/afero"
"golang.org/x/term"
"sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer"
)
// envSignKeyPassphrase names the environment variable holding the
// passphrase of a protected signing key.
//
//nolint:gosec // G101: the name of a variable, not a credential
const envSignKeyPassphrase = "MFER_SIGN_KEY_PASSPHRASE"
// errNoPassphrase indicates a protected signing key whose passphrase is
// neither in the environment nor can be asked for on a terminal.
var errNoPassphrase = errors.New(
"signing key is protected: set " + envSignKeyPassphrase + " to its passphrase")
// signingOptions returns the signing options for the OpenPGP secret key in
// the file path, which must be able to sign. The passphrase of a protected
// key comes from MFER_SIGN_KEY_PASSPHRASE, or else from the terminal on
// stdin, and must unlock the key.
func (mfa *CLIApp) signingOptions(path string) (*mfer.SigningOptions, error) {
secretKey, err := afero.ReadFile(mfa.Fs, path)
if err != nil {
return nil, fmt.Errorf("read signing key: %w", err)
}
protected, err := mfer.SecretKeyIsProtected(secretKey)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
log.Infof("signing manifest with the OpenPGP key in %s", path)
opts := &mfer.SigningOptions{SecretKey: secretKey}
if protected {
opts.Passphrase, err = mfa.readPassphrase(path)
if err != nil {
return nil, err
}
}
// gen and freshen read the signing options before any file, so a key
// that cannot sign, or a wrong passphrase, stops them before they hash
// anything.
err = mfer.CheckSigningKey(opts)
if err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
return opts, nil
}
// readPassphrase returns MFER_SIGN_KEY_PASSPHRASE when it is set, or else
// asks for the passphrase of the key in the file path on the terminal on
// stdin.
func (mfa *CLIApp) readPassphrase(path string) ([]byte, error) {
passphrase := os.Getenv(envSignKeyPassphrase)
if passphrase != "" {
return []byte(passphrase), nil
}
stdin, ok := mfa.Stdin.(*os.File)
if !ok || !term.IsTerminal(int(stdin.Fd())) {
return nil, errNoPassphrase
}
_, _ = fmt.Fprintf(mfa.Stderr, "Passphrase for %s: ", path)
typed, err := term.ReadPassword(int(stdin.Fd()))
_, _ = fmt.Fprintln(mfa.Stderr)
if err != nil {
return nil, fmt.Errorf("read passphrase: %w", err)
}
return typed, nil
}
internal/cli/signing_test.go
+209
View File
@@ -0,0 +1,209 @@
//nolint:testpackage // white-box tests exercise unexported internals
package cli
import (
"bufio"
"io"
"path/filepath"
"strings"
"testing"
"time"
"github.com/ProtonMail/go-crypto/openpgp/packet"
"github.com/creack/pty"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
testFlagSignKey = "--sign-key"
testKeyFile = "/key.asc"
)
// TestGenAndFreshenSignWithKeyFile runs gen, then freshen after a file is
// added, with --sign-key naming a key file: one key with no passphrase and
// one protected by the passphrase in MFER_SIGN_KEY_PASSPHRASE. check
// --require-signature must accept each manifest as signed by that key.
// freshen leaves its manifest out of the listing only on the real
// filesystem, so the test uses that.
func TestGenAndFreshenSignWithKeyFile(t *testing.T) {
for name, passphrase := range map[string][]byte{
"unprotected": nil,
"protected": []byte("passphrase"),
} {
t.Run(name, func(t *testing.T) {
t.Setenv(envSignKeyPassphrase, string(passphrase))
secretKey, fingerprint := testSecretKey(t, passphrase, nil)
fs := afero.NewOsFs()
keyFile := filepath.Join(t.TempDir(), "key.asc")
root := t.TempDir()
manifestPath := filepath.Join(root, defaultManifestName)
require.NoError(t, afero.WriteFile(fs, keyFile, secretKey, 0o600))
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "hello")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", testFlagSignKey, keyFile,
"-o", manifestPath, root,
}, fs)
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
check := []string{
testApp, cmdCheck, "-q",
"--" + flagRequireSignature, fingerprint, manifestPath,
}
opts = testOpts(check, fs)
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
writeTestFile(t, fs, filepath.Join(root, "added.txt"), "added")
opts = testOpts([]string{
testApp, cmdFreshen, "-q", testFlagSignKey, keyFile, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
opts = testOpts(check, fs)
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
assert.Len(t, manifestFiles(t, fs, manifestPath), 2)
})
}
}
// TestSignWithProtectedKeyNeedsPassphrase runs gen with a protected key,
// with MFER_SIGN_KEY_PASSPHRASE empty and no terminal to ask on. gen must
// fail, naming the variable, and write no manifest.
func TestSignWithProtectedKeyNeedsPassphrase(t *testing.T) {
t.Setenv(envSignKeyPassphrase, "")
secretKey, _ := testSecretKey(t, []byte("secret"), nil)
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, testKeyFile, secretKey, 0o600))
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", testFlagSignKey, testKeyFile,
"-o", testMF, testDir,
}, fs)
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
"signing key is protected: set MFER_SIGN_KEY_PASSPHRASE to its passphrase")
exists, err := afero.Exists(fs, testMF)
require.NoError(t, err)
assert.False(t, exists)
}
// TestSignWithKeyThatCannotSignFailsFirst runs gen on a directory and
// freshen on a manifest, neither of which exists, with keys that cannot
// sign: a protected key with a wrong MFER_SIGN_KEY_PASSPHRASE, a key that
// expired in 2020, and a version 6 key. Each run must fail on the key: it
// checks the key before it reads any file, so a missing file goes
// unnoticed.
func TestSignWithKeyThatCannotSignFailsFirst(t *testing.T) {
t.Setenv(envSignKeyPassphrase, "wrong")
wrongPassphrase, _ := testSecretKey(t, []byte("right"), nil)
made := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
expired, _ := testSecretKey(t, nil, &packet.Config{
Algorithm: packet.PubKeyAlgoEdDSA,
Time: func() time.Time { return made },
KeyLifetimeSecs: uint32((24 * time.Hour).Seconds()),
})
version6, _ := testSecretKey(t, nil, &packet.Config{
Algorithm: packet.PubKeyAlgoEd25519,
V6Keys: true,
})
for want, secretKey := range map[string][]byte{
"unlock signing key": wrongPassphrase,
"signing key cannot sign": expired,
"signing key must be an OpenPGP version 4 key": version6,
} {
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, testKeyFile, secretKey, 0o600))
for _, args := range [][]string{
{
testApp, cmdGenerate, "-q", testFlagSignKey, testKeyFile,
"-o", testMF, "/missing",
},
{testApp, cmdFreshen, "-q", testFlagSignKey, testKeyFile, "/missing.mf"},
} {
opts := testOpts(args, fs)
assert.Equal(t, 1, runCLI(opts), args[1], want)
assert.Contains(t, testStderr(t, opts), testKeyFile+": "+want, args[1])
}
}
}
// TestGenAsksForPassphraseOnTerminal runs gen with a protected key, no
// MFER_SIGN_KEY_PASSPHRASE, and a terminal as stdin and stderr. gen must
// ask for the passphrase on stderr, and sign with what is typed after the
// prompt.
func TestGenAsksForPassphraseOnTerminal(t *testing.T) {
t.Setenv(envSignKeyPassphrase, "")
secretKey, fingerprint := testSecretKey(t, []byte("passphrase"), nil)
fs := afero.NewOsFs()
keyFile := filepath.Join(t.TempDir(), "key.asc")
root := t.TempDir()
manifestPath := filepath.Join(root, defaultManifestName)
require.NoError(t, afero.WriteFile(fs, keyFile, secretKey, 0o600))
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "hello")
terminal, tty, err := pty.Open()
require.NoError(t, err)
t.Cleanup(func() { _ = terminal.Close() })
opts := testOpts([]string{
testApp, cmdGenerate, "-q", testFlagSignKey, keyFile,
"-o", manifestPath, root,
}, fs)
opts.Stdin = tty
opts.Stderr = tty
exitCode := make(chan int, 1)
go func() {
exitCode <- runCLI(opts)
// Once gen has ended, reading the terminal fails instead of
// waiting for a prompt that will not come.
_ = tty.Close()
}()
prompt := "Passphrase for " + keyFile + ": "
output := bufio.NewReader(terminal)
written := ""
for !strings.HasSuffix(written, prompt) {
b, err := output.ReadByte()
require.NoError(t, err, "gen wrote %q and no prompt", written)
written += string(b)
}
_, err = terminal.WriteString("passphrase\n")
require.NoError(t, err)
code := <-exitCode
rest, _ := io.ReadAll(output)
require.Equal(t, 0, code, "gen wrote %q", rest)
check := testOpts([]string{
testApp, cmdCheck, "-q",
"--" + flagRequireSignature, fingerprint, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(check), testStderr(t, check))
}
internal/cli/testdata/seeded.mf
Vendored
BIN
View File
Binary file not shown.
mfer/builder.go
+38 -21
View File
@@ -8,6 +8,7 @@ import (
"errors"
"fmt"
"io"
"io/fs"
"sort"
"strings"
"sync"
@@ -37,6 +38,7 @@ var (
errNegativeSize = errors.New("size cannot be negative")
errHashNotMultihash = errors.New("hash is not a valid multihash")
errHashTooShort = errors.New("hash digest is too short")
errDuplicatePath = errors.New("duplicate path")
)
// ValidatePath checks that a file path conforms to manifest path invariants:
@@ -63,7 +65,7 @@ func ValidatePath(p string) error {
return fmt.Errorf("path %q %w", p, errPathAbsolute)
}
for _, seg := range strings.Split(p, "/") {
for seg := range strings.SplitSeq(p, "/") {
if seg == "" {
return fmt.Errorf("path %q %w", p, errPathEmptySegment)
}
@@ -114,6 +116,7 @@ type FileHashProgress struct {
type Builder struct {
mu sync.Mutex
files []*MFFilePath
paths map[string]bool // the path of each entry in files
createdAt time.Time
includeTimestamps bool
signingOptions *SigningOptions
@@ -124,6 +127,7 @@ type Builder struct {
func NewBuilder() *Builder {
return &Builder{
files: make([]*MFFilePath, 0),
paths: make(map[string]bool),
createdAt: time.Now(),
}
}
@@ -137,12 +141,15 @@ func (b *Builder) SetSeed(seed string) {
}
// AddFile reads file content from reader, computes hashes, and adds to manifest.
// A path already added is refused once the file is read.
// Only mode's permission bits (mode.Perm()) are recorded; 0 records none.
// Progress updates are sent to the progress channel (if non-nil) without blocking.
// Returns the number of bytes read.
func (b *Builder) AddFile(
path RelFilePath,
size FileSize,
mtime ModTime,
mode fs.FileMode,
reader io.Reader,
progress chan<- FileHashProgress,
) (FileSize, error) {
@@ -198,13 +205,10 @@ func (b *Builder) AddFile(
{MultiHash: mh},
},
Mtime: mtime.Timestamp(),
Mode: uint32(mode.Perm()),
}
b.mu.Lock()
b.files = append(b.files, entry)
b.mu.Unlock()
return totalRead, nil
return totalRead, b.addEntry(entry)
}
// sendFileHashProgress sends a progress update without blocking.
@@ -229,17 +233,20 @@ func (b *Builder) FileCount() int {
// AddFileWithHash adds a file entry with a pre-computed hash.
// This is useful when the hash is already known (e.g., from an existing manifest).
// Returns an error if path is invalid, size is negative, or hash is not a
// multihash with a digest of at least 32 bytes, as long as SHA-256's.
// Only mode's permission bits (mode.Perm()) are recorded; 0 records none.
// Returns an error if path is invalid or already added, size is negative,
// or hash is not a multihash with a digest of at least 32 bytes, as long
// as SHA-256's.
func (b *Builder) AddFileWithHash(
path RelFilePath,
size FileSize,
mtime ModTime,
mode fs.FileMode,
hash Multihash,
) error {
err := ValidatePath(string(path))
if err != nil {
return fmt.Errorf("add file: %w", err)
return err
}
if size < 0 {
@@ -268,13 +275,10 @@ func (b *Builder) AddFileWithHash(
{MultiHash: hash},
},
Mtime: mtime.Timestamp(),
Mode: uint32(mode.Perm()),
}
b.mu.Lock()
b.files = append(b.files, entry)
b.mu.Unlock()
return nil
return b.addEntry(entry)
}
// SetIncludeTimestamps controls whether the manifest includes a createdAt timestamp.
@@ -286,7 +290,7 @@ func (b *Builder) SetIncludeTimestamps(include bool) {
b.includeTimestamps = include
}
// SetSigningOptions sets the GPG signing options for the manifest.
// SetSigningOptions sets the key the manifest is signed with.
// If opts is non-nil, the manifest will be signed when Build() is called.
func (b *Builder) SetSigningOptions(opts *SigningOptions) {
b.mu.Lock()
@@ -295,8 +299,8 @@ func (b *Builder) SetSigningOptions(opts *SigningOptions) {
b.signingOptions = opts
}
// Build finalizes the manifest and writes it to the writer. ctx bounds the
// gpg runs that sign the manifest when signing options are set.
// Build finalizes the manifest and writes it to the writer. When signing
// options are set, it does not sign once ctx has ended.
func (b *Builder) Build(ctx context.Context, w io.Writer) error {
b.mu.Lock()
defer b.mu.Unlock()
@@ -325,20 +329,33 @@ func (b *Builder) Build(ctx context.Context, w io.Writer) error {
// Generate outer wrapper
err := m.generateOuter(ctx)
if err != nil {
return fmt.Errorf("build: generate outer: %w", err)
return err
}
// Generate final output
err = m.generate(ctx)
if err != nil {
return fmt.Errorf("build: generate: %w", err)
return err
}
// Write to output
_, err = w.Write(m.output.Bytes())
if err != nil {
return fmt.Errorf("build: write output: %w", err)
return err
}
// addEntry adds entry to the manifest unless an entry with its path is
// already there.
func (b *Builder) addEntry(entry *MFFilePath) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.paths[entry.GetPath()] {
return fmt.Errorf("%w %q", errDuplicatePath, entry.GetPath())
}
b.paths[entry.GetPath()] = true
b.files = append(b.files, entry)
return nil
}
mfer/builder_test.go
+44 -18
View File
@@ -35,7 +35,7 @@ func TestBuilderAddFile(t *testing.T) {
reader := bytes.NewReader(content)
bytesRead, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil,
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
)
require.NoError(t, err)
assert.Equal(t, FileSize(len(content)), bytesRead)
@@ -49,7 +49,7 @@ func TestBuilderAddFileWithHash(t *testing.T) {
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
err = b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), hash)
err = b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, hash)
require.NoError(t, err)
assert.Equal(t, 1, b.FileCount())
}
@@ -64,7 +64,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("", 100, ModTime(time.Now()), sha256Hash)
err := b.AddFileWithHash("", 100, ModTime(time.Now()), 0, sha256Hash)
require.Error(t, err)
assert.Contains(t, err.Error(), "path")
})
@@ -73,7 +73,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), sha256Hash)
err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), 0, sha256Hash)
require.Error(t, err)
assert.Contains(t, err.Error(), "size")
})
@@ -82,7 +82,7 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), sha256Hash)
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, sha256Hash)
require.NoError(t, err)
assert.Equal(t, 1, b.FileCount())
})
@@ -118,13 +118,39 @@ func TestBuilderAddFileWithHashRejectsBadHashes(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), tt.hash)
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), 0, tt.hash)
require.ErrorIs(t, err, tt.want)
assert.Equal(t, 0, b.FileCount())
})
}
}
// TestBuilderRefusesPathAlreadyAdded adds a path, then adds it again with
// AddFile and with AddFileWithHash. Each must refuse it, naming it, and
// keep the one entry already added.
func TestBuilderRefusesPathAlreadyAdded(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
b := NewBuilder()
require.NoError(t, b.AddFileWithHash("dir/a.txt", 4, ModTime{}, 0, hash))
content := []byte("data")
_, err = b.AddFile(
"dir/a.txt", FileSize(len(content)), ModTime{}, 0, bytes.NewReader(content), nil,
)
require.ErrorIs(t, err, errDuplicatePath)
require.EqualError(t, err, `duplicate path "dir/a.txt"`)
err = b.AddFileWithHash("dir/a.txt", 4, ModTime{}, 0, hash)
require.ErrorIs(t, err, errDuplicatePath)
require.EqualError(t, err, `duplicate path "dir/a.txt"`)
assert.Equal(t, 1, b.FileCount())
}
func TestBuilderBuild(t *testing.T) {
t.Parallel()
@@ -133,7 +159,7 @@ func TestBuilderBuild(t *testing.T) {
reader := bytes.NewReader(content)
_, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil,
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
)
require.NoError(t, err)
@@ -196,7 +222,7 @@ func TestBuilderDeterministicOutput(t *testing.T) {
for _, f := range files {
r := bytes.NewReader([]byte(f.content))
_, err := b.AddFile(
RelFilePath(f.path), FileSize(len(f.content)), mtime, r, nil,
RelFilePath(f.path), FileSize(len(f.content)), mtime, 0, r, nil,
)
require.NoError(t, err)
}
@@ -279,7 +305,7 @@ func TestBuilderAddFileSizeMismatch(t *testing.T) {
reader := bytes.NewReader(content)
// Declare wrong size
_, err := b.AddFile("test.txt", FileSize(100), ModTime(time.Now()), reader, nil)
_, err := b.AddFile("test.txt", FileSize(100), ModTime(time.Now()), 0, reader, nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "size mismatch")
}
@@ -291,12 +317,12 @@ func TestBuilderAddFileInvalidPath(t *testing.T) {
content := []byte("data")
reader := bytes.NewReader(content)
_, err := b.AddFile("", FileSize(len(content)), ModTime(time.Now()), reader, nil)
_, err := b.AddFile("", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil)
require.Error(t, err)
reader.Reset(content)
_, err = b.AddFile(
"/absolute", FileSize(len(content)), ModTime(time.Now()), reader, nil,
"/absolute", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
)
assert.Error(t, err)
}
@@ -310,7 +336,7 @@ func TestBuilderAddFileWithProgress(t *testing.T) {
progress := make(chan FileHashProgress, 100)
bytesRead, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, progress,
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, progress,
)
close(progress)
require.NoError(t, err)
@@ -345,7 +371,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
for _, f := range files {
reader := bytes.NewReader(f.content)
_, err := b.AddFile(
RelFilePath(f.path), FileSize(len(f.content)), ModTime(now), reader, nil,
RelFilePath(f.path), FileSize(len(f.content)), ModTime(now), 0, reader, nil,
)
require.NoError(t, err)
}
@@ -387,7 +413,7 @@ func TestBuilderBuildRoundTripLargeManifest(t *testing.T) {
for i := range 4000 {
path := RelFilePath(fmt.Sprintf("dir/file-%05d.txt", i))
require.NoError(t, b.AddFileWithHash(path, FileSize(i), ModTime{}, hash))
require.NoError(t, b.AddFileWithHash(path, FileSize(i), ModTime{}, 0, hash))
}
var buf bytes.Buffer
@@ -452,7 +478,7 @@ func TestManifestString(t *testing.T) {
content := []byte("test")
reader := bytes.NewReader(content)
_, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()), reader, nil,
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
)
require.NoError(t, err)
@@ -484,7 +510,7 @@ func TestBuilderOmitsCreatedAtByDefault(t *testing.T) {
b := NewBuilder()
content := []byte("hello")
_, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()),
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0,
bytes.NewReader(content), nil,
)
require.NoError(t, err)
@@ -506,7 +532,7 @@ func TestBuilderIncludesCreatedAtWhenRequested(t *testing.T) {
content := []byte("hello")
_, err := b.AddFile(
"test.txt", FileSize(len(content)), ModTime(time.Now()),
"test.txt", FileSize(len(content)), ModTime(time.Now()), 0,
bytes.NewReader(content), nil,
)
require.NoError(t, err)
@@ -532,7 +558,7 @@ func TestBuilderDeterministicFileOrder(t *testing.T) {
content := []byte("content of " + name)
_, err := b.AddFile(
RelFilePath(name), FileSize(len(content)),
ModTime(time.Unix(1000, 0)), bytes.NewReader(content), nil,
ModTime(time.Unix(1000, 0)), 0, bytes.NewReader(content), nil,
)
require.NoError(t, err)
}
mfer/checker.go
+24 -16
View File
@@ -15,7 +15,6 @@ import (
)
var (
errNoSigningPubKey = errors.New("manifest has no signing public key")
errManifestPathEmpty = errors.New("manifest path cannot be empty")
errBasePathEmpty = errors.New("base path cannot be empty")
)
@@ -36,6 +35,7 @@ const (
StatusMissing // File not found on disk
StatusSizeMismatch // File size differs from manifest
StatusHashMismatch // File hash differs from manifest
StatusModeMismatch // File permission bits differ from a recorded mode
StatusExtra // File exists on disk but not in manifest
StatusError // Error occurred during verification
)
@@ -50,6 +50,8 @@ func (s Status) String() string {
return "SIZE_MISMATCH"
case StatusHashMismatch:
return "HASH_MISMATCH"
case StatusModeMismatch:
return "MODE_MISMATCH"
case StatusExtra:
return "EXTRA"
case StatusError:
@@ -170,8 +172,14 @@ func (c *Checker) IsSigned() bool {
return len(c.signature) > 0
}
// Signer returns the signer fingerprint if the manifest is signed, nil otherwise.
// Signer returns the fingerprint of the key that made the manifest's
// signature, which loading the manifest checked, or nil if the manifest is
// not signed.
func (c *Checker) Signer() []byte {
if !c.IsSigned() {
return nil
}
return c.signer
}
@@ -181,17 +189,6 @@ func (c *Checker) SigningPubKey() []byte {
return c.signingPubKey
}
// ExtractEmbeddedSigningKeyFP imports the manifest's embedded public key into a
// temporary keyring and extracts its fingerprint. This validates the key and
// returns its actual fingerprint from the key material itself.
func (c *Checker) ExtractEmbeddedSigningKeyFP(ctx context.Context) (string, error) {
if len(c.signingPubKey) == 0 {
return "", errNoSigningPubKey
}
return gpgExtractPubKeyFingerprint(ctx, c.signingPubKey)
}
// Check verifies all files against the manifest.
// Results are sent to the results channel as files are checked.
// Progress updates are sent to the progress channel approximately once per second.
@@ -408,11 +405,22 @@ func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
return Result{Path: relPath, Status: StatusError, Message: err.Error()}
}
// Check against all hashes in manifest (at least one must match)
// Check against all hashes in manifest (at least one must match),
// then against the recorded mode, where one is: 0 means none was.
for _, hash := range entry.GetHashes() {
if bytes.Equal(computed, hash.GetMultiHash()) {
return Result{Path: relPath, Status: StatusOK}
if !bytes.Equal(computed, hash.GetMultiHash()) {
continue
}
if entry.GetMode() != 0 && info.Mode().Perm() != os.FileMode(entry.GetMode()) {
return Result{
Path: relPath,
Status: StatusModeMismatch,
Message: "mode mismatch",
}
}
return Result{Path: relPath, Status: StatusOK}
}
return Result{
mfer/checker_test.go
+52 -2
View File
@@ -34,6 +34,7 @@ func TestStatusString(t *testing.T) {
{StatusMissing, "MISSING"},
{StatusSizeMismatch, "SIZE_MISMATCH"},
{StatusHashMismatch, "HASH_MISMATCH"},
{StatusModeMismatch, "MODE_MISMATCH"},
{StatusExtra, "EXTRA"},
{StatusError, "ERROR"},
{Status(99), "UNKNOWN"},
@@ -59,7 +60,7 @@ func createTestManifest(
for path, content := range files {
reader := bytes.NewReader(content)
_, err := builder.AddFile(
RelFilePath(path), FileSize(len(content)), ModTime(time.Now()), reader, nil,
RelFilePath(path), FileSize(len(content)), ModTime(time.Now()), 0, reader, nil,
)
require.NoError(t, err)
}
@@ -279,7 +280,8 @@ func TestCheckMissingFile(t *testing.T) {
missingCount++
assert.Equal(t, RelFilePath("missing.txt"), r.Path)
case StatusSizeMismatch, StatusHashMismatch, StatusExtra, StatusError:
case StatusSizeMismatch, StatusHashMismatch, StatusModeMismatch,
StatusExtra, StatusError:
// Not expected in this test; counted assertions below will fail.
}
}
@@ -349,6 +351,54 @@ func TestCheckHashMismatch(t *testing.T) {
assert.Equal(t, RelFilePath(testFileName), r.Path)
}
// A recorded mode other than 0000 that differs from the file's permission
// bits fails the check; a recorded 0000 is never checked.
func TestCheckMode(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
recorded os.FileMode
onDisk os.FileMode
want Status
}{
{"recorded mode matches", 0o640, 0o640, StatusOK},
{"recorded mode differs", 0o640, 0o600, StatusModeMismatch},
{"0000 is not checked", 0, 0o600, StatusOK},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
content := []byte("content")
b := NewBuilder()
_, err := b.AddFile(testFileName, FileSize(len(content)), ModTime{},
tc.recorded, bytes.NewReader(content), nil)
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, afero.WriteFile(fs, testManifestPath, buf.Bytes(), 0o644))
require.NoError(t, fs.MkdirAll(testDataDir, 0o755))
require.NoError(t, afero.WriteFile(fs,
testDataDir+"/"+testFileName, content, tc.onDisk))
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
require.NoError(t, err)
results := make(chan Result, 1)
require.NoError(t, chk.Check(context.Background(), results, nil))
assert.Equal(t, tc.want, (<-results).Status)
})
}
}
func TestCheckWithProgress(t *testing.T) {
t.Parallel()
mfer/constants.go
+12 -5
View File
@@ -8,10 +8,17 @@ const (
ReleaseDate = "2025-12-17"
// MaxDecompressedSize is the maximum allowed size of decompressed manifest
// data (256 MB). This prevents decompression bombs from consuming excessive
// data (256 MiB). This prevents decompression bombs from consuming excessive
// memory.
MaxDecompressedSize int64 = 256 * 1024 * 1024
// MaxManifestSize is the largest manifest file mfer reads (258 MiB).
// zstd's worst case grows data it cannot compress by 1/256, so an inner
// message of MaxDecompressedSize compresses to at most 257 MiB; the
// last MiB is room for the signature, the signing key and the other
// outer fields.
MaxManifestSize = MaxDecompressedSize + MaxDecompressedSize/256 + 1<<20
// zstdWindowSize is the zstd window zstd.SpeedBestCompression gives mfer's writer.
zstdWindowSize = 8 << 20
@@ -29,8 +36,8 @@ const (
// Bytes decoding sets aside for each file entry, hash, timestamp and
// MIME type, however short its encoding. checkDecodedSize refuses an
// inner message for which these add up to more than maxDecodedGrowth
// times its size.
decodedFileEntrySize = 160
// times its size. The mode is held in the file entry itself.
decodedFileEntrySize = 176
decodedHashSize = 112
decodedTimestampSize = 64
decodedMIMETypeSize = 16
@@ -38,7 +45,7 @@ const (
// Each file entry mfer writes holds a path of at least one byte, a
// multihash at least as long as SHA-256's 34 bytes (AddFileWithHash
// refuses shorter ones) and a modification time: at least 47 bytes,
// counted at 336. So its manifests add up to at most about 7.15 times
// their size, and this limit is about 12% above that.
// counted at 352. So its manifests add up to at most about 7.49 times
// their size, and this limit is about 7% above that.
maxDecodedGrowth = 8
)
mfer/deserialize.go
+59 -31
View File
@@ -2,13 +2,12 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
"io"
"strings"
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/spf13/afero"
"google.golang.org/protobuf/encoding/protowire"
@@ -19,29 +18,26 @@ import (
var (
errInvalidUUIDLength = errors.New("invalid UUID length")
errInvalidUUIDFormat = errors.New("invalid UUID format")
errUnknownVersion = errors.New("unknown version")
errUnknownCompression = errors.New("unknown compression type")
errCompressedHashWrong = errors.New("compressed data hash mismatch")
errSignatureNoPubKey = errors.New("signature present but no public key")
errDecompressedTooLarge = errors.New("decompressed data exceeds maximum allowed size")
errManifestTooLarge = errors.New("file exceeds maximum allowed size")
errUUIDMismatch = errors.New("outer and inner UUID mismatch")
errInvalidFileFormat = errors.New("invalid file format")
errInvalidManifestPath = errors.New("manifest contains invalid path")
errDecodedTooLarge = errors.New(
"manifest would take too much memory to decode")
errInvalidManifestPath = errors.New("invalid file entry")
errDecodedTooLarge = errors.New("too much memory needed")
errSignerNotSigningKey = errors.New(
"signer is not the fingerprint of the key that made the signature")
)
// validateUUID checks that the byte slice is a valid UUID (16 bytes, parseable).
// validateUUID checks that the byte slice is the 16 bytes of a binary UUID.
// Any 16 bytes are one, so the length is all there is to check.
func validateUUID(data []byte) error {
if len(data) != uuidLength {
return errInvalidUUIDLength
}
// Try to parse as UUID to validate format
_, err := uuid.FromBytes(data)
if err != nil {
return errInvalidUUIDFormat
}
return nil
}
@@ -60,20 +56,22 @@ func (m *manifest) validateOuterHeader() error {
// Validate outer UUID before any decompression
err := validateUUID(m.pbOuter.GetUuid())
if err != nil {
return fmt.Errorf("outer UUID invalid: %w", err)
return fmt.Errorf("outer message: %w", err)
}
return nil
}
// verifyOuterIntegrity checks the hash of the compressed payload and,
// if a signature is present, verifies it against the embedded public key.
// verifyOuterIntegrity checks the hash of the compressed payload and, if a
// signature is present, verifies it against the embedded public key, which
// must be one key, and checks that the signer field is that key's
// fingerprint.
func (m *manifest) verifyOuterIntegrity() error {
h := sha256.New()
_, err := h.Write(m.pbOuter.GetInnerMessage())
if err != nil {
return fmt.Errorf("deserialize: hash write: %w", err)
return fmt.Errorf("hash inner message: %w", err)
}
sha256Hash := h.Sum(nil)
@@ -91,20 +89,21 @@ func (m *manifest) verifyOuterIntegrity() error {
sigString, err := m.signatureString()
if err != nil {
return fmt.Errorf(
"failed to generate signature string for verification: %w", err,
)
return fmt.Errorf("build signature string: %w", err)
}
// Loading a manifest takes no context; gpgTimeout still bounds gpg.
err = gpgVerify(
context.Background(),
signingKey, err := verifySignature(
[]byte(sigString),
m.pbOuter.GetSignature(),
m.pbOuter.GetSigningPubKey(),
)
if err != nil {
return fmt.Errorf("signature verification failed: %w", err)
return err
}
if !strings.EqualFold(string(m.pbOuter.GetSigner()), signingKey) {
return fmt.Errorf("%w: signer %q, signing key %s",
errSignerNotSigningKey, m.pbOuter.GetSigner(), signingKey)
}
log.Infof("signature verified successfully")
@@ -130,7 +129,7 @@ func (m *manifest) decompressInner() ([]byte, error) {
zstd.WithDecodeBuffersBelow(0),
zstd.WithDecoderMaxWindow(zstdWindowSize))
if err != nil {
return nil, fmt.Errorf("deserialize: zstd reader: %w", err)
return nil, fmt.Errorf("create decompressor: %w", err)
}
defer zr.Close()
@@ -145,7 +144,7 @@ func (m *manifest) decompressInner() ([]byte, error) {
dat, err := io.ReadAll(limitedReader)
if err != nil {
return nil, fmt.Errorf("deserialize: decompress: %w", err)
return nil, fmt.Errorf("decompress inner message: %w", err)
}
if int64(len(dat)) >= MaxDecompressedSize {
@@ -261,7 +260,7 @@ func (m *manifest) deserializeInner() error {
err = checkDecodedSize(dat)
if err != nil {
return fmt.Errorf("deserialize: unmarshal inner: %w", err)
return fmt.Errorf("unmarshal inner message: %w", err)
}
// Deserialize inner message
@@ -270,13 +269,17 @@ func (m *manifest) deserializeInner() error {
// Unknown fields would cost memory; mfer never writes a loaded manifest out.
err = proto.UnmarshalOptions{DiscardUnknown: true}.Unmarshal(dat, m.pbInner)
if err != nil {
return fmt.Errorf("deserialize: unmarshal inner: %w", err)
return fmt.Errorf("unmarshal inner message: %w", err)
}
if m.pbInner.GetVersion() != MFFile_VERSION_ONE {
return errUnknownVersion
}
// Validate inner UUID
err = validateUUID(m.pbInner.GetUuid())
if err != nil {
return fmt.Errorf("inner UUID invalid: %w", err)
return fmt.Errorf("inner message: %w", err)
}
// Verify UUIDs match
@@ -289,12 +292,21 @@ func (m *manifest) deserializeInner() error {
// extract path tomorrow — acts on a traversal or absolute path from an
// untrusted .mf. Reject loudly on the first offender rather than
// dropping entries, which would let a hostile manifest hide files from a
// check.
// check. A path listed twice is refused too: check would check the one
// file against both entries.
seen := make(map[string]bool, len(m.pbInner.GetFiles()))
for _, f := range m.pbInner.GetFiles() {
err = ValidatePath(f.GetPath())
if err != nil {
return fmt.Errorf("%w: %w", errInvalidManifestPath, err)
}
if seen[f.GetPath()] {
return fmt.Errorf("%w %q", errDuplicatePath, f.GetPath())
}
seen[f.GetPath()] = true
}
log.Infof("loaded manifest with %d files", len(m.pbInner.GetFiles()))
@@ -314,13 +326,14 @@ func validateMagic(dat []byte) bool {
return bytes.Equal(got, expected)
}
// NewManifestFromReader reads a manifest from an io.Reader.
// NewManifestFromReader reads a manifest from an io.Reader. It refuses a
// manifest larger than MaxManifestSize, reading at most one byte past it.
//
//nolint:revive // unexported-return: exporting manifest is owner question 13
func NewManifestFromReader(input io.Reader) (*manifest, error) {
m := &manifest{}
dat, err := io.ReadAll(input)
dat, err := readAtMost(input, MaxManifestSize)
if err != nil {
return nil, err
}
@@ -352,6 +365,21 @@ func NewManifestFromReader(input io.Reader) (*manifest, error) {
return m, nil
}
// readAtMost reads all of input, or refuses it with errManifestTooLarge
// once it passes maxSize bytes, after reading one byte past maxSize.
func readAtMost(input io.Reader, maxSize int64) ([]byte, error) {
dat, err := io.ReadAll(io.LimitReader(input, maxSize+1))
if err != nil {
return nil, err
}
if int64(len(dat)) > maxSize {
return nil, fmt.Errorf("%w of %d bytes", errManifestTooLarge, maxSize)
}
return dat, nil
}
// ManifestFromFileOptions configures NewManifestFromFile.
type ManifestFromFileOptions struct {
// Path is the manifest file to read (required).
mfer/deserialize_fuzz_test.go
-8
View File
@@ -19,14 +19,6 @@ import (
// input and of the decompressed data it may read, plus room for the
// decoder's window buffers. A panic or a hang fails the test on its own.
func FuzzNewManifestFromReader(f *testing.F) {
// A signed manifest makes the parser write the key and signature to a
// temporary directory and run gpg on them. With gpg off the PATH and
// temporary files kept in the test's own directory, no process is
// started and nothing is written elsewhere; such input ends in an
// error instead.
f.Setenv("PATH", "")
f.Setenv("TMPDIR", f.TempDir())
f.Fuzz(func(t *testing.T, data []byte) {
var before, after runtime.MemStats
mfer/deserialize_path_test.go
+72 -20
View File
@@ -7,11 +7,10 @@ import (
"crypto/sha256"
"fmt"
"strconv"
"strings"
"testing"
"time"
"uuid"
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/multiformats/go-multihash"
"github.com/stretchr/testify/assert"
@@ -92,7 +91,7 @@ func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
id := uuid.New()
id := uuid.NewV4()
data := wrapInner(t, id, craftInnerBytes(id, tt.path))
_, err := NewManifestFromReader(bytes.NewReader(data))
@@ -118,12 +117,61 @@ func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
}
}
// A manifest that lists a path twice is refused as it is loaded, naming the
// path. Paths are compared byte for byte: two that differ only in letter case
// load, and fetch refuses those itself. Each entry has a hash, as entries mfer
// writes do; entries of a path alone would take too much memory to decode.
func TestDeserializeRefusesPathListedTwice(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
tests := []struct {
name string
paths []string
refused bool
}{
{"same path twice", []string{"dir/a.txt", "other.txt", "dir/a.txt"}, true},
{"paths differing in letter case", []string{"dir/b.txt", "dir/B.txt"}, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
id := uuid.NewV4()
inner := &MFFile{Version: MFFile_VERSION_ONE, Uuid: id[:]}
for _, p := range tt.paths {
inner.Files = append(inner.Files, &MFFilePath{
Path: p,
Hashes: []*MFFileChecksum{{MultiHash: hash}},
})
}
innerData, err := proto.Marshal(inner)
require.NoError(t, err)
m, err := NewManifestFromReader(bytes.NewReader(wrapInner(t, id, innerData)))
if tt.refused {
require.ErrorIs(t, err, errDuplicatePath)
require.EqualError(t, err, `duplicate path "dir/a.txt"`)
} else {
require.NoError(t, err)
assert.Len(t, m.Files(), len(tt.paths))
}
})
}
}
// Entries of a path, an empty hash, an empty MIME type and empty modification
// and change times are counted at 416 bytes each (160 + 112 + 16 + 64 + 64)
// and take 16 bytes plus the path to encode. A 35-character path makes that
// 51 bytes, about 8.2 times: refused, and leaving any one of the five
// uncounted, even the MIME type, brings it under 8. A 37-character path makes
// it 53 bytes, about 7.8 times: loaded.
// and change times are counted at 432 bytes each (176 + 112 + 16 + 64 + 64)
// and take 16 bytes plus the path to encode. A 37-character path makes that
// 53 bytes, about 8.2 times: refused, and leaving any one of the five
// uncounted, even the MIME type, brings it under 8. A 39-character path makes
// it 55 bytes, about 7.9 times: loaded. Each entry's path is its number,
// padded with zeros to that length, since a manifest lists a path only once.
func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
t.Parallel()
@@ -131,16 +179,23 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
pathLen int
refused bool
}{
{35, true},
{37, false},
{37, true},
{39, false},
}
for _, tt := range tests {
t.Run(strconv.Itoa(tt.pathLen), func(t *testing.T) {
t.Parallel()
id := uuid.NewV4()
inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
for i := range 1000 {
entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
entry = protowire.AppendString(entry, strings.Repeat("a", tt.pathLen))
entry = protowire.AppendString(entry, fmt.Sprintf("%0*d", tt.pathLen, i))
entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
entry = protowire.AppendBytes(entry, nil)
entry = protowire.AppendTag(entry, 301, protowire.BytesType) // MFFilePath.mimeType
@@ -150,11 +205,6 @@ func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
entry = protowire.AppendTag(entry, 303, protowire.BytesType) // MFFilePath.ctime
entry = protowire.AppendBytes(entry, nil)
id := uuid.New()
inner := protowire.AppendTag(nil, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
for range 1000 {
inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
inner = protowire.AppendBytes(inner, entry)
}
@@ -181,8 +231,10 @@ func TestDeserializeDropsUnknownFields(t *testing.T) {
entry = protowire.AppendString(entry, "a")
entry = append(entry, unknown...)
id := uuid.New()
inner := protowire.AppendTag(nil, 101, protowire.BytesType) // MFFile.files
id := uuid.NewV4()
inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
inner = protowire.AppendBytes(inner, entry)
inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
@@ -215,7 +267,7 @@ func TestDeserializeLoadsDensestManifest(t *testing.T) {
const files = 10000
for i := range files {
name := RelFilePath(strconv.FormatInt(int64(i), 36))
require.NoError(t, b.AddFileWithHash(name, 0, ModTime(time.Unix(0, 0)), hash))
require.NoError(t, b.AddFileWithHash(name, 0, ModTime(time.Unix(0, 0)), 0, hash))
}
var buf bytes.Buffer
@@ -233,7 +285,7 @@ func TestDeserializeValidManifestRoundTrips(t *testing.T) {
require.NoError(t, err)
b := NewBuilder()
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, 0, hash))
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
mfer/deserialize_test.go
+54
View File
@@ -0,0 +1,54 @@
//nolint:testpackage // white-box tests exercise unexported internals
package mfer
import (
"bytes"
"testing"
"uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
)
// An inner message whose version is not VERSION_ONE, whether version 0 or a
// later one, is refused with the same error as an outer message's.
func TestDeserializeRefusesUnknownInnerVersion(t *testing.T) {
t.Parallel()
for _, version := range []MFFile_Version{MFFile_VERSION_NONE, MFFile_VERSION_ONE + 1} {
t.Run(version.String(), func(t *testing.T) {
t.Parallel()
id := uuid.NewV4()
inner, err := proto.Marshal(&MFFile{Version: version, Uuid: id[:]})
require.NoError(t, err)
_, err = NewManifestFromReader(bytes.NewReader(wrapInner(t, id, inner)))
require.ErrorIs(t, err, errUnknownVersion)
})
}
}
// TestReadAtMost gives readAtMost exactly its maximum, which it must
// return whole, and twice its maximum, which it must refuse after reading
// one byte past the maximum, and no more. NewManifestFromReader reads
// through it with MaxManifestSize; the test uses 64 KiB, since reading
// MaxManifestSize under the race detector takes gigabytes of memory.
func TestReadAtMost(t *testing.T) {
t.Parallel()
const maxSize = 64 << 10
dat, err := readAtMost(bytes.NewReader(make([]byte, maxSize)), maxSize)
require.NoError(t, err)
assert.Len(t, dat, maxSize)
input := bytes.NewReader(make([]byte, 2*maxSize))
_, err = readAtMost(input, maxSize)
require.ErrorIs(t, err, errManifestTooLarge)
require.EqualError(t, err,
"file exceeds maximum allowed size of 65536 bytes")
assert.Equal(t, maxSize-1, input.Len(), "bytes left unread")
}
mfer/errmsg_test.go
+11 -7
View File
@@ -74,14 +74,18 @@ func TestValidatePathMessagesVerbatim(t *testing.T) {
}
}
// TestSerializeInternalErrorMessagesVerbatim pins the two distinct
// "internal error" messages, which differ between generate and
// generateOuter and have always done so.
func TestSerializeInternalErrorMessagesVerbatim(t *testing.T) {
// TestSerializeInnerNotSetMessagesVerbatim pins the messages generate and
// generateOuter return when the inner message is missing.
func TestSerializeInnerNotSetMessagesVerbatim(t *testing.T) {
t.Parallel()
m := &manifest{}
require.EqualError(t, m.generate(context.Background()),
"internal error: pbInner not set")
require.EqualError(t, m.generateOuter(context.Background()), "internal error")
err := m.generate(context.Background())
require.ErrorIs(t, err, errInnerNotSet)
require.EqualError(t, err, "inner message not set")
err = m.generateOuter(context.Background())
require.ErrorIs(t, err, errInternal)
require.EqualError(t, err, "inner message not set")
}
mfer/gpg.go
-309
View File
@@ -1,309 +0,0 @@
package mfer
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
)
const (
// gpgTimeout bounds every gpg run, which can otherwise wait forever on
// a passphrase prompt or a stalled gpg-agent. A minute leaves a person
// time to type a passphrase or touch a smartcard.
gpgTimeout = time.Minute
// gpgWaitDelay is how long a gpg run keeps waiting for gpg's stdout
// and stderr to close once gpg has been killed or has exited. Reading
// what gpg itself wrote takes far less; only a process gpg left behind
// holds them open longer.
gpgWaitDelay = time.Second
// privateDirPerms is the permission mode for temporary GPG home
// directories.
privateDirPerms os.FileMode = 0o700
// privateFilePerms is the permission mode for temporary key,
// signature, and data files.
privateFilePerms os.FileMode = 0o600
// gpgFingerprintField is the record type tag for fingerprint lines
// in gpg --with-colons output.
gpgFingerprintField = "fpr"
// gpgFingerprintMinFields is the minimum number of colon-separated
// fields in a gpg fingerprint record (the fingerprint is field 10).
gpgFingerprintMinFields = 10
// gpg option names used from more than one call site.
gpgOptArmor = "--armor"
gpgOptHomedir = "--homedir"
gpgOptVerify = "--verify"
)
var (
errGPGKeyNotFound = errors.New("gpg key not found")
errFingerprintNotFound = errors.New("fingerprint not found for key")
errImportedFPRNotFound = errors.New("fingerprint not found in imported key")
)
// GPGKeyID represents a GPG key identifier (fingerprint or key ID).
type GPGKeyID string
// SigningOptions contains options for GPG signing.
type SigningOptions struct {
KeyID GPGKeyID
}
// gpgArgs builds a gpg argument list from opts followed by positional
// arguments, separated by an explicit "--" end-of-options marker.
//
// This matters because key IDs reach gpg as bare positional arguments
// (from --sign-key / MFER_SIGN_KEY) and gpg would otherwise parse a value
// beginning with "-" as one of its own options. Callers must route every
// non-option argument through here.
func gpgArgs(opts []string, positional ...string) []string {
args := make([]string, 0, len(opts)+1+len(positional))
args = append(args, opts...)
args = append(args, "--")
args = append(args, positional...)
return args
}
// runGPG runs the gpg binary in batch mode with the given arguments and
// optional stdin, returning captured stdout and stderr. gpg is killed when
// ctx ends or gpgTimeout passes, whichever comes first.
func runGPG(
ctx context.Context, stdin io.Reader, args ...string,
) (*bytes.Buffer, *bytes.Buffer, error) {
// exec.CommandContext kills only gpg itself. A gpg-agent that gpg
// starts runs detached and holds none of gpg's output, but another
// process gpg leaves behind (a wrapper script that runs the real gpg
// without exec, for example) can keep gpg's stdout or stderr open, and
// Run would wait for it to exit. WaitDelay stops that wait
// gpgWaitDelay after the kill; that process is left running.
ctx, cancel := context.WithTimeout(ctx, gpgTimeout)
defer cancel()
fullArgs := append([]string{"--batch", "--no-tty"}, args...)
// G204: the executable name is a compile-time constant. The arguments
// are not, so the guarantee that matters is placement: every
// caller-supplied value is passed either as the value of a named
// option or after the "--" end-of-options marker inserted by gpgArgs,
// and therefore cannot be reinterpreted by gpg as an option.
cmd := exec.CommandContext( //nolint:gosec // G204: see comment above
ctx, "gpg", fullArgs...)
cmd.WaitDelay = gpgWaitDelay
cmd.Stdin = stdin
var stdout, stderr bytes.Buffer
cmd.Stdout = &stdout
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil && ctx.Err() != nil {
// gpg was killed because ctx ended, which Run reports only as
// "signal: killed"; return the reason instead.
err = ctx.Err()
if errors.Is(err, context.DeadlineExceeded) {
err = fmt.Errorf("gpg timed out: %w", err)
}
}
return &stdout, &stderr, err
}
// parseFingerprint extracts the first fingerprint from gpg --with-colons
// output, or returns ok=false if none is present.
func parseFingerprint(colonOutput string) (string, bool) {
for _, line := range strings.Split(colonOutput, "\n") {
fields := strings.Split(line, ":")
if len(fields) >= gpgFingerprintMinFields &&
fields[0] == gpgFingerprintField {
return fields[9], true
}
}
return "", false
}
// gpgSign creates a detached signature of the data using the specified key.
// Returns the armored detached signature.
func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, bytes.NewReader(data),
"--detach-sign",
gpgOptArmor,
"--local-user", string(keyID),
)
if err != nil {
return nil, fmt.Errorf("gpg sign failed: %w: %s", err, stderr.String())
}
return stdout.Bytes(), nil
}
// gpgExportPublicKey exports the public key for the specified key ID.
// Returns the armored public key.
func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
gpgArgs([]string{"--export", gpgOptArmor}, string(keyID))...,
)
if err != nil {
return nil, fmt.Errorf("gpg export failed: %w: %s", err, stderr.String())
}
if stdout.Len() == 0 {
return nil, fmt.Errorf("%w: %s", errGPGKeyNotFound, keyID)
}
return stdout.Bytes(), nil
}
// gpgGetKeyFingerprint gets the full fingerprint for a key ID.
func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
gpgArgs([]string{"--with-colons", "--fingerprint"}, string(keyID))...,
)
if err != nil {
return nil, fmt.Errorf(
"gpg fingerprint lookup failed: %w: %s", err, stderr.String(),
)
}
fpr, ok := parseFingerprint(stdout.String())
if !ok {
return nil, fmt.Errorf("%w: %s", errFingerprintNotFound, keyID)
}
return []byte(fpr), nil
}
// gpgExtractPubKeyFingerprint imports a public key into a temporary keyring
// and extracts its fingerprint. This verifies the key is valid and returns
// the actual fingerprint from the key material.
func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, error) {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-fingerprint-*")
if err != nil {
return "", fmt.Errorf("failed to create temp dir: %w", err)
}
defer func() { _ = os.RemoveAll(tmpDir) }()
// Set restrictive permissions
err = os.Chmod(tmpDir, privateDirPerms)
if err != nil {
return "", fmt.Errorf("failed to set temp dir permissions: %w", err)
}
// Write public key to temp file
pubKeyFile := filepath.Join(tmpDir, "pubkey.asc")
err = os.WriteFile(pubKeyFile, pubKey, privateFilePerms)
if err != nil {
return "", fmt.Errorf("failed to write public key: %w", err)
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
return "", fmt.Errorf(
"failed to import public key: %w: %s", err, importStderr.String(),
)
}
// List keys to get fingerprint
listStdout, listStderr, err := runGPG(ctx, nil,
"--homedir", tmpDir,
"--with-colons",
"--fingerprint",
)
if err != nil {
return "", fmt.Errorf(
"failed to list keys: %w: %s", err, listStderr.String(),
)
}
fpr, ok := parseFingerprint(listStdout.String())
if !ok {
return "", errImportedFPRNotFound
}
return fpr, nil
}
// gpgVerify verifies a detached signature against data using the provided public key.
// It creates a temporary keyring to import the public key for verification.
func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-verify-*")
if err != nil {
return fmt.Errorf("failed to create temp dir: %w", err)
}
defer func() { _ = os.RemoveAll(tmpDir) }()
// Set restrictive permissions
err = os.Chmod(tmpDir, privateDirPerms)
if err != nil {
return fmt.Errorf("failed to set temp dir permissions: %w", err)
}
// Write public key to temp file
pubKeyFile := filepath.Join(tmpDir, "pubkey.asc")
err = os.WriteFile(pubKeyFile, pubKey, privateFilePerms)
if err != nil {
return fmt.Errorf("failed to write public key: %w", err)
}
// Write signature to temp file
sigFile := filepath.Join(tmpDir, "signature.asc")
err = os.WriteFile(sigFile, signature, privateFilePerms)
if err != nil {
return fmt.Errorf("failed to write signature: %w", err)
}
// Write data to temp file
dataFile := filepath.Join(tmpDir, "data")
err = os.WriteFile(dataFile, data, privateFilePerms)
if err != nil {
return fmt.Errorf("failed to write data: %w", err)
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
return fmt.Errorf(
"failed to import public key: %w: %s", err, importStderr.String(),
)
}
// Verify the signature
_, verifyStderr, err := runGPG(ctx, nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, gpgOptVerify},
sigFile, dataFile)...,
)
if err != nil {
return fmt.Errorf(
"signature verification failed: %w: %s", err, verifyStderr.String(),
)
}
return nil
}
mfer/gpg_test.go
-488
View File
@@ -1,488 +0,0 @@
//nolint:testpackage // white-box tests exercise unexported internals
package mfer
import (
"bytes"
"context"
"io"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// testGPGEnv sets up a temporary GPG home directory with a test key.
// Returns the key ID and the GPG home directory; callers must point
// GNUPGHOME at the returned directory (via t.Setenv) before using the
// gpg helpers under test.
func testGPGEnv(t *testing.T) (GPGKeyID, string) {
t.Helper()
// Check if gpg is installed
_, err := exec.LookPath("gpg")
if err != nil {
t.Skip("gpg not installed, skipping signing test")
}
// Create temporary GPG home directory (0700 by default)
gpgHome := t.TempDir()
// Generate a test key with no passphrase
keyParams := `%no-protection
Key-Type: RSA
Key-Length: 2048
Name-Real: MFER Test Key
Name-Email: test@mfer.test
Expire-Date: 0
%commit
`
paramsFile := filepath.Join(gpgHome, "key-params")
require.NoError(t, os.WriteFile(paramsFile, []byte(keyParams), 0o600))
ctx, cancel := context.WithTimeout(context.Background(), gpgTimeout)
defer cancel()
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
cmd := exec.CommandContext(ctx, "gpg",
"--batch", "--gen-key", paramsFile)
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
output, err := cmd.CombinedOutput()
if err != nil {
t.Skipf("failed to generate test GPG key: %v: %s", err, output)
}
// Get the key fingerprint
cmd = exec.CommandContext(ctx, "gpg",
"--list-keys", "--with-colons", "test@mfer.test")
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
output, err = cmd.Output()
if err != nil {
t.Fatalf("failed to list test key: %v", err)
}
// Parse fingerprint from output
var keyID string
for _, line := range strings.Split(string(output), "\n") {
fields := strings.Split(line, ":")
if len(fields) >= gpgFingerprintMinFields &&
fields[0] == gpgFingerprintField {
keyID = fields[9]
break
}
}
if keyID == "" {
t.Fatal("failed to find test key fingerprint")
}
return GPGKeyID(keyID), gpgHome
}
func TestGPGSign(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
assert.NotEmpty(t, sig)
assert.Contains(t, string(sig), "-----BEGIN PGP SIGNATURE-----")
assert.Contains(t, string(sig), "-----END PGP SIGNATURE-----")
}
func TestGPGExportPublicKey(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
assert.NotEmpty(t, pubKey)
assert.Contains(t, string(pubKey), "-----BEGIN PGP PUBLIC KEY BLOCK-----")
assert.Contains(t, string(pubKey), "-----END PGP PUBLIC KEY BLOCK-----")
}
func TestGPGGetKeyFingerprint(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
fingerprint, err := gpgGetKeyFingerprint(context.Background(), keyID)
require.NoError(t, err)
assert.NotEmpty(t, fingerprint)
// The fingerprint should be 40 hex chars
assert.Len(t, fingerprint, 40, "fingerprint should be 40 hex chars")
}
// TestGPGArgsSeparatesPositionals pins that caller-supplied values are
// placed after an end-of-options marker. Key IDs arrive from --sign-key
// and MFER_SIGN_KEY as bare positional arguments, so without the marker
// a value beginning with "-" would be parsed by gpg as one of its own
// options.
func TestGPGArgsSeparatesPositionals(t *testing.T) {
t.Parallel()
assert.Equal(t,
[]string{"--opt-a", "--opt-b", "--", "--version"},
gpgArgs([]string{"--opt-a", "--opt-b"}, "--version"))
assert.Equal(t,
[]string{"--opt-c", "--", "sig", "data"},
gpgArgs([]string{"--opt-c"}, "sig", "data"))
assert.Equal(t, []string{"--opt-d", "--"},
gpgArgs([]string{"--opt-d"}))
}
// TestGPGOptionLikeKeyIDIsNotAnOption drives real gpg with a key ID that
// looks like an option and asserts it is treated as a (nonexistent) key
// rather than executed as gpg's own --version.
func TestGPGOptionLikeKeyIDIsNotAnOption(t *testing.T) {
_, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(context.Background(), GPGKeyID("--version"))
require.Error(t, err)
require.ErrorIs(t, err, errGPGKeyNotFound)
assert.NotContains(t, string(pubKey), "gpg (GnuPG)")
fpr, err := gpgGetKeyFingerprint(context.Background(), GPGKeyID("--version"))
require.Error(t, err)
assert.NotContains(t, string(fpr), "gpg (GnuPG)")
}
func TestGPGSignInvalidKey(t *testing.T) {
// Set up test environment (we need GNUPGHOME set)
_, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
_, err := gpgSign(context.Background(), data,
GPGKeyID("NONEXISTENT_KEY_ID_12345"))
assert.Error(t, err)
}
func TestBuilderWithSigning(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
// Create a builder with signing options
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{
KeyID: keyID,
})
// Add a test file
content := []byte("test file content")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil)
require.NoError(t, err)
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are populated
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.NotNil(t, manifest.pbOuter)
assert.NotEmpty(t, manifest.pbOuter.GetSignature(),
"signature should be populated")
assert.NotEmpty(t, manifest.pbOuter.GetSigner(), "signer should be populated")
assert.NotEmpty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be populated")
// Verify signature is a valid PGP signature
assert.Contains(t, string(manifest.pbOuter.GetSignature()),
"-----BEGIN PGP SIGNATURE-----")
// Verify public key is a valid PGP public key block
assert.Contains(t, string(manifest.pbOuter.GetSigningPubKey()),
"-----BEGIN PGP PUBLIC KEY BLOCK-----")
}
func TestScannerWithSigning(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
// Create in-memory filesystem with test files
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755))
require.NoError(t,
afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0o644))
require.NoError(t,
afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0o644))
// Create scanner with signing options
opts := &ScannerOptions{
Fs: fs,
SigningOptions: &SigningOptions{
KeyID: keyID,
},
}
s := NewScannerWithOptions(opts)
// Enumerate files
require.NoError(t, s.EnumeratePath("/testdir", nil))
assert.Equal(t, FileCount(2), s.FileCount())
// Generate signed manifest
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
// Parse and verify
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
assert.NotEmpty(t, manifest.pbOuter.GetSignature())
assert.NotEmpty(t, manifest.pbOuter.GetSigner())
assert.NotEmpty(t, manifest.pbOuter.GetSigningPubKey())
}
func TestGPGVerify(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign and verify")
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
// Verify the signature
err = gpgVerify(context.Background(), data, sig, pubKey)
require.NoError(t, err)
}
func TestGPGVerifyInvalidSignature(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
require.NoError(t, err)
// Try to verify with different data - should fail
wrongData := []byte("different data")
err = gpgVerify(context.Background(), wrongData, sig, pubKey)
assert.Error(t, err)
}
func TestGPGVerifyBadPublicKey(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
sig, err := gpgSign(context.Background(), data, keyID)
require.NoError(t, err)
// Try to verify with invalid public key - should fail
badPubKey := []byte("not a valid public key")
err = gpgVerify(context.Background(), data, sig, badPubKey)
assert.Error(t, err)
}
func TestManifestSignatureVerification(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
// Create a builder with signing options
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{
KeyID: keyID,
})
// Add a test file
content := []byte("test file content for verification")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil)
require.NoError(t, err)
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest - signature should be verified during load
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.NotNil(t, manifest)
// Signature should be present and valid
assert.NotEmpty(t, manifest.pbOuter.GetSignature())
}
func TestManifestTamperedSignatureFails(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
// Create a signed manifest
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{
KeyID: keyID,
})
content := []byte("test file content")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil)
require.NoError(t, err)
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Tamper with the signature by replacing some bytes
data := buf.Bytes()
// Find and modify a byte in the signature portion
for i := range data {
if i > 100 && data[i] == 'A' {
data[i] = 'B'
break
}
}
// Try to load the tampered manifest - should fail
_, err = NewManifestFromReader(bytes.NewReader(data))
assert.Error(t, err)
}
func TestBuilderWithoutSigning(t *testing.T) {
t.Parallel()
// Create a builder without signing options
b := NewBuilder()
// Add a test file
content := []byte("test file content")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, reader, nil)
require.NoError(t, err)
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are empty
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.NotNil(t, manifest.pbOuter)
assert.Empty(t, manifest.pbOuter.GetSignature(),
"signature should be empty when not signing")
assert.Empty(t, manifest.pbOuter.GetSigner(),
"signer should be empty when not signing")
assert.Empty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be empty when not signing")
}
// fakeGPGPath writes script as an executable named gpg into a temporary
// directory and returns a PATH value with that directory first.
func fakeGPGPath(t *testing.T, script string) string {
t.Helper()
binDir := t.TempDir()
//nolint:gosec // G306: the fake gpg has to be executable
require.NoError(t, os.WriteFile(filepath.Join(binDir, "gpg"),
[]byte(script), 0o700))
return binDir + string(os.PathListSeparator) + os.Getenv("PATH")
}
// TestGPGTimeoutKillsGPG puts a fake gpg that never finishes first on
// PATH and checks that a run past its deadline is killed and reported as
// a timeout of the named operation, instead of hanging.
func TestGPGTimeoutKillsGPG(t *testing.T) {
t.Setenv("PATH", fakeGPGPath(t, "#!/bin/sh\nexec sleep 10\n"))
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
require.ErrorIs(t, err, context.DeadlineExceeded)
assert.Contains(t, err.Error(), "gpg sign failed: gpg timed out")
}
// TestGPGCancelWhenChildHoldsOutput uses a fake gpg that runs sleep as a
// child instead of exec-ing it, the way a wrapper script around the real
// gpg might. Killing the fake gpg leaves sleep holding its stdout and
// stderr open; the call must still return once ctx ends instead of waiting
// for sleep to exit. The fake gpg writes the process ID of sleep to a named
// pipe; the test ends ctx only after reading it, so sleep is running by
// then, and kills sleep before returning.
func TestGPGCancelWhenChildHoldsOutput(t *testing.T) {
pidPipe := filepath.Join(t.TempDir(), "sleep.pid")
require.NoError(t, syscall.Mkfifo(pidPipe, 0o600))
// sleep outlasts the 10 s wait below, so a call that waits for it fails.
t.Setenv("PATH", fakeGPGPath(t,
"#!/bin/sh\nsleep 60 &\necho $! >'"+pidPipe+"'\nwait\n"))
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
signErr := make(chan error, 1)
go func() {
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
signErr <- err
}()
pid, err := os.ReadFile(pidPipe) //nolint:gosec // G304: path inside t.TempDir()
require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(pid)))
require.NoError(t, err)
sleep, err := os.FindProcess(n)
require.NoError(t, err)
t.Cleanup(func() { require.NoError(t, sleep.Kill()) })
cancel()
// The call should return about gpgWaitDelay (one second) after the
// cancel. 10 s is far above that and well under the 30 s test timeout,
// which would abort the whole package before the cleanup kills sleep.
select {
case err := <-signErr:
require.ErrorIs(t, err, context.Canceled)
case <-time.After(10 * time.Second):
t.Fatal("the call waited for the child holding gpg's output to exit")
}
}
// TestBuildPassesContextToSigning checks that a caller can cancel the gpg
// runs that sign a manifest through the context given to Build.
func TestBuildPassesContextToSigning(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{KeyID: "any"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
require.ErrorIs(t, b.Build(ctx, io.Discard), context.Canceled)
}
mfer/manifest.go
+2 -2
View File
@@ -10,7 +10,7 @@ import (
)
var (
errOuterNotSet = errors.New("pbOuter not set")
errOuterNotSet = errors.New("outer message not set")
errUUIDNotSet = errors.New("UUID not set")
errSHA256NotSet = errors.New("SHA256 hash not set")
)
@@ -65,7 +65,7 @@ func (m *manifest) signatureString() (string, error) {
mh, err := multihash.Encode(m.pbOuter.GetSha256(), multihash.SHA2_256)
if err != nil {
return "", fmt.Errorf("failed to encode multihash: %w", err)
return "", fmt.Errorf("encode multihash: %w", err)
}
uuidStr := hex.EncodeToString(m.pbOuter.GetUuid())
mfer/mf.go
-3
View File
@@ -1,3 +0,0 @@
package mfer
//go:generate protoc ./mf.proto --go_out=paths=source_relative:.
mfer/mf.pb.go
+13 -3
View File
@@ -1,6 +1,6 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
// protoc-gen-go v1.36.11
// protoc-gen-go v1.36.12
// protoc v6.33.4
// source: mf.proto
@@ -340,6 +340,8 @@ type MFFilePath struct {
MimeType *string `protobuf:"bytes,301,opt,name=mimeType,proto3,oneof" json:"mimeType,omitempty"`
Mtime *Timestamp `protobuf:"bytes,302,opt,name=mtime,proto3,oneof" json:"mtime,omitempty"`
Ctime *Timestamp `protobuf:"bytes,303,opt,name=ctime,proto3,oneof" json:"ctime,omitempty"`
// permission bits, at most 0777; 0 when not recorded
Mode uint32 `protobuf:"varint,304,opt,name=mode,proto3" json:"mode,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -416,6 +418,13 @@ func (x *MFFilePath) GetCtime() *Timestamp {
return nil
}
func (x *MFFilePath) GetMode() uint32 {
if x != nil {
return x.Mode
}
return 0
}
type MFFileChecksum struct {
state protoimpl.MessageState `protogen:"open.v1"`
// 1.0 golang implementation must write a multihash here
@@ -561,7 +570,7 @@ const file_mf_proto_rawDesc = "" +
"\n" +
"_signatureB\t\n" +
"\a_signerB\x10\n" +
"\x0e_signingPubKey\"\xf0\x01\n" +
"\x0e_signingPubKey\"\x85\x02\n" +
"\n" +
"MFFilePath\x12\x12\n" +
"\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" +
@@ -571,7 +580,8 @@ const file_mf_proto_rawDesc = "" +
"\x05mtime\x18\xae\x02 \x01(\v2\n" +
".TimestampH\x01R\x05mtime\x88\x01\x01\x12&\n" +
"\x05ctime\x18\xaf\x02 \x01(\v2\n" +
".TimestampH\x02R\x05ctime\x88\x01\x01B\v\n" +
".TimestampH\x02R\x05ctime\x88\x01\x01\x12\x13\n" +
"\x04mode\x18\xb0\x02 \x01(\rR\x04modeB\v\n" +
"\t_mimeTypeB\b\n" +
"\x06_mtimeB\b\n" +
"\x06_ctime\".\n" +
mfer/mf.proto
+2
View File
@@ -59,6 +59,8 @@ message MFFilePath {
optional string mimeType = 301;
optional Timestamp mtime = 302;
optional Timestamp ctime = 303;
// permission bits, at most 0777; 0 when not recorded
uint32 mode = 304;
}
message MFFileChecksum {
mfer/mf.proto.sha256
+1 -1
View File
@@ -1 +1 @@
fa6fceaba5553c8667c631994535fe5c307c8adb19f3ad289babf79a0f438650 mf.proto
3d4dcb0b2f4640dd3ae6bb48b833e9f26ae9771e2d3e88bd646e7f1724dda654 mf.proto
mfer/mf_test.go
+26
View File
@@ -8,6 +8,8 @@ import (
"testing"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/reflect/protoreflect"
"sneak.berlin/go/mfer/mfer"
)
// mf.pb.go is generated from mf.proto and committed. `make generate`
@@ -27,3 +29,27 @@ func TestGeneratedCodeMatchesProto(t *testing.T) {
"mfer/mf.proto has changed since mfer/mf.pb.go was generated "+
"from it: run `make generate` and commit the result")
}
// A file entry has exactly the fields docs/FORMAT.md lists for MFFilePath.
func TestFileEntryFieldsMatchSpec(t *testing.T) {
t.Parallel()
want := map[string]protoreflect.FieldNumber{
"path": 1,
"size": 2,
"hashes": 3,
"mimeType": 301,
"mtime": 302,
"ctime": 303,
"mode": 304,
}
got := map[string]protoreflect.FieldNumber{}
fields := (&mfer.MFFilePath{}).ProtoReflect().Descriptor().Fields()
for i := range fields.Len() {
got[string(fields.Get(i).Name())] = fields.Get(i).Number()
}
require.Equal(t, want, got)
}
mfer/openpgp.go
+376
View File
@@ -0,0 +1,376 @@
package mfer
import (
"bytes"
"encoding/hex"
"errors"
"fmt"
"io"
"slices"
"strings"
"github.com/ProtonMail/go-crypto/openpgp"
"github.com/ProtonMail/go-crypto/openpgp/armor"
pgperrors "github.com/ProtonMail/go-crypto/openpgp/errors"
"github.com/ProtonMail/go-crypto/openpgp/packet"
)
const (
// The tags of OpenPGP signature, key and subkey packets (RFC 9580,
// section 5). Subkeys have tags of their own, so each secret or public
// key packet is one primary key.
signaturePacketTag = 2
secretKeyPacketTag = 5
publicKeyPacketTag = 6
secretSubkeyPacketTag = 7
publicSubkeyPacketTag = 14
// In the body of a key or subkey packet the algorithm octet follows
// the version octet and the four-octet creation time, and from version
// 5 on a four-octet length as well (RFC 9580, section 5.5.2).
keyAlgorithmOffset = 5
firstKeyVersionWithLength = 5
keyAlgorithmOffsetAfterLength = 9
// signingKeyVersion is the only OpenPGP key version mfer signs with.
// Its fingerprints are 40 hex characters, the length
// --require-signature takes.
signingKeyVersion = 4
// armorBegin and armorEnd start the lines that begin and end an
// armored block.
armorBegin = "-----BEGIN "
armorEnd = "-----END "
)
var (
errKeyCount = errors.New("must hold exactly one key")
errDSAKey = errors.New("must not hold a DSA key")
errSecretKey = errors.New("must not hold a secret key")
errNoSecretKey = errors.New("signing key file holds no secret key")
errNotV4Key = errors.New("signing key must be an OpenPGP version 4 key, " +
"the only kind whose fingerprint --require-signature takes")
errNoPassphrase = errors.New(
"signing key is protected and no passphrase was given")
errNotOneSignature = errors.New(
"signature must hold exactly one signature")
errNotOneArmoredBlock = errors.New(
"must be exactly one armored block and nothing else")
errMalformedArmor = errors.New("armor is malformed")
)
// SigningOptions holds the key a manifest is signed with.
type SigningOptions struct {
// SecretKey is an OpenPGP secret key, armored or binary, as
// gpg --export-secret-keys writes it. It must hold one primary key.
SecretKey []byte
// Passphrase unlocks SecretKey when it is protected.
Passphrase []byte
}
// SecretKeyIsProtected reports whether the OpenPGP secret key secretKey,
// armored or binary, needs a passphrase to sign. It fails unless
// secretKey holds one version 4 primary key with its secret key.
func SecretKeyIsProtected(secretKey []byte) (bool, error) {
key, err := readSecretKey(secretKey)
if err != nil {
return false, err
}
return isProtected(key), nil
}
// CheckSigningKey fails unless opts can sign now: opts.SecretKey must hold
// one version 4 primary key with a secret key that may sign and has not
// expired or been revoked, and opts.Passphrase must unlock it when it is
// protected. It lets a caller find a key that cannot sign before it builds
// a manifest.
func CheckSigningKey(opts *SigningOptions) error {
key, err := readSigningKey(opts)
if err != nil {
return err
}
// Signing nothing fails wherever signing the manifest would.
err = openpgp.DetachSign(io.Discard, key, bytes.NewReader(nil), nil)
if err != nil {
return fmt.Errorf("signing key cannot sign: %w", err)
}
return nil
}
// readSigningKey returns the key in opts.SecretKey, unlocked with
// opts.Passphrase if it is protected.
func readSigningKey(opts *SigningOptions) (*openpgp.Entity, error) {
key, err := readSecretKey(opts.SecretKey)
if err != nil {
return nil, err
}
if !isProtected(key) {
return key, nil
}
if len(opts.Passphrase) == 0 {
return nil, errNoPassphrase
}
err = key.DecryptPrivateKeys(opts.Passphrase)
if err != nil {
return nil, fmt.Errorf("unlock signing key: %w", err)
}
return key, nil
}
// readSecretKey returns the one key in secretKey, armored or binary,
// which must be a version 4 key and include its secret key.
func readSecretKey(secretKey []byte) (*openpgp.Entity, error) {
key, err := readOneKey(secretKey, "signing key file", false)
if err != nil {
return nil, err
}
if key.PrimaryKey.Version != signingKeyVersion {
return nil, fmt.Errorf("%w; this key is version %d",
errNotV4Key, key.PrimaryKey.Version)
}
if key.PrivateKey == nil {
return nil, errNoSecretKey
}
return key, nil
}
// readOneKey returns the key in data, armored or binary, which must hold
// exactly one primary key and no DSA key or subkey, and when publicOnly no
// secret key or subkey either. what names data in errors.
func readOneKey(data []byte, what string, publicOnly bool) (*openpgp.Entity, error) {
packets, err := dearmor(data)
if err != nil {
return nil, fmt.Errorf("read %s: %w", what, err)
}
keys, err := countPackets(packets, secretKeyPacketTag, publicKeyPacketTag)
if err != nil {
return nil, fmt.Errorf("read %s: %w", what, err)
}
if keys != 1 {
return nil, fmt.Errorf("%s %w, found %d", what, errKeyCount, keys)
}
// openpgp.ReadKeyRing checks the numbers of every secret key it reads,
// and an ElGamal secret subkey with a very large prime makes that take
// minutes.
secretKeys, err := countPackets(packets, secretKeyPacketTag, secretSubkeyPacketTag)
if err != nil {
return nil, fmt.Errorf("read %s: %w", what, err)
}
if publicOnly && secretKeys != 0 {
return nil, fmt.Errorf("%s %w", what, errSecretKey)
}
// openpgp.ReadKeyRing also checks every self-signature, and a DSA key
// with very large numbers makes each check take seconds to minutes.
dsa, err := holdsDSAKey(packets)
if err != nil {
return nil, fmt.Errorf("read %s: %w", what, err)
}
if dsa {
return nil, fmt.Errorf("%s %w", what, errDSAKey)
}
keyring, err := openpgp.ReadKeyRing(bytes.NewReader(packets))
if err != nil {
return nil, fmt.Errorf("read %s: %w", what, err)
}
// openpgp.ReadKeyRing also reads a subkey packet at the start as a
// primary key.
if len(keyring) != 1 {
return nil, fmt.Errorf("%s %w, found %d", what, errKeyCount, len(keyring))
}
return keyring[0], nil
}
// isProtected reports whether any secret key in key needs a passphrase.
func isProtected(key *openpgp.Entity) bool {
if key.PrivateKey.Encrypted {
return true
}
for _, subkey := range key.Subkeys {
if subkey.PrivateKey != nil && subkey.PrivateKey.Encrypted {
return true
}
}
return false
}
// armoredPublicKey returns the public part of key, armored.
func armoredPublicKey(key *openpgp.Entity) ([]byte, error) {
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PublicKeyType, nil)
if err != nil {
return nil, err
}
err = key.Serialize(w)
if err != nil {
return nil, fmt.Errorf("write public key: %w", err)
}
err = w.Close()
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
// fingerprint returns the fingerprint of key's primary key in upper-case
// hex, as gpg prints it.
func fingerprint(key *openpgp.Entity) string {
return strings.ToUpper(hex.EncodeToString(key.PrimaryKey.Fingerprint))
}
// verifySignature checks that signature is one good OpenPGP signature
// over data, made by the one primary key in pubKey or one of its subkeys,
// and returns that primary key's fingerprint. signature and pubKey may each
// be armored or binary.
func verifySignature(data, signature, pubKey []byte) (string, error) {
key, err := readOneKey(pubKey, "embedded public key block", true)
if err != nil {
return "", err
}
sigData, err := dearmor(signature)
if err != nil {
return "", fmt.Errorf("read signature: %w", err)
}
sigs, err := countPackets(sigData, signaturePacketTag)
if err != nil {
return "", fmt.Errorf("read signature: %w", err)
}
if sigs != 1 {
return "", fmt.Errorf("%w, found %d", errNotOneSignature, sigs)
}
_, err = openpgp.CheckDetachedSignature(openpgp.EntityList{key},
bytes.NewReader(data), bytes.NewReader(sigData), nil)
// A manifest outlives its signing key, so a signature by a key that
// has expired since is still good.
if err != nil && !errors.Is(err, pgperrors.ErrKeyExpired) {
return "", fmt.Errorf("verify signature: %w", err)
}
return fingerprint(key), nil
}
// dearmor returns the binary OpenPGP data in data: data itself when it is
// not armored, or else the body of its armored block. Armored data must be
// one block and nothing else: its first line is the only BEGIN line and
// its last line the only END line, white space around them aside.
// armor.Decode skips any text before a BEGIN line and reads only the first
// block, so without this a second key or signature would go unseen.
func dearmor(data []byte) ([]byte, error) {
if !bytes.Contains(data, []byte(armorBegin)) {
return data, nil
}
text := bytes.TrimSpace(data)
lastLine := text[bytes.LastIndexByte(text, '\n')+1:]
if !bytes.HasPrefix(text, []byte(armorBegin)) ||
bytes.Count(text, []byte(armorBegin)) != 1 ||
!bytes.HasPrefix(lastLine, []byte(armorEnd)) ||
bytes.Count(text, []byte(armorEnd)) != 1 {
return nil, errNotOneArmoredBlock
}
// armor.Decode passes over a block it cannot read, such as one with a
// header line that has no colon, and returns io.EOF on finding no
// other.
block, err := armor.Decode(bytes.NewReader(text))
if err != nil {
return nil, errMalformedArmor
}
body, err := io.ReadAll(block.Body)
if err != nil {
return nil, fmt.Errorf("%w: %w", errMalformedArmor, err)
}
return body, nil
}
// countPackets returns how many packets in the binary OpenPGP data have
// one of tags. It reads only each packet's header, so it also counts
// packets that openpgp.ReadKeyRing skips, such as a key with no user ID
// or of an algorithm it does not know.
func countPackets(data []byte, tags ...uint8) (int, error) {
packets := packet.NewOpaqueReader(bytes.NewReader(data))
count := 0
for {
p, err := packets.Next()
if errors.Is(err, io.EOF) {
return count, nil
}
if err != nil {
return 0, err
}
if slices.Contains(tags, p.Tag) {
count++
}
}
}
// holdsDSAKey reports whether any key or subkey packet in the binary
// OpenPGP data holds a DSA key. It reads each packet's algorithm octet
// rather than parsing the packet, since parsing a secret key packet checks
// its numbers, which for a DSA key with very large numbers is as slow as
// checking a self-signature.
func holdsDSAKey(data []byte) (bool, error) {
packets := packet.NewOpaqueReader(bytes.NewReader(data))
for {
p, err := packets.Next()
if errors.Is(err, io.EOF) {
return false, nil
}
if err != nil {
return false, err
}
if !slices.Contains([]uint8{
secretKeyPacketTag, publicKeyPacketTag,
secretSubkeyPacketTag, publicSubkeyPacketTag,
}, p.Tag) {
continue
}
offset := keyAlgorithmOffset
if len(p.Contents) > 0 && p.Contents[0] >= firstKeyVersionWithLength {
offset = keyAlgorithmOffsetAfterLength
}
if len(p.Contents) > offset &&
packet.PublicKeyAlgorithm(p.Contents[offset]) == packet.PubKeyAlgoDSA {
return true, nil
}
}
}
mfer/openpgp_test.go
+803
View File
@@ -0,0 +1,803 @@
//nolint:testpackage // white-box tests exercise unexported internals
package mfer
import (
"bytes"
"context"
"crypto/dsa" //nolint:staticcheck // SA1019: tests need a DSA key to refuse
"io"
"math/big"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
"github.com/ProtonMail/go-crypto/openpgp"
"github.com/ProtonMail/go-crypto/openpgp/armor"
"github.com/ProtonMail/go-crypto/openpgp/elgamal"
"github.com/ProtonMail/go-crypto/openpgp/packet"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
)
// newTestKey returns a new Ed25519 key, which is quick to make, for
// "MFER Test Key <test@mfer.test>". config may set when it is made and how
// long it lasts.
func newTestKey(t *testing.T, config *packet.Config) *openpgp.Entity {
t.Helper()
if config == nil {
config = &packet.Config{}
}
config.Algorithm = packet.PubKeyAlgoEdDSA
key, err := openpgp.NewEntity("MFER Test Key", "", "test@mfer.test", config)
require.NoError(t, err)
return key
}
// armoredSecretKeys returns keys with their secret keys in one armored
// block, as gpg --export-secret-keys --armor writes them.
func armoredSecretKeys(t *testing.T, keys ...*openpgp.Entity) []byte {
t.Helper()
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PrivateKeyType, nil)
require.NoError(t, err)
for _, key := range keys {
require.NoError(t, key.SerializePrivateWithoutSigning(w, nil))
}
require.NoError(t, w.Close())
return buf.Bytes()
}
// armoredPublicKeys returns the public parts of keys in one armored block,
// as gpg --export --armor writes them.
func armoredPublicKeys(t *testing.T, keys ...*openpgp.Entity) []byte {
t.Helper()
var buf bytes.Buffer
w, err := armor.Encode(&buf, openpgp.PublicKeyType, nil)
require.NoError(t, err)
for _, key := range keys {
require.NoError(t, key.Serialize(w))
}
require.NoError(t, w.Close())
return buf.Bytes()
}
// testSigningOptions returns signing options for a new key with no
// passphrase.
func testSigningOptions(t *testing.T) *SigningOptions {
t.Helper()
return &SigningOptions{SecretKey: armoredSecretKeys(t, newTestKey(t, nil))}
}
// joinArmored returns the armored block first followed by the armored
// block second on the next line. armor.Encode ends a block without a
// newline, unlike gpg, and a block only starts at the start of a line.
func joinArmored(first, second []byte) []byte {
return slices.Concat(first, []byte("\n"), second)
}
// signedTestManifest returns a manifest of one file signed with opts.
func signedTestManifest(t *testing.T, opts *SigningOptions) []byte {
t.Helper()
b := NewBuilder()
b.SetSigningOptions(opts)
content := []byte("signed file content")
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0,
bytes.NewReader(content), nil)
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
return buf.Bytes()
}
// rewriteOuter returns manifest with its outer message changed by edit.
// A signature stays good as long as edit leaves the UUID and hash alone.
func rewriteOuter(t *testing.T, manifest []byte, edit func(*MFFileOuter)) []byte {
t.Helper()
outer := new(MFFileOuter)
require.NoError(t, proto.Unmarshal(manifest[len(MAGIC):], outer))
edit(outer)
data, err := proto.Marshal(outer)
require.NoError(t, err)
return append([]byte(MAGIC), data...)
}
func TestBuilderWithSigning(t *testing.T) {
t.Parallel()
key := newTestKey(t, nil)
// Create a builder with signing options
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{SecretKey: armoredSecretKeys(t, key)})
// Add a test file
content := []byte("test file content")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err)
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are populated
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.NotNil(t, manifest.pbOuter)
assert.NotEmpty(t, manifest.pbOuter.GetSignature(),
"signature should be populated")
assert.NotEmpty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be populated")
// The signer is the key's fingerprint in 40 upper-case hex characters.
assert.Equal(t, fingerprint(key), string(manifest.pbOuter.GetSigner()))
assert.Regexp(t, "^[0-9A-F]{40}$", string(manifest.pbOuter.GetSigner()))
// Verify signature is a valid PGP signature
assert.Contains(t, string(manifest.pbOuter.GetSignature()),
"-----BEGIN PGP SIGNATURE-----")
// Verify public key is a valid PGP public key block
assert.Contains(t, string(manifest.pbOuter.GetSigningPubKey()),
"-----BEGIN PGP PUBLIC KEY BLOCK-----")
}
func TestScannerWithSigning(t *testing.T) {
t.Parallel()
// Create in-memory filesystem with test files
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/testdir", 0o755))
require.NoError(t,
afero.WriteFile(fs, "/testdir/file1.txt", []byte("content1"), 0o644))
require.NoError(t,
afero.WriteFile(fs, "/testdir/file2.txt", []byte("content2"), 0o644))
// Create scanner with signing options
opts := &ScannerOptions{
Fs: fs,
SigningOptions: testSigningOptions(t),
}
s := NewScannerWithOptions(opts)
// Enumerate files
require.NoError(t, s.EnumeratePath("/testdir", nil))
assert.Equal(t, FileCount(2), s.FileCount())
// Generate signed manifest
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
// Parse and verify
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
assert.NotEmpty(t, manifest.pbOuter.GetSignature())
assert.NotEmpty(t, manifest.pbOuter.GetSigner())
assert.NotEmpty(t, manifest.pbOuter.GetSigningPubKey())
}
// TestSigningWithBinarySecretKey signs with a secret key that is not
// armored, as gpg --export-secret-keys writes it without --armor.
func TestSigningWithBinarySecretKey(t *testing.T) {
t.Parallel()
var secretKey bytes.Buffer
require.NoError(t, newTestKey(t, nil).SerializePrivateWithoutSigning(&secretKey, nil))
_, err := NewManifestFromReader(bytes.NewReader(
signedTestManifest(t, &SigningOptions{SecretKey: secretKey.Bytes()})))
require.NoError(t, err)
}
// TestSigningWithProtectedKey signs with a key protected by a passphrase:
// with the passphrase, without one, and with a wrong one.
func TestSigningWithProtectedKey(t *testing.T) {
t.Parallel()
key := newTestKey(t, nil)
require.NoError(t, key.EncryptPrivateKeys([]byte("right"), nil))
secretKey := armoredSecretKeys(t, key)
protected, err := SecretKeyIsProtected(secretKey)
require.NoError(t, err)
assert.True(t, protected)
_, err = NewManifestFromReader(bytes.NewReader(signedTestManifest(t,
&SigningOptions{SecretKey: secretKey, Passphrase: []byte("right")})))
require.NoError(t, err)
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{SecretKey: secretKey})
require.ErrorIs(t, b.Build(context.Background(), io.Discard), errNoPassphrase)
b.SetSigningOptions(&SigningOptions{
SecretKey: secretKey, Passphrase: []byte("wrong"),
})
assert.ErrorContains(t, b.Build(context.Background(), io.Discard),
"unlock signing key")
}
func TestSecretKeyIsProtectedWithoutPassphrase(t *testing.T) {
t.Parallel()
protected, err := SecretKeyIsProtected(testSigningOptions(t).SecretKey)
require.NoError(t, err)
assert.False(t, protected)
}
// TestSigningKeyFileWithTwoKeys signs with a key file that holds two keys,
// as gpg writes it for a user ID that two keys have. It names no one key
// to sign with, so signing must fail.
func TestSigningKeyFileWithTwoKeys(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{SecretKey: armoredSecretKeys(t,
newTestKey(t, nil), newTestKey(t, nil))})
err := b.Build(context.Background(), io.Discard)
require.ErrorIs(t, err, errKeyCount)
assert.EqualError(t, err, "signing key file must hold exactly one key, found 2")
}
// TestSigningKeyFileWithoutSecretKey signs with a file that holds only a
// public key.
func TestSigningKeyFileWithoutSecretKey(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{
SecretKey: armoredPublicKeys(t, newTestKey(t, nil)),
})
require.ErrorIs(t, b.Build(context.Background(), io.Discard), errNoSecretKey)
}
// TestCheckSigningKeyRefusesKeyThatCannotSign checks a key that can sign,
// and keys that read and need no passphrase but cannot sign now: one that
// expired in 2020 and one that has been revoked. CheckSigningKey must
// refuse each of the two, as signing a manifest with it would fail.
func TestCheckSigningKeyRefusesKeyThatCannotSign(t *testing.T) {
t.Parallel()
require.NoError(t, CheckSigningKey(testSigningOptions(t)))
made := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
expired := newTestKey(t, &packet.Config{
Time: func() time.Time { return made },
KeyLifetimeSecs: uint32((24 * time.Hour).Seconds()),
})
revoked := newTestKey(t, nil)
require.NoError(t, revoked.RevokeKey(packet.KeyRetired, "", nil))
for name, key := range map[string]*openpgp.Entity{
"expired": expired,
"revoked": revoked,
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
assert.ErrorContains(t, CheckSigningKey(
&SigningOptions{SecretKey: armoredSecretKeys(t, key)}),
"signing key cannot sign")
})
}
}
// TestSigningRefusesVersion6Key signs with an OpenPGP version 6 key, whose
// fingerprint is 64 hex characters. mfer signs only with version 4 keys.
func TestSigningRefusesVersion6Key(t *testing.T) {
t.Parallel()
key, err := openpgp.NewEntity("MFER Test Key", "", "test@mfer.test",
&packet.Config{V6Keys: true, Algorithm: packet.PubKeyAlgoEd25519})
require.NoError(t, err)
secretKey := armoredSecretKeys(t, key)
_, err = SecretKeyIsProtected(secretKey)
require.ErrorIs(t, err, errNotV4Key)
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{SecretKey: secretKey})
require.ErrorIs(t, b.Build(context.Background(), io.Discard), errNotV4Key)
}
// TestMalformedArmorIsNamed reads a signing key file, a signature and an
// embedded public key block whose armor is malformed: a header line with
// no colon, or no blank line after the BEGIN line. Each must fail saying
// the armor is malformed.
func TestMalformedArmorIsNamed(t *testing.T) {
t.Parallel()
opts := testSigningOptions(t)
manifest := signedTestManifest(t, opts)
for name, change := range map[string]func([]byte) []byte{
"header line with no colon": func(block []byte) []byte {
return bytes.Replace(block,
[]byte("-----\n"), []byte("-----\nno colon\n"), 1)
},
"no blank line after the BEGIN line": func(block []byte) []byte {
return bytes.Replace(block, []byte("-----\n\n"), []byte("-----\n"), 1)
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
_, err := SecretKeyIsProtected(change(opts.SecretKey))
require.ErrorIs(t, err, errMalformedArmor)
for _, changed := range [][]byte{
rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.Signature = change(outer.GetSignature())
}),
rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.SigningPubKey = change(outer.GetSigningPubKey())
}),
} {
_, err = NewManifestFromReader(bytes.NewReader(changed))
require.ErrorIs(t, err, errMalformedArmor)
}
})
}
}
func TestVerifySignature(t *testing.T) {
t.Parallel()
key := newTestKey(t, nil)
data := []byte("test data to sign and verify")
var armored, binary bytes.Buffer
require.NoError(t, openpgp.ArmoredDetachSign(&armored, key,
bytes.NewReader(data), nil))
require.NoError(t, openpgp.DetachSign(&binary, key, bytes.NewReader(data), nil))
pubKey, err := armoredPublicKey(key)
require.NoError(t, err)
var binaryPubKey bytes.Buffer
require.NoError(t, key.Serialize(&binaryPubKey))
// Verifying names the key that made the signature, whether the
// signature and key are armored or not.
signer, err := verifySignature(data, armored.Bytes(), pubKey)
require.NoError(t, err)
assert.Equal(t, fingerprint(key), signer)
signer, err = verifySignature(data, binary.Bytes(), binaryPubKey.Bytes())
require.NoError(t, err)
assert.Equal(t, fingerprint(key), signer)
// A signature over other data is bad.
_, err = verifySignature([]byte("different data"), armored.Bytes(), pubKey)
require.Error(t, err)
// A public key that is not one cannot verify anything.
_, err = verifySignature(data, armored.Bytes(), []byte("not a public key"))
assert.Error(t, err)
}
// TestVerifySignatureKeyExpiredSince verifies a signature made in 2020 by
// a key that expired a day after it was made. The signature is still good.
func TestVerifySignatureKeyExpiredSince(t *testing.T) {
t.Parallel()
made := time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC)
config := &packet.Config{
Time: func() time.Time { return made },
KeyLifetimeSecs: uint32((24 * time.Hour).Seconds()),
}
key := newTestKey(t, config)
data := []byte("signed in 2020")
var sig bytes.Buffer
require.NoError(t, openpgp.ArmoredDetachSign(&sig, key, bytes.NewReader(data), config))
pubKey, err := armoredPublicKey(key)
require.NoError(t, err)
signer, err := verifySignature(data, sig.Bytes(), pubKey)
require.NoError(t, err)
assert.Equal(t, fingerprint(key), signer)
}
func TestManifestSignatureVerification(t *testing.T) {
t.Parallel()
// Parse the manifest - signature should be verified during load
manifest, err := NewManifestFromReader(bytes.NewReader(
signedTestManifest(t, testSigningOptions(t))))
require.NoError(t, err)
require.NotNil(t, manifest)
// Signature should be present and valid
assert.NotEmpty(t, manifest.pbOuter.GetSignature())
}
func TestManifestTamperedSignatureFails(t *testing.T) {
t.Parallel()
// Change one character of the signature's base64 body, which starts
// after the blank line that ends the armor headers.
data := rewriteOuter(t, signedTestManifest(t, testSigningOptions(t)),
func(outer *MFFileOuter) {
sig := outer.GetSignature()
i := bytes.Index(sig, []byte("\n\n")) + len("\n\n") + 20
sig[i]++
})
// Try to load the tampered manifest - should fail
_, err := NewManifestFromReader(bytes.NewReader(data))
assert.Error(t, err)
}
// TestManifestSignedByGPGLoads loads the signed seed of
// FuzzNewManifestFromReader, a manifest signed with gpg before mfer signed
// and verified manifests itself.
func TestManifestSignedByGPGLoads(t *testing.T) {
t.Parallel()
seed, err := os.ReadFile(filepath.Join(
"testdata", "fuzz", "FuzzNewManifestFromReader", "signed"))
require.NoError(t, err)
// After its header line the seed holds the manifest as []byte("...").
_, quoted, found := strings.Cut(string(seed), "[]byte(")
require.True(t, found)
manifest, err := strconv.Unquote(
strings.TrimSuffix(strings.TrimSpace(quoted), ")"))
require.NoError(t, err)
m, err := NewManifestFromReader(strings.NewReader(manifest))
require.NoError(t, err)
assert.Equal(t, "4F562BFB863FDC6B51B4EE88872A51176CEF23AE",
string(m.pbOuter.GetSigner()))
}
// TestManifestRefusesSecondEmbeddedKey loads manifests whose embedded
// public key block holds another key besides the key that signed it: as a
// public key, as a secret key with no user ID, which openpgp.ReadKeyRing
// skips, and written as a subkey packet at the start of the block, which
// openpgp.ReadKeyRing reads as a primary key. Loading must refuse each,
// although the signature is good and the signer field names the key that
// made it.
func TestManifestRefusesSecondEmbeddedKey(t *testing.T) {
t.Parallel()
other := newTestKey(t, nil)
signer := newTestKey(t, nil)
manifest := signedTestManifest(t,
&SigningOptions{SecretKey: armoredSecretKeys(t, signer)})
otherWithoutUserID := newTestKey(t, nil)
otherWithoutUserID.Identities = map[string]*openpgp.Identity{}
otherAsSubkey := newTestKey(t, nil)
otherAsSubkey.PrimaryKey.IsSubkey = true
for name, block := range map[string][]byte{
"public key": armoredPublicKeys(t, other, signer),
"secret key without user ID": armoredSecretKeys(t, signer, otherWithoutUserID),
"subkey packet first": armoredPublicKeys(t, otherAsSubkey, signer),
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
embedded := rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.SigningPubKey = block
})
_, err := NewManifestFromReader(bytes.NewReader(embedded))
require.ErrorIs(t, err, errKeyCount)
})
}
}
// TestManifestRefusesSecondEmbeddedKeyWithoutUserID loads a manifest whose
// embedded public key block holds, before the key that signed it, another
// key with no user ID, which openpgp.ReadKeyRing skips. Loading must
// refuse it: the block holds two keys.
func TestManifestRefusesSecondEmbeddedKeyWithoutUserID(t *testing.T) {
t.Parallel()
other := newTestKey(t, nil)
other.Identities = map[string]*openpgp.Identity{}
signer := newTestKey(t, nil)
manifest := rewriteOuter(t, signedTestManifest(t,
&SigningOptions{SecretKey: armoredSecretKeys(t, signer)}),
func(outer *MFFileOuter) {
outer.SigningPubKey = armoredPublicKeys(t, other, signer)
})
_, err := NewManifestFromReader(bytes.NewReader(manifest))
require.ErrorIs(t, err, errKeyCount)
}
// dsaKeyPacket returns a public key packet holding a DSA key, or a public
// subkey packet when isSubkey. Its numbers are not a working key: loading
// must refuse the packet before it uses them.
func dsaKeyPacket(t *testing.T, isSubkey bool) []byte {
t.Helper()
key := packet.NewDSAPublicKey(time.Now(), &dsa.PublicKey{
P: big.NewInt(23), Q: big.NewInt(11), G: big.NewInt(4), Y: big.NewInt(8),
})
key.IsSubkey = isSubkey
var buf bytes.Buffer
require.NoError(t, key.Serialize(&buf))
return buf.Bytes()
}
// TestManifestRefusesDSAKey loads manifests whose embedded public key
// block holds a DSA key: alone, or as a subkey after the key that signed
// the manifest. openpgp.ReadKeyRing checks a key's self-signatures, which
// for a DSA key with very large numbers takes minutes each. Loading must
// refuse each block before that.
func TestManifestRefusesDSAKey(t *testing.T) {
t.Parallel()
signer := newTestKey(t, nil)
manifest := signedTestManifest(t,
&SigningOptions{SecretKey: armoredSecretKeys(t, signer)})
var signerKey bytes.Buffer
require.NoError(t, signer.Serialize(&signerKey))
for name, block := range map[string][]byte{
"DSA key": dsaKeyPacket(t, false),
"DSA subkey": slices.Concat(signerKey.Bytes(), dsaKeyPacket(t, true)),
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
embedded := rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.SigningPubKey = block
})
_, err := NewManifestFromReader(bytes.NewReader(embedded))
require.ErrorIs(t, err, errDSAKey)
})
}
}
// elGamalSecretSubkeyPacket returns a secret subkey packet holding an
// ElGamal key. Its numbers are not a working key: loading must refuse the
// packet before it uses them.
func elGamalSecretSubkeyPacket(t *testing.T) []byte {
t.Helper()
key := packet.NewElGamalPrivateKey(time.Now(), &elgamal.PrivateKey{
P: big.NewInt(23), G: big.NewInt(4), Y: big.NewInt(8), X: big.NewInt(3),
})
key.IsSubkey = true
var buf bytes.Buffer
require.NoError(t, key.Serialize(&buf))
return buf.Bytes()
}
// TestManifestRefusesSecretKey loads manifests whose embedded public key
// block holds a secret key: the key that signed the manifest with its
// secret key, or its public key followed by an ElGamal secret subkey.
// openpgp.ReadKeyRing checks the numbers of every secret key it reads,
// which for an ElGamal key with a very large prime takes minutes. Loading
// must refuse each block before that.
func TestManifestRefusesSecretKey(t *testing.T) {
t.Parallel()
signer := newTestKey(t, nil)
manifest := signedTestManifest(t,
&SigningOptions{SecretKey: armoredSecretKeys(t, signer)})
var signerKey bytes.Buffer
require.NoError(t, signer.Serialize(&signerKey))
for _, block := range [][]byte{
armoredSecretKeys(t, signer),
slices.Concat(signerKey.Bytes(), elGamalSecretSubkeyPacket(t)),
} {
embedded := rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.SigningPubKey = block
})
_, err := NewManifestFromReader(bytes.NewReader(embedded))
require.ErrorIs(t, err, errSecretKey)
}
}
// TestManifestRefusesTwoSignatures loads a manifest whose signature field
// holds its good signature twice, not armored. Loading must refuse it.
func TestManifestRefusesTwoSignatures(t *testing.T) {
t.Parallel()
manifest := rewriteOuter(t, signedTestManifest(t, testSigningOptions(t)),
func(outer *MFFileOuter) {
sig, err := dearmor(outer.GetSignature())
require.NoError(t, err)
outer.Signature = slices.Concat(sig, sig)
})
_, err := NewManifestFromReader(bytes.NewReader(manifest))
require.ErrorIs(t, err, errNotOneSignature)
}
// TestManifestRefusesFieldNotOneArmoredBlock loads manifests whose
// signature or embedded public key block holds its good armored block with
// something else: a second armored block, text after the END line, or many
// END lines before the block. Decoding the block once for each END line
// before it would take time and memory that grow with the square of the
// field's size. Loading must refuse each.
func TestManifestRefusesFieldNotOneArmoredBlock(t *testing.T) {
t.Parallel()
manifest := signedTestManifest(t, testSigningOptions(t))
for name, change := range map[string]func([]byte) []byte{
"second armored block": func(block []byte) []byte {
return joinArmored(block, block)
},
"text after the END line": func(block []byte) []byte {
return slices.Concat(block, []byte("\nmore text\n"))
},
"END lines before the block": func(block []byte) []byte {
return slices.Concat(
[]byte(strings.Repeat(armorEnd+"\n", 1000)), block)
},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
for _, changed := range [][]byte{
rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.Signature = change(outer.GetSignature())
}),
rewriteOuter(t, manifest, func(outer *MFFileOuter) {
outer.SigningPubKey = change(outer.GetSigningPubKey())
}),
} {
_, err := NewManifestFromReader(bytes.NewReader(changed))
require.ErrorIs(t, err, errNotOneArmoredBlock)
}
})
}
}
// TestManifestSignedWithSubkey signs with a key that has a signing subkey,
// which signs in place of the primary key. The manifest must load, with
// the primary key's fingerprint as signer.
func TestManifestSignedWithSubkey(t *testing.T) {
t.Parallel()
key := newTestKey(t, nil)
require.NoError(t, key.AddSigningSubkey(
&packet.Config{Algorithm: packet.PubKeyAlgoEdDSA}))
m, err := NewManifestFromReader(bytes.NewReader(signedTestManifest(t,
&SigningOptions{SecretKey: armoredSecretKeys(t, key)})))
require.NoError(t, err)
assert.Equal(t, fingerprint(key), string(m.pbOuter.GetSigner()))
block, err := armor.Decode(bytes.NewReader(m.pbOuter.GetSignature()))
require.NoError(t, err)
p, err := packet.Read(block.Body)
require.NoError(t, err)
sig, ok := p.(*packet.Signature)
require.True(t, ok)
subkey := key.Subkeys[len(key.Subkeys)-1].PublicKey
assert.Equal(t, subkey.KeyId, *sig.IssuerKeyId,
"the signing subkey made the signature")
}
// TestManifestRefusesSignerOtherThanSigningKey loads a manifest whose
// signer field names a key other than the one that made the signature.
func TestManifestRefusesSignerOtherThanSigningKey(t *testing.T) {
t.Parallel()
manifest := rewriteOuter(t, signedTestManifest(t, testSigningOptions(t)),
func(outer *MFFileOuter) {
outer.Signer = []byte(strings.Repeat("A", len(outer.GetSigner())))
})
_, err := NewManifestFromReader(bytes.NewReader(manifest))
require.ErrorIs(t, err, errSignerNotSigningKey)
}
func TestBuilderWithoutSigning(t *testing.T) {
t.Parallel()
// Create a builder without signing options
b := NewBuilder()
// Add a test file
content := []byte("test file content")
reader := bytes.NewReader(content)
_, err := b.AddFile("test.txt", FileSize(len(content)), ModTime{}, 0, reader, nil)
require.NoError(t, err)
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are empty
manifest, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.NotNil(t, manifest.pbOuter)
assert.Empty(t, manifest.pbOuter.GetSignature(),
"signature should be empty when not signing")
assert.Empty(t, manifest.pbOuter.GetSigner(),
"signer should be empty when not signing")
assert.Empty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be empty when not signing")
}
// TestBuildPassesContextToSigning checks that Build does not sign once the
// context given to it has ended.
func TestBuildPassesContextToSigning(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{SecretKey: []byte("any")})
ctx, cancel := context.WithCancel(context.Background())
cancel()
require.ErrorIs(t, b.Build(ctx, io.Discard), context.Canceled)
}
mfer/scanner.go
+53 -16
View File
@@ -2,6 +2,7 @@ package mfer
import (
"context"
"fmt"
"io"
"io/fs"
"os"
@@ -52,9 +53,12 @@ type ScannerOptions struct {
// IncludeTimestamps includes a createdAt timestamp in the manifest
// (default: omit for determinism).
IncludeTimestamps bool
// IncludePermissions records each file's permission bits, 0777 at
// most, in the manifest (default: record 0000).
IncludePermissions bool
// Fs is the filesystem to use, defaults to OsFs if nil.
Fs afero.Fs
// SigningOptions holds GPG signing options (nil = no signing).
// SigningOptions holds the key to sign with (nil = no signing).
SigningOptions *SigningOptions
// Seed, if set, derives a deterministic UUID from this seed.
Seed string
@@ -69,12 +73,14 @@ type FileEntry struct {
Size FileSize // File size in bytes
Mtime ModTime // Last modification time
Ctime time.Time // Creation time (platform-dependent)
Mode fs.FileMode // Permission bits (Perm() of the file's mode)
}
// Scanner accumulates files and generates manifests from them.
type Scanner struct {
mu sync.RWMutex
files []*FileEntry
paths map[RelFilePath]AbsFilePath // the file at each path in files
totalBytes FileSize // cached sum of all file sizes
options *ScannerOptions
fs afero.Fs
@@ -99,6 +105,7 @@ func NewScannerWithOptions(opts *ScannerOptions) *Scanner {
s := &Scanner{
files: make([]*FileEntry, 0),
paths: make(map[RelFilePath]AbsFilePath),
options: opts,
fs: fs,
}
@@ -132,28 +139,20 @@ func (s *Scanner) EnumerateFile(filePath string) error {
return s.enumerateFileWithInfo(filepath.Base(abs), basePath, info, nil)
}
// EnumeratePath walks a directory path and adds all files to the scanner.
// EnumeratePath adds inputPath, a directory or a file, to the scanner as
// EnumeratePaths adds each of its paths.
// If progress is non-nil, status updates are sent as files are discovered.
// The progress channel is closed when the method returns.
func (s *Scanner) EnumeratePath(
inputPath string,
progress chan<- EnumerateStatus,
) error {
if progress != nil {
defer close(progress)
return s.EnumeratePaths(progress, inputPath)
}
abs, err := filepath.Abs(inputPath)
if err != nil {
return err
}
afs := afero.NewReadOnlyFs(afero.NewBasePathFs(s.fs, abs))
return s.enumerateFS(afs, abs, progress)
}
// EnumeratePaths walks multiple directory paths and adds all files to the scanner.
// EnumeratePaths adds to the scanner the files under each directory path,
// listed by their paths under it, and each file path, listed by its name
// as EnumerateFile lists it.
// If progress is non-nil, status updates are sent as files are discovered.
// The progress channel is closed when the method returns.
func (s *Scanner) EnumeratePaths(
@@ -170,9 +169,27 @@ func (s *Scanner) EnumeratePaths(
return err
}
afs := afero.NewReadOnlyFs(afero.NewBasePathFs(s.fs, abs))
info, err := s.fs.Stat(abs)
if err != nil {
return err
}
if info.IsDir() {
// The walk does not follow a symlink at its top, so a directory
// named through one is resolved first. If that fails, the
// directory is walked as named and the walk reports the problem.
resolved, evalErr := filepath.EvalSymlinks(abs)
if evalErr == nil {
abs = resolved
}
afs := afero.NewReadOnlyFs(afero.NewBasePathFs(s.fs, abs))
err = s.enumerateFS(afs, abs, progress)
} else {
err = s.enumerateFileWithInfo(
filepath.Base(abs), filepath.Dir(abs), info, progress)
}
if err != nil {
return err
}
@@ -353,11 +370,18 @@ func (s *Scanner) scanFile(
}(scannedBytes, scannedFiles)
}
// A mode of 0 records 0000, which means none was recorded.
var mode fs.FileMode
if s.options.IncludePermissions {
mode = entry.Mode
}
// Add to manifest with progress channel
bytesRead, err := builder.AddFile(
entry.Path,
entry.Size,
entry.Mtime,
mode,
f,
fileProgress,
)
@@ -461,11 +485,24 @@ func (s *Scanner) enumerateFileWithInfo(
AbsPath: AbsFilePath(absPath),
Size: FileSize(info.Size()),
Mtime: ModTime(info.ModTime()),
Mode: info.Mode().Perm(),
// Note: Ctime not available from fs.FileInfo on all platforms
// Will need platform-specific code to extract it
}
s.mu.Lock()
// Each path is relative to the input path it was found under, so files
// under two input paths can share one.
first, ok := s.paths[entry.Path]
if ok {
s.mu.Unlock()
return fmt.Errorf("%w %q: %s and %s",
errDuplicatePath, entry.Path, first, entry.AbsPath)
}
s.paths[entry.Path] = entry.AbsPath
s.files = append(s.files, entry)
s.totalBytes += entry.Size
filesFound := FileCount(len(s.files))
mfer/scanner_test.go
+84
View File
@@ -4,6 +4,7 @@ package mfer
import (
"bytes"
"context"
"os"
"testing"
"time"
@@ -117,6 +118,27 @@ func TestScannerEnumeratePathWithProgress(t *testing.T) {
assert.Equal(t, FileSize(6), final.BytesFound)
}
// TestScannerEnumeratePathFile gives EnumeratePath a file: it is listed
// by its name, as EnumerateFile lists it, and the manifest can be built.
func TestScannerEnumeratePathFile(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/dir", 0o755))
require.NoError(t, afero.WriteFile(fs, "/dir/one.txt", []byte("1"), 0o644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
require.NoError(t, s.EnumeratePath("/dir/one.txt", nil))
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
require.Len(t, m.Files(), 1)
assert.Equal(t, "one.txt", m.Files()[0].GetPath())
}
func TestScannerEnumeratePaths(t *testing.T) {
t.Parallel()
@@ -133,6 +155,28 @@ func TestScannerEnumeratePaths(t *testing.T) {
assert.Equal(t, FileCount(2), s.FileCount())
}
// TestScannerEnumeratePathsFile gives EnumeratePaths a directory and a
// file: the file is listed by its name, as EnumerateFile lists it.
func TestScannerEnumeratePathsFile(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll("/dir/sub", 0o755))
require.NoError(t, fs.MkdirAll("/other", 0o755))
require.NoError(t, afero.WriteFile(fs, "/dir/sub/one.txt", []byte("1"), 0o644))
require.NoError(t, afero.WriteFile(fs, "/other/two.txt", []byte("2"), 0o644))
s := NewScannerWithOptions(&ScannerOptions{Fs: fs})
require.NoError(t, s.EnumeratePaths(nil, "/dir", "/other/two.txt"))
paths := make([]RelFilePath, 0, s.FileCount())
for _, f := range s.Files() {
paths = append(paths, f.Path)
}
assert.Equal(t, []RelFilePath{"sub/one.txt", "two.txt"}, paths)
}
func TestScannerExcludeDotfiles(t *testing.T) {
t.Parallel()
@@ -350,6 +394,46 @@ func TestScannerFileEntryFields(t *testing.T) {
assert.WithinDuration(t, now, time.Time(entry.Mtime), 2*time.Second)
}
// A manifest records every mode as 0000 unless the creator asks for
// permissions; then it records each file's permission bits and never the
// setuid, setgid or sticky bits.
func TestScannerRecordsModeOnlyWhenAsked(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/"+testFile1, []byte("a"), 0o640))
require.NoError(t, afero.WriteFile(fs, "/run.sh", []byte("b"), 0o755))
require.NoError(t, afero.WriteFile(fs, "/su", []byte("c"), 0o755))
require.NoError(t, fs.Chmod("/su", 0o755|os.ModeSetuid|os.ModeSetgid|os.ModeSticky))
for _, tc := range []struct {
includePermissions bool
want map[string]uint32
}{
{false, map[string]uint32{testFile1: 0, "run.sh": 0, "su": 0}},
{true, map[string]uint32{testFile1: 0o640, "run.sh": 0o755, "su": 0o755}},
} {
s := NewScannerWithOptions(&ScannerOptions{
Fs: fs,
IncludePermissions: tc.includePermissions,
})
require.NoError(t, s.EnumerateFS(fs, "/", nil))
var buf bytes.Buffer
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
got := map[string]uint32{}
for _, f := range m.Files() {
got[f.GetPath()] = f.GetMode()
}
assert.Equal(t, tc.want, got, "IncludePermissions: %v", tc.includePermissions)
}
}
func TestScannerLargeFileEnumeration(t *testing.T) {
t.Parallel()
mfer/serialize.go
+33 -25
View File
@@ -7,9 +7,11 @@ import (
"errors"
"fmt"
"math"
"strings"
"time"
"uuid"
"github.com/google/uuid"
"github.com/ProtonMail/go-crypto/openpgp"
"github.com/klauspost/compress/zstd"
"google.golang.org/protobuf/proto"
)
@@ -20,11 +22,9 @@ const MAGIC string = "ZNAVSRFG"
var (
// errInnerNotSet is returned by generate when the inner manifest is
// missing.
errInnerNotSet = errors.New("internal error: pbInner not set")
errInnerNotSet = errors.New("inner message not set")
// errInternal is returned by generateOuter for the same condition.
// The two messages differ, and both are load-bearing for callers that
// match on text, so they are kept distinct.
errInternal = errors.New("internal error")
errInternal = errors.New("inner message not set")
)
// nanosecondsInt32 converts t's nanosecond component to int32.
@@ -65,14 +65,14 @@ func (m *manifest) generate(ctx context.Context) error {
dat, err := proto.MarshalOptions{Deterministic: true}.Marshal(m.pbOuter)
if err != nil {
return fmt.Errorf("serialize: marshal outer: %w", err)
return fmt.Errorf("marshal outer message: %w", err)
}
m.output = bytes.NewBufferString(MAGIC)
_, err = m.output.Write(dat)
if err != nil {
return fmt.Errorf("serialize: write output: %w", err)
return fmt.Errorf("write outer message: %w", err)
}
return nil
@@ -88,14 +88,14 @@ func (m *manifest) generateOuter(ctx context.Context) error {
if len(m.fixedUUID) == uuidLength {
copy(manifestUUID[:], m.fixedUUID)
} else {
manifestUUID = uuid.New()
manifestUUID = uuid.NewV4()
}
m.pbInner.Uuid = manifestUUID[:]
innerData, err := proto.MarshalOptions{Deterministic: true}.Marshal(m.pbInner)
if err != nil {
return fmt.Errorf("serialize: marshal inner: %w", err)
return fmt.Errorf("marshal inner message: %w", err)
}
// Compress the inner data
@@ -103,12 +103,12 @@ func (m *manifest) generateOuter(ctx context.Context) error {
zw, err := zstd.NewWriter(idc, zstd.WithEncoderLevel(zstd.SpeedBestCompression))
if err != nil {
return fmt.Errorf("serialize: create compressor: %w", err)
return fmt.Errorf("create compressor: %w", err)
}
_, err = zw.Write(innerData)
if err != nil {
return fmt.Errorf("serialize: compress: %w", err)
return fmt.Errorf("compress inner message: %w", err)
}
_ = zw.Close()
@@ -120,7 +120,7 @@ func (m *manifest) generateOuter(ctx context.Context) error {
_, err = h.Write(compressedData)
if err != nil {
return fmt.Errorf("serialize: hash write: %w", err)
return fmt.Errorf("hash inner message: %w", err)
}
sha256Hash := h.Sum(nil)
@@ -135,40 +135,48 @@ func (m *manifest) generateOuter(ctx context.Context) error {
}
// Sign the manifest if signing options are provided
if m.signingOptions != nil && m.signingOptions.KeyID != "" {
if m.signingOptions != nil {
return m.signOuter(ctx)
}
return nil
}
// signOuter signs the outer message with the configured GPG key and
// embeds the signature, signer fingerprint, and public key.
// signOuter signs the outer message with the secret key in the signing
// options and embeds the signature, the key's fingerprint and its public
// key.
func (m *manifest) signOuter(ctx context.Context) error {
// Unlocking a protected key can take a while; do not start once ctx
// has ended.
err := ctx.Err()
if err != nil {
return err
}
sigString, err := m.signatureString()
if err != nil {
return fmt.Errorf("failed to generate signature string: %w", err)
return fmt.Errorf("build signature string: %w", err)
}
sig, err := gpgSign(ctx, []byte(sigString), m.signingOptions.KeyID)
key, err := readSigningKey(m.signingOptions)
if err != nil {
return fmt.Errorf("failed to sign manifest: %w", err)
return err
}
m.pbOuter.Signature = sig
var sig bytes.Buffer
fingerprint, err := gpgGetKeyFingerprint(ctx, m.signingOptions.KeyID)
err = openpgp.ArmoredDetachSign(&sig, key, strings.NewReader(sigString), nil)
if err != nil {
return fmt.Errorf("failed to get key fingerprint: %w", err)
return fmt.Errorf("sign manifest: %w", err)
}
m.pbOuter.Signer = fingerprint
pubKey, err := gpgExportPublicKey(ctx, m.signingOptions.KeyID)
pubKey, err := armoredPublicKey(key)
if err != nil {
return fmt.Errorf("failed to export public key: %w", err)
return err
}
m.pbOuter.Signature = sig.Bytes()
m.pbOuter.Signer = []byte(fingerprint(key))
m.pbOuter.SigningPubKey = pubKey
return nil
package.json
-1
View File
@@ -1,6 +1,5 @@
{
"name": "mfer",
"version": "0.1.0",
"private": true,
"description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.",
"license": "WTFPL",
script/bootstrap
+116 -5
View File
@@ -13,11 +13,24 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06. Never "latest" or "lts"; exact versions.
NODE_VERSION="22.17.0"
# The node version is in .nvmrc, where script/prettier reads it too.
NODE_VERSION="$(cat "$ROOT/.nvmrc")"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
# protoc v33.4, 2026-10-04, for script/generate. The sha256 of each
# platform's release archive is in ensure_protoc.
PROTOC_VERSION="33.4"
# gofumpt v0.12.0 for script/gofumpt, 2026-10-04, and protoc-gen-go
# v1.36.12 for script/generate, 2026-10-06: each is installed into bin/
# with `go install`, pinned to the commit its release tag names. Those two
# scripts refuse any other version, so a new pin is changed there too.
# protoc-gen-go stays at the google.golang.org/protobuf version in go.mod.
GOFUMPT_VERSION="v0.12.0"
GOFUMPT_COMMIT="3e07e7e70ac93761d8e79ca0083a19e3d59f753d"
PROTOC_GEN_GO_VERSION="v1.36.12"
PROTOC_GEN_GO_COMMIT="cdd4c5f7406e82462949c7a65defa9f3029c162d"
PKGMGR=""
SUDO=""
@@ -74,9 +87,11 @@ verify_sha256() {
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
# nvm is a bash script; run a command in a bash with nvm loaded.
# --no-use: otherwise loading nvm here switches to the version .nvmrc
# names, and fails silently while that version is not installed yet.
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
bash -c ". \"\$HOME/.nvm/nvm.sh\" --no-use && $*"
}
ensure_nvm() {
@@ -122,6 +137,97 @@ install_js_deps() {
fi
}
# Unpack protoc's release archive for this platform into bin/protoc, after
# checking the archive's sha256, unless bin/protoc already holds the pinned
# version. script/generate runs bin/protoc/bin/protoc, so that is the
# binary checked again after unpacking.
ensure_protoc() {
dir="$ROOT/bin/protoc"
if [ "$("$dir/bin/protoc" --version 2>/dev/null)" = \
"libprotoc $PROTOC_VERSION" ]; then
echo "protoc $PROTOC_VERSION"
return 0
fi
case "$(uname -s) $(uname -m)" in
"Linux x86_64")
platform="linux-x86_64"
sha256="c0040ea9aef08fdeb2c74ca609b18d5fdbfc44ea0042fcfbfb38860d35f7dd66"
;;
"Linux aarch64" | "Linux arm64")
platform="linux-aarch_64"
sha256="15aa988f4a6090636525ec236a8e4b3aab41eef402751bd5bb2df6afd9b7b5a5"
;;
"Darwin x86_64")
platform="osx-x86_64"
sha256="a49bec10d039e902d3b43e49938c42526f90011467609864fa6386ac4014da58"
;;
"Darwin arm64")
platform="osx-aarch_64"
sha256="726297dcfed58592fd35620a5a6246ae020c39e88f3fd4cb1827df7bcf3dfcf1"
;;
*)
echo "bootstrap: no protoc archive pinned for $(uname -s) $(uname -m)" >&2
exit 1
;;
esac
if missing curl; then pkg_install curl curl curl curl; fi
if missing unzip; then pkg_install unzip unzip unzip unzip; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/protoc.zip" \
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${platform}.zip"
verify_sha256 "$tmp/protoc.zip" "$sha256"
rm -rf "$dir"
unzip -q "$tmp/protoc.zip" -d "$dir"
rm -rf "$tmp"
actual="$("$dir/bin/protoc" --version 2>/dev/null || true)"
if [ "$actual" != "libprotoc $PROTOC_VERSION" ]; then
echo "bootstrap: $dir/bin/protoc reports '$actual'," \
"not libprotoc $PROTOC_VERSION" >&2
exit 1
fi
echo "protoc $PROTOC_VERSION"
}
# Install gofumpt into bin/ unless bin/gofumpt already reports the pinned
# version. script/gofumpt runs bin/gofumpt, so that is the binary checked
# again after installing. Its --version prints the version, then the Go
# version it was built with. The old file is removed first because
# `go install` refuses to replace a file that is not a Go binary.
ensure_gofumpt() {
tool="$ROOT/bin/gofumpt"
if [ "$("$tool" --version 2>/dev/null | cut -d' ' -f1)" != \
"$GOFUMPT_VERSION" ]; then
rm -f "$tool"
GOBIN="$ROOT/bin" go install "mvdan.cc/gofumpt@$GOFUMPT_COMMIT"
fi
actual="$("$tool" --version 2>/dev/null | cut -d' ' -f1)"
if [ "$actual" != "$GOFUMPT_VERSION" ]; then
echo "bootstrap: $tool reports '$actual', not $GOFUMPT_VERSION" >&2
exit 1
fi
echo "gofumpt $GOFUMPT_VERSION"
}
# Install protoc-gen-go into bin/ unless bin/protoc-gen-go already reports
# the pinned version, as ensure_gofumpt does. script/generate runs
# bin/protoc-gen-go, so that is the binary checked again after installing.
ensure_protoc_gen_go() {
tool="$ROOT/bin/protoc-gen-go"
if [ "$("$tool" --version 2>/dev/null)" != \
"protoc-gen-go $PROTOC_GEN_GO_VERSION" ]; then
rm -f "$tool"
GOBIN="$ROOT/bin" go install \
"google.golang.org/protobuf/cmd/protoc-gen-go@$PROTOC_GEN_GO_COMMIT"
fi
actual="$("$tool" --version 2>/dev/null || true)"
if [ "$actual" != "protoc-gen-go $PROTOC_GEN_GO_VERSION" ]; then
echo "bootstrap: $tool reports '$actual'," \
"not protoc-gen-go $PROTOC_GEN_GO_VERSION" >&2
exit 1
fi
echo "protoc-gen-go $PROTOC_GEN_GO_VERSION"
}
main() {
cd "$ROOT"
@@ -132,8 +238,10 @@ main() {
# ---- JS / docs repos ----
# This is a Go repo, but node and yarn are required anyway: prettier
# formats the Markdown and JSON, and script/fmt-check verifies it.
# The version is pinned by package.json/yarn.lock, whose integrity
# hashes --frozen-lockfile enforces.
# The version is pinned by package.json/yarn.lock: yarn checks every
# package it fetches against its yarn.lock integrity hash, and
# --frozen-lockfile fails instead of rewriting a yarn.lock that no
# longer matches package.json.
ensure_node
ensure_yarn
install_js_deps
@@ -142,6 +250,9 @@ main() {
if missing go; then pkg_install go golang go go; fi
# No golangci-lint: script/lint runs it in Docker only.
go mod download
ensure_gofumpt
ensure_protoc
ensure_protoc_gen_go
# ---- Python repos ----
# if missing python3; then pkg_install python3 python3 python3 python3; fi
script/check
+3 -2
View File
@@ -1,7 +1,8 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
# Generic: usually needs no adaptation.
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
script/cibuild
+8 -4
View File
@@ -1,8 +1,10 @@
#!/bin/sh
# script/cibuild: run the CI build; the Gitea workflow runs this on push.
# It builds the image with the same command as script/docker. --no-cache
# because the checks the final stage depends on are RUN steps, and a
# cached one is a check that did not run.
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -10,6 +12,8 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
script/generate
+25 -21
View File
@@ -4,26 +4,19 @@
# regenerates mf.pb.go: it is committed, so building and checking need no
# protoc. A test fails while mf.proto no longer matches the recorded hash.
#
# Needs exactly the protoc and protoc-gen-go versions named in the header of
# the committed mf.pb.go (README.md says how to install them). Another
# version writes a different mf.pb.go, so the script refuses to run.
# Runs the protoc that script/bootstrap unpacks into bin/protoc, and the
# protoc-gen-go it installs into bin/. Another version of either writes a
# different mf.pb.go.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# protoc 33.4 names itself v6.33.4 in the mf.pb.go header.
# The versions script/bootstrap installs. protoc 33.4 names itself v6.33.4
# in the mf.pb.go header.
PROTOC_VERSION="33.4"
PROTOC_GEN_GO_VERSION="v1.36.11"
# require_version <command> <its exact --version output>
require_version() {
actual="$("$1" --version 2>/dev/null || true)"
if [ "$actual" != "$2" ]; then
echo "generate: needs $2 on PATH, found: ${actual:-none}" >&2
echo " README.md says how to install it." >&2
exit 1
fi
}
PROTOC="$ROOT/bin/protoc/bin/protoc"
PROTOC_GEN_GO_VERSION="v1.36.12"
PROTOC_GEN_GO="$ROOT/bin/protoc-gen-go"
# sha256 <file>: print "<hash> <file>", with sha256sum, or with shasum
# where there is no sha256sum.
@@ -39,16 +32,27 @@ sha256() {
}
main() {
# A bin/protoc or bin/protoc-gen-go left from before its pin moved
# fails here, until script/bootstrap replaces it.
actual="$("$PROTOC" --version 2>/dev/null || true)"
if [ "$actual" != "libprotoc $PROTOC_VERSION" ]; then
echo "generate: needs protoc $PROTOC_VERSION in bin/protoc," \
"found: ${actual:-none}; run script/bootstrap" >&2
exit 1
fi
actual="$("$PROTOC_GEN_GO" --version 2>/dev/null || true)"
if [ "$actual" != "protoc-gen-go $PROTOC_GEN_GO_VERSION" ]; then
echo "generate: needs protoc-gen-go $PROTOC_GEN_GO_VERSION in" \
"bin/protoc-gen-go, found: ${actual:-none}; run script/bootstrap" >&2
exit 1
fi
cd "$ROOT/mfer"
# `go install` puts protoc-gen-go in $(go env GOPATH)/bin, which is
# often not on PATH.
PATH="$PATH:$(go env GOPATH)/bin"
require_version protoc "libprotoc $PROTOC_VERSION"
require_version protoc-gen-go "protoc-gen-go $PROTOC_GEN_GO_VERSION"
# Hashed before regenerating, so a missing hash tool stops the script
# before it changes anything. Regenerating leaves mf.proto as it is.
proto_hash="$(sha256 mf.proto)"
go generate .
"$PROTOC" --plugin=protoc-gen-go="$PROTOC_GEN_GO" \
--go_out=paths=source_relative:. ./mf.proto
echo "$proto_hash" >mf.proto.sha256
}
script/gofumpt
+18 -12
View File
@@ -3,17 +3,16 @@
#
# Takes exactly one mode argument, --write or --check, and runs the same
# gofumpt version over the same files in both modes. script/fmt and
# script/fmt-check both go through here, and so does the Docker lint
# stage, so what gets formatted and what gets verified cannot drift
# apart.
# script/fmt-check both go through here, so what gets formatted and what
# gets verified cannot drift apart.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# gofumpt v0.12.0, 2026-10-04. `go run` fetches and builds exactly this
# version, so neither a developer machine nor the lint image needs
# gofumpt installed.
GOFUMPT="mvdan.cc/gofumpt@v0.12.0"
# The gofumpt script/bootstrap installs into bin/. Must match the pin
# there.
GOFUMPT_VERSION="v0.12.0"
GOFUMPT="$ROOT/bin/gofumpt"
usage() {
echo "usage: script/gofumpt --write|--check" >&2
@@ -22,15 +21,22 @@ usage() {
main() {
[ "$#" -eq 1 ] || usage
cd "$ROOT"
# Every Go file in the repo. gofumpt holds generated files, such as
# mfer/mf.pb.go, to gofmt's rules only.
# A bin/gofumpt left from before the pin moved formats differently, so
# it fails here until script/bootstrap replaces it.
actual="$("$GOFUMPT" --version 2>/dev/null | cut -d' ' -f1)"
if [ "$actual" != "$GOFUMPT_VERSION" ]; then
echo "gofumpt: needs $GOFUMPT_VERSION in bin/gofumpt," \
"found: ${actual:-none}; run script/bootstrap" >&2
exit 1
fi
# Every Go file in the repo, from $ROOT down. gofumpt holds generated
# files, such as mfer/mf.pb.go, to gofmt's rules only.
case "$1" in
--write) go run "$GOFUMPT" -l -w . ;;
--write) "$GOFUMPT" -l -w "$ROOT" ;;
--check)
# Own line: a failing command inside `[ -n "$(...)" ]` does
# not trip `set -e`, so a gofumpt that never ran would pass.
unformatted="$(go run "$GOFUMPT" -l .)"
unformatted="$("$GOFUMPT" -l "$ROOT")"
if [ -n "$unformatted" ]; then
echo "gofumpt: files need formatting (run make fmt):" >&2
echo "$unformatted" >&2
script/install-precommit
-1
View File
@@ -1,7 +1,6 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
# Generic: needs no adaptation.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
script/lint
+12 -9
View File
@@ -1,8 +1,13 @@
#!/bin/sh
# script/lint: run golangci-lint, in Docker only. Builds the lint stage of
# the Dockerfile, whose build runs the linter, so a successful build is a
# clean lint. --no-cache because a cached build runs no linter. The image
# is removed afterwards, whatever the outcome.
# script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -10,11 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Tagged per run, so concurrent runs never remove each other's image.
image="$("$SCRIPT_DIR/projectname")-lint:$$"
# A failed build leaves no image, so there is nothing to remove then.
trap 'docker image rm "$image" >/dev/null 2>&1 || true' EXIT INT TERM
docker build --no-cache --target lint -t "$image" .
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
}
main "$@"
script/prettier
+25 -26
View File
@@ -18,24 +18,9 @@ usage() {
exit 2
}
# Prefer the version pinned by package.json/yarn.lock so that CI and
# developer machines format identically. Fall back to a prettier on PATH,
# but say so, because a different version formats differently.
find_prettier() {
if [ -x "$ROOT/node_modules/.bin/prettier" ]; then
printf '%s\n' "$ROOT/node_modules/.bin/prettier"
return 0
fi
if command -v prettier >/dev/null 2>&1; then
echo "prettier: node_modules/.bin/prettier is absent; using the" \
"prettier on PATH, which may be a different version than the" \
"one pinned in package.json. Run script/bootstrap to install" \
"the pinned version." >&2
command -v prettier
return 0
fi
return 1
}
# Only the prettier yarn installed from yarn.lock, never one on PATH: a
# different version formats differently.
PRETTIER="$ROOT/node_modules/.bin/prettier"
main() {
[ "$#" -eq 1 ] || usage
@@ -46,24 +31,38 @@ main() {
cd "$ROOT"
if ! prettier_bin="$(find_prettier)"; then
echo "prettier: not found." >&2
echo " Install it with: script/bootstrap" >&2
echo " (installs the version pinned in package.json/yarn.lock)" >&2
# Where there is no node on PATH, script/bootstrap installs the version
# .nvmrc names through nvm, which keeps it in this directory.
if ! command -v node >/dev/null 2>&1; then
PATH="$HOME/.nvm/versions/node/v$(cat .nvmrc)/bin:$PATH"
fi
if ! command -v node >/dev/null 2>&1; then
echo "prettier: node is missing; run script/bootstrap" >&2
exit 1
fi
# node_modules keeps the old prettier after package.json moves to a new
# one, until script/bootstrap runs again, so compare the two.
if ! installed="$("$PRETTIER" --version 2>/dev/null)"; then
echo "prettier: not installed; run script/bootstrap" >&2
exit 1
fi
pinned="$(node -p 'require("./package.json").devDependencies.prettier')"
if [ "$installed" != "$pinned" ]; then
echo "prettier: package.json pins $pinned but $installed is" \
"installed; run script/bootstrap" >&2
exit 1
fi
# Markdown and JSON, repo-wide rather than root-only, so files in
# subdirectories (docs/, once it exists) are covered too. Exclusions
# live in .prettierignore; REPO_POLICIES.md is excluded there because
# it is a verbatim copy of an upstream document.
# live in .prettierignore.
#
# --no-error-on-unmatched-pattern is deliberately NOT used: both
# patterns always match at least one tracked file (README.md,
# package.json), so an empty match means the glob broke, and prettier
# erroring out is exactly what we want rather than a vacuous pass.
"$prettier_bin" "$mode" "**/*.md"
"$prettier_bin" "$mode" "**/*.json"
"$PRETTIER" "$mode" "**/*.md" "**/*.json"
}
main "$@"
script/setup
+1 -2
View File
@@ -1,7 +1,6 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies (script/bootstrap) and the git pre-commit hook.
# Add any repo-specific initialization (db init, .env template) here.
# installs dependencies and the git pre-commit hook.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
script/test
+10 -8
View File
@@ -1,17 +1,19 @@
#!/bin/sh
# script/test: run the test suite.
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
go test -timeout 30s -race -cover ./... ||
{
echo "--- Rerunning with -v for details ---"
go test -timeout 30s -race -v ./...
exit 1
}
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
}
main "$@"
script/vulncheck
Executable
+22
View File
@@ -0,0 +1,22 @@
#!/bin/sh
# script/vulncheck: report known vulnerabilities in the code mfer calls,
# with govulncheck, which reads the Go vulnerability database online.
# It runs as the vulncheck stage of the Dockerfile, on the same Go as the
# test phase, and this builds that stage alone, on the same terms as
# script/lint and script/test.
#
# script/check does not run it: the gate's result depends on this tree
# alone, and this one changes whenever a new advisory is published.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
--target vulncheck \
-t "$("$SCRIPT_DIR/projectname")-vulncheck" .
}
main "$@"