Author SHA1 Message Date
sneak 19a57e30b3 Add the required README sections (closes #75)
check / check (push) Waiting to run
The Description first line now names the project, purpose, category,
WTFPL license, and author. A new Getting Started section gives a
copy-pasteable build-from-source block and gen/check/fetch usage. Problem
Statement and Proposed Solution move under a new Rationale heading, the
prose kept. A new Design section documents the package layout: the mfer/
library with its committed protobuf code, the internal/cli commands,
internal/log and internal/bork, and the cmd/mfer entrypoint. Authors is
renamed Author with the canonical link.

Getting Started uses go build because the make target that builds the
binary runs protoc, which script/bootstrap does not install.

Model: opus-4-8 (implementation); opus-5-5 (rebase)
2026-10-04 01:52:26 +00:00
15 changed files with 139 additions and 213 deletions
+65 -11
View File
@@ -1,12 +1,12 @@
# mfer
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
thin wrapper command-line utility written in [Go](https://golang.org) and first
published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
It specifies and generates `.mf` manifest files over a directory tree of files
to encapsulate metadata about them (such as cryptographic checksums or
signatures over same) to aid in archiving, downloading, and streaming, or
mirroring. The manifest files' data is serialized with Google's
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
(public domain) [Go](https://golang.org) library and command-line tool by
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
over a directory tree to encapsulate metadata about the files — such as
cryptographic checksums and signatures over same — to aid in archiving,
downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
@@ -21,6 +21,36 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
as a de-facto standard and be incorporated into other software. A compatible
javascript library is planned.
# Getting Started
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
is committed, so no `protoc` toolchain is required:
```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
go build -o bin/mfer ./cmd/mfer
```
Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:
```sh
# Write .index.mf, a manifest of the files under the current directory.
bin/mfer gen .
# Verify the files on disk against the manifest. Exits nonzero if any file
# is missing or corrupted.
bin/mfer check .index.mf
# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/
```
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
for a single command's options.
# Build Status
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
@@ -82,7 +112,9 @@ Any changes submitted to this project must also be
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions.
# Problem Statement
# Rationale
## The problem
Given a plain URL, there is no standard way to safely and programmatically
download everything "under" that URL path. `wget -r` can traverse directory
@@ -106,7 +138,7 @@ Real issues I face:
- when I download a large file via HTTP, I have no way of knowing if the file
content is what it's supposed to be
# Proposed Solution
## The solution
A standard, a manifest file format, and a tool for generating same.
@@ -138,6 +170,28 @@ The manifest file would do several important things:
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
top-level infohash for the whole manifest?
# Design
The repository is split into a reusable library and a thin command-line wrapper
around it.
- `mfer/` is the reusable library and the heart of the project: it defines the
manifest format and implements building, scanning, checking, serialization,
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
with `go get` and needs no `protoc` toolchain.
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
`check`, `freshen`, `export`, `list`, `fetch`, and `version` — that wire the
library to the command-line interface.
- `internal/log/` provides the logging used by the commands and the library.
- `internal/bork/` defines the errors the library returns when reading a
manifest that is truncated or lacks the magic bytes.
- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
into the `mfer` binary.
Everything under `internal/` is private to this repository; only the `mfer/`
package is intended for import by other software.
# Design Goals
- Replace SHASUMS/SHASUMS.asc files
@@ -271,9 +325,9 @@ Open work, open design questions included, is tracked in this repo's issues:
- Issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
# Authors
# Author
- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
- [@sneak](https://sneak.berlin)
# License
+3 -6
View File
@@ -2,7 +2,6 @@
package cli
import (
"context"
"encoding/hex"
"errors"
"fmt"
@@ -127,9 +126,7 @@ func (mfa *CLIApp) fetchManifestToTemp(url string) (string, error) {
// verifyRequiredSigner enforces the --require-signature fingerprint
// against the manifest's embedded signing key.
func verifyRequiredSigner(
ctx context.Context, chk *mfer.Checker, requiredSigner string,
) error {
func verifyRequiredSigner(chk *mfer.Checker, requiredSigner string) error {
// Validate fingerprint format: must be exactly 40 hex characters
if len(requiredSigner) != fingerprintHexLen {
return fmt.Errorf("%w, got %d", errInvalidFingerprint, len(requiredSigner))
@@ -148,7 +145,7 @@ func verifyRequiredSigner(
// Extract fingerprint from the embedded public key (not from the
// signer field). This validates the key is importable and gets its
// actual fingerprint.
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(ctx)
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
if err != nil {
return fmt.Errorf(
"failed to extract fingerprint from embedded signing key: %w", err)
@@ -306,7 +303,7 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
// Check signature requirement
requiredSigner := ctx.String("require-signature")
if requiredSigner != "" {
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
err = verifyRequiredSigner(chk, requiredSigner)
if err != nil {
return err
}
+5 -7
View File
@@ -83,8 +83,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
t.Run("invalid fingerprint length", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), "12345678")
err := verifyRequiredSigner(unsignedChecker(t), "12345678")
require.ErrorIs(t, err, errInvalidFingerprint)
assert.EqualError(t, err,
"invalid fingerprint: must be exactly 40 hex characters, got 8")
@@ -93,8 +92,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
t.Run("manifest not signed", func(t *testing.T) {
t.Parallel()
err := verifyRequiredSigner(context.Background(),
unsignedChecker(t), msgFpA)
err := verifyRequiredSigner(unsignedChecker(t), msgFpA)
require.ErrorIs(t, err, errManifestNotSigned)
assert.EqualError(t, err,
"manifest is not signed, but signature from "+msgFpA+" is required")
@@ -111,10 +109,10 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
func TestSignerMismatchMessage(t *testing.T) {
chk := signedChecker(t)
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
require.NoError(t, err)
err = verifyRequiredSigner(context.Background(), chk, msgFpB)
err = verifyRequiredSigner(chk, msgFpB)
require.ErrorIs(t, err, errSignerMismatch)
assert.EqualError(t, err,
"embedded signing key fingerprint "+embeddedFP+
@@ -162,7 +160,7 @@ func signedChecker(t *testing.T) *mfer.Checker {
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
+3 -4
View File
@@ -1,7 +1,6 @@
package cli
import (
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -305,7 +304,7 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
// writeFreshenedManifest writes the manifest atomically (write to a
// temp file, then rename over the target).
func writeFreshenedManifest(
ctx context.Context, afs afero.Fs, builder *mfer.Builder, manifestPath string,
afs afero.Fs, builder *mfer.Builder, manifestPath string,
) error {
tmpPath := manifestPath + ".tmp"
@@ -314,7 +313,7 @@ func writeFreshenedManifest(
return fmt.Errorf("failed to create temp file: %w", err)
}
err = builder.Build(ctx, outFile)
err = builder.Build(outFile)
_ = outFile.Close()
if err != nil {
@@ -531,7 +530,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
}
// Write updated manifest atomically (write to temp, then rename)
err = writeFreshenedManifest(ctx.Context, mfa.Fs, hasher.builder, manifestPath)
err = writeFreshenedManifest(mfa.Fs, hasher.builder, manifestPath)
if err != nil {
return err
}
+4 -6
View File
@@ -3,7 +3,6 @@
package mfer
import (
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -282,9 +281,8 @@ func (b *Builder) SetSigningOptions(opts *SigningOptions) {
b.signingOptions = opts
}
// Build finalizes the manifest and writes it to the writer. ctx bounds the
// gpg runs that sign the manifest when signing options are set.
func (b *Builder) Build(ctx context.Context, w io.Writer) error {
// Build finalizes the manifest and writes it to the writer.
func (b *Builder) Build(w io.Writer) error {
b.mu.Lock()
defer b.mu.Unlock()
@@ -310,13 +308,13 @@ func (b *Builder) Build(ctx context.Context, w io.Writer) error {
}
// Generate outer wrapper
err := m.generateOuter(ctx)
err := m.generateOuter()
if err != nil {
return fmt.Errorf("build: generate outer: %w", err)
}
// Generate final output
err = m.generate(ctx)
err = m.generate()
if err != nil {
return fmt.Errorf("build: generate: %w", err)
}
+8 -9
View File
@@ -3,7 +3,6 @@ package mfer
import (
"bytes"
"context"
"strings"
"testing"
"time"
@@ -114,7 +113,7 @@ func TestBuilderBuild(t *testing.T) {
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
err = b.Build(&buf)
require.NoError(t, err)
// Should have magic bytes
@@ -178,7 +177,7 @@ func TestBuilderDeterministicOutput(t *testing.T) {
var buf bytes.Buffer
err := b.Build(context.Background(), &buf)
err := b.Build(&buf)
require.NoError(t, err)
return buf.Bytes()
@@ -326,7 +325,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
}
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -384,7 +383,7 @@ func TestManifestString(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -398,7 +397,7 @@ func TestBuilderBuildEmpty(t *testing.T) {
var buf bytes.Buffer
err := b.Build(context.Background(), &buf)
err := b.Build(&buf)
require.NoError(t, err)
// Should still produce valid manifest with 0 files
@@ -417,7 +416,7 @@ func TestBuilderOmitsCreatedAtByDefault(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -439,7 +438,7 @@ func TestBuilderIncludesCreatedAtWhenRequested(t *testing.T) {
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
@@ -465,7 +464,7 @@ func TestBuilderDeterministicFileOrder(t *testing.T) {
}
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
+2 -2
View File
@@ -164,12 +164,12 @@ func (c *Checker) SigningPubKey() []byte {
// ExtractEmbeddedSigningKeyFP imports the manifest's embedded public key into a
// temporary keyring and extracts its fingerprint. This validates the key and
// returns its actual fingerprint from the key material itself.
func (c *Checker) ExtractEmbeddedSigningKeyFP(ctx context.Context) (string, error) {
func (c *Checker) ExtractEmbeddedSigningKeyFP() (string, error) {
if len(c.signingPubKey) == 0 {
return "", errNoSigningPubKey
}
return gpgExtractPubKeyFingerprint(ctx, c.signingPubKey)
return gpgExtractPubKeyFingerprint(c.signingPubKey)
}
// Check verifies all files against the manifest.
+1 -1
View File
@@ -61,7 +61,7 @@ func createTestManifest(
}
var buf bytes.Buffer
require.NoError(t, builder.Build(context.Background(), &buf))
require.NoError(t, builder.Build(&buf))
require.NoError(t, afero.WriteFile(fs, manifestPath, buf.Bytes(), 0o644))
}
-3
View File
@@ -2,7 +2,6 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -93,9 +92,7 @@ func (m *manifest) verifyOuterIntegrity() error {
)
}
// Loading a manifest takes no context; gpgTimeout still bounds gpg.
err = gpgVerify(
context.Background(),
[]byte(sigString),
m.pbOuter.GetSignature(),
m.pbOuter.GetSigningPubKey(),
+1 -2
View File
@@ -3,7 +3,6 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"fmt"
"testing"
@@ -123,7 +122,7 @@ func TestDeserializeValidManifestRoundTrips(t *testing.T) {
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
require.NoError(t, b.Build(&buf))
m, err := NewManifestFromReader(bytes.NewReader(buf.Bytes()))
require.NoError(t, err)
+2 -4
View File
@@ -2,7 +2,6 @@
package mfer
import (
"context"
"testing"
"github.com/stretchr/testify/assert"
@@ -81,7 +80,6 @@ func TestSerializeInternalErrorMessagesVerbatim(t *testing.T) {
t.Parallel()
m := &manifest{}
require.EqualError(t, m.generate(context.Background()),
"internal error: pbInner not set")
require.EqualError(t, m.generateOuter(context.Background()), "internal error")
require.EqualError(t, m.generate(), "internal error: pbInner not set")
require.EqualError(t, m.generateOuter(), "internal error")
}
+15 -48
View File
@@ -10,21 +10,9 @@ import (
"os/exec"
"path/filepath"
"strings"
"time"
)
const (
// gpgTimeout bounds every gpg run, which can otherwise wait forever on
// a passphrase prompt or a stalled gpg-agent. A minute leaves a person
// time to type a passphrase or touch a smartcard.
gpgTimeout = time.Minute
// gpgWaitDelay is how long a gpg run keeps waiting for gpg's stdout
// and stderr to close once gpg has been killed or has exited. Reading
// what gpg itself wrote takes far less; only a process gpg left behind
// holds them open longer.
gpgWaitDelay = time.Second
// privateDirPerms is the permission mode for temporary GPG home
// directories.
privateDirPerms os.FileMode = 0o700
@@ -78,20 +66,8 @@ func gpgArgs(opts []string, positional ...string) []string {
}
// runGPG runs the gpg binary in batch mode with the given arguments and
// optional stdin, returning captured stdout and stderr. gpg is killed when
// ctx ends or gpgTimeout passes, whichever comes first.
func runGPG(
ctx context.Context, stdin io.Reader, args ...string,
) (*bytes.Buffer, *bytes.Buffer, error) {
// exec.CommandContext kills only gpg itself. A gpg-agent that gpg
// starts runs detached and holds none of gpg's output, but another
// process gpg leaves behind (a wrapper script that runs the real gpg
// without exec, for example) can keep gpg's stdout or stderr open, and
// Run would wait for it to exit. WaitDelay stops that wait
// gpgWaitDelay after the kill; that process is left running.
ctx, cancel := context.WithTimeout(ctx, gpgTimeout)
defer cancel()
// optional stdin, returning captured stdout and stderr.
func runGPG(stdin io.Reader, args ...string) (*bytes.Buffer, *bytes.Buffer, error) {
fullArgs := append([]string{"--batch", "--no-tty"}, args...)
// G204: the executable name is a compile-time constant. The arguments
@@ -100,8 +76,7 @@ func runGPG(
// option or after the "--" end-of-options marker inserted by gpgArgs,
// and therefore cannot be reinterpreted by gpg as an option.
cmd := exec.CommandContext( //nolint:gosec // G204: see comment above
ctx, "gpg", fullArgs...)
cmd.WaitDelay = gpgWaitDelay
context.Background(), "gpg", fullArgs...)
cmd.Stdin = stdin
var stdout, stderr bytes.Buffer
@@ -110,14 +85,6 @@ func runGPG(
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil && ctx.Err() != nil {
// gpg was killed because ctx ended, which Run reports only as
// "signal: killed"; return the reason instead.
err = ctx.Err()
if errors.Is(err, context.DeadlineExceeded) {
err = fmt.Errorf("gpg timed out: %w", err)
}
}
return &stdout, &stderr, err
}
@@ -138,8 +105,8 @@ func parseFingerprint(colonOutput string) (string, bool) {
// gpgSign creates a detached signature of the data using the specified key.
// Returns the armored detached signature.
func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, bytes.NewReader(data),
func gpgSign(data []byte, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(bytes.NewReader(data),
"--detach-sign",
gpgOptArmor,
"--local-user", string(keyID),
@@ -153,8 +120,8 @@ func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
// gpgExportPublicKey exports the public key for the specified key ID.
// Returns the armored public key.
func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
func gpgExportPublicKey(keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(nil,
gpgArgs([]string{"--export", gpgOptArmor}, string(keyID))...,
)
if err != nil {
@@ -169,8 +136,8 @@ func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
}
// gpgGetKeyFingerprint gets the full fingerprint for a key ID.
func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(ctx, nil,
func gpgGetKeyFingerprint(keyID GPGKeyID) ([]byte, error) {
stdout, stderr, err := runGPG(nil,
gpgArgs([]string{"--with-colons", "--fingerprint"}, string(keyID))...,
)
if err != nil {
@@ -190,7 +157,7 @@ func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
// gpgExtractPubKeyFingerprint imports a public key into a temporary keyring
// and extracts its fingerprint. This verifies the key is valid and returns
// the actual fingerprint from the key material.
func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, error) {
func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-fingerprint-*")
if err != nil {
@@ -214,7 +181,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(ctx, nil,
_, importStderr, err := runGPG(nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
@@ -224,7 +191,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
}
// List keys to get fingerprint
listStdout, listStderr, err := runGPG(ctx, nil,
listStdout, listStderr, err := runGPG(nil,
"--homedir", tmpDir,
"--with-colons",
"--fingerprint",
@@ -245,7 +212,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
// gpgVerify verifies a detached signature against data using the provided public key.
// It creates a temporary keyring to import the public key for verification.
func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
func gpgVerify(data, signature, pubKey []byte) error {
// Create temporary directory for GPG operations
tmpDir, err := os.MkdirTemp("", "mfer-gpg-verify-*")
if err != nil {
@@ -285,7 +252,7 @@ func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
}
// Import the public key into the temporary keyring
_, importStderr, err := runGPG(ctx, nil,
_, importStderr, err := runGPG(nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
)
if err != nil {
@@ -295,7 +262,7 @@ func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
}
// Verify the signature
_, verifyStderr, err := runGPG(ctx, nil,
_, verifyStderr, err := runGPG(nil,
gpgArgs([]string{gpgOptHomedir, tmpDir, gpgOptVerify},
sigFile, dataFile)...,
)
+20 -100
View File
@@ -4,14 +4,11 @@ package mfer
import (
"bytes"
"context"
"io"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
@@ -46,11 +43,8 @@ Expire-Date: 0
paramsFile := filepath.Join(gpgHome, "key-params")
require.NoError(t, os.WriteFile(paramsFile, []byte(keyParams), 0o600))
ctx, cancel := context.WithTimeout(context.Background(), gpgTimeout)
defer cancel()
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
cmd := exec.CommandContext(ctx, "gpg",
cmd := exec.CommandContext(context.Background(), "gpg",
"--batch", "--gen-key", paramsFile)
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
@@ -61,7 +55,7 @@ Expire-Date: 0
}
// Get the key fingerprint
cmd = exec.CommandContext(ctx, "gpg",
cmd = exec.CommandContext(context.Background(), "gpg",
"--list-keys", "--with-colons", "test@mfer.test")
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
@@ -96,7 +90,7 @@ func TestGPGSign(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(context.Background(), data, keyID)
sig, err := gpgSign(data, keyID)
require.NoError(t, err)
assert.NotEmpty(t, sig)
assert.Contains(t, string(sig), "-----BEGIN PGP SIGNATURE-----")
@@ -107,7 +101,7 @@ func TestGPGExportPublicKey(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
pubKey, err := gpgExportPublicKey(keyID)
require.NoError(t, err)
assert.NotEmpty(t, pubKey)
assert.Contains(t, string(pubKey), "-----BEGIN PGP PUBLIC KEY BLOCK-----")
@@ -118,7 +112,7 @@ func TestGPGGetKeyFingerprint(t *testing.T) {
keyID, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
fingerprint, err := gpgGetKeyFingerprint(context.Background(), keyID)
fingerprint, err := gpgGetKeyFingerprint(keyID)
require.NoError(t, err)
assert.NotEmpty(t, fingerprint)
// The fingerprint should be 40 hex chars
@@ -152,12 +146,12 @@ func TestGPGOptionLikeKeyIDIsNotAnOption(t *testing.T) {
_, gpgHome := testGPGEnv(t)
t.Setenv("GNUPGHOME", gpgHome)
pubKey, err := gpgExportPublicKey(context.Background(), GPGKeyID("--version"))
pubKey, err := gpgExportPublicKey(GPGKeyID("--version"))
require.Error(t, err)
require.ErrorIs(t, err, errGPGKeyNotFound)
assert.NotContains(t, string(pubKey), "gpg (GnuPG)")
fpr, err := gpgGetKeyFingerprint(context.Background(), GPGKeyID("--version"))
fpr, err := gpgGetKeyFingerprint(GPGKeyID("--version"))
require.Error(t, err)
assert.NotContains(t, string(fpr), "gpg (GnuPG)")
}
@@ -168,8 +162,7 @@ func TestGPGSignInvalidKey(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
_, err := gpgSign(context.Background(), data,
GPGKeyID("NONEXISTENT_KEY_ID_12345"))
_, err := gpgSign(data, GPGKeyID("NONEXISTENT_KEY_ID_12345"))
assert.Error(t, err)
}
@@ -192,7 +185,7 @@ func TestBuilderWithSigning(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
err = b.Build(&buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are populated
@@ -258,14 +251,14 @@ func TestGPGVerify(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign and verify")
sig, err := gpgSign(context.Background(), data, keyID)
sig, err := gpgSign(data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
pubKey, err := gpgExportPublicKey(keyID)
require.NoError(t, err)
// Verify the signature
err = gpgVerify(context.Background(), data, sig, pubKey)
err = gpgVerify(data, sig, pubKey)
require.NoError(t, err)
}
@@ -274,15 +267,15 @@ func TestGPGVerifyInvalidSignature(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data to sign")
sig, err := gpgSign(context.Background(), data, keyID)
sig, err := gpgSign(data, keyID)
require.NoError(t, err)
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
pubKey, err := gpgExportPublicKey(keyID)
require.NoError(t, err)
// Try to verify with different data - should fail
wrongData := []byte("different data")
err = gpgVerify(context.Background(), wrongData, sig, pubKey)
err = gpgVerify(wrongData, sig, pubKey)
assert.Error(t, err)
}
@@ -291,12 +284,12 @@ func TestGPGVerifyBadPublicKey(t *testing.T) {
t.Setenv("GNUPGHOME", gpgHome)
data := []byte("test data")
sig, err := gpgSign(context.Background(), data, keyID)
sig, err := gpgSign(data, keyID)
require.NoError(t, err)
// Try to verify with invalid public key - should fail
badPubKey := []byte("not a valid public key")
err = gpgVerify(context.Background(), data, sig, badPubKey)
err = gpgVerify(data, sig, badPubKey)
assert.Error(t, err)
}
@@ -319,7 +312,7 @@ func TestManifestSignatureVerification(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
err = b.Build(&buf)
require.NoError(t, err)
// Parse the manifest - signature should be verified during load
@@ -348,7 +341,7 @@ func TestManifestTamperedSignatureFails(t *testing.T) {
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
err = b.Build(&buf)
require.NoError(t, err)
// Tamper with the signature by replacing some bytes
@@ -382,7 +375,7 @@ func TestBuilderWithoutSigning(t *testing.T) {
// Build the manifest
var buf bytes.Buffer
err = b.Build(context.Background(), &buf)
err = b.Build(&buf)
require.NoError(t, err)
// Parse the manifest and verify signature fields are empty
@@ -397,76 +390,3 @@ func TestBuilderWithoutSigning(t *testing.T) {
assert.Empty(t, manifest.pbOuter.GetSigningPubKey(),
"signing public key should be empty when not signing")
}
// fakeGPGPath writes script as an executable named gpg into a temporary
// directory and returns a PATH value with that directory first.
func fakeGPGPath(t *testing.T, script string) string {
t.Helper()
binDir := t.TempDir()
//nolint:gosec // G306: the fake gpg has to be executable
require.NoError(t, os.WriteFile(filepath.Join(binDir, "gpg"),
[]byte(script), 0o700))
return binDir + string(os.PathListSeparator) + os.Getenv("PATH")
}
// TestGPGTimeoutKillsGPG puts a fake gpg that never finishes first on
// PATH and checks that a run past its deadline is killed and reported as
// a timeout of the named operation, instead of hanging.
func TestGPGTimeoutKillsGPG(t *testing.T) {
t.Setenv("PATH", fakeGPGPath(t, "#!/bin/sh\nexec sleep 10\n"))
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
require.ErrorIs(t, err, context.DeadlineExceeded)
assert.Contains(t, err.Error(), "gpg sign failed: gpg timed out")
}
// TestGPGTimeoutWhenChildHoldsOutput uses a fake gpg that runs sleep as a
// child instead of exec-ing it, the way a wrapper script around the real
// gpg might. Killing the fake gpg leaves sleep holding its stdout and
// stderr open; the call must still return shortly after the deadline
// instead of waiting for sleep to exit. The fake gpg writes the process ID
// of sleep to a file so that the test can kill it before returning.
func TestGPGTimeoutWhenChildHoldsOutput(t *testing.T) {
pidFile := filepath.Join(t.TempDir(), "sleep.pid")
t.Setenv("PATH", fakeGPGPath(t,
"#!/bin/sh\nsleep 3 &\necho $! >'"+pidFile+"'\nwait\n"))
t.Cleanup(func() {
pid, err := os.ReadFile(pidFile) //nolint:gosec // G304: path inside t.TempDir()
require.NoError(t, err)
n, err := strconv.Atoi(strings.TrimSpace(string(pid)))
require.NoError(t, err)
sleep, err := os.FindProcess(n)
require.NoError(t, err)
require.NoError(t, sleep.Kill())
})
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
start := time.Now()
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
require.ErrorIs(t, err, context.DeadlineExceeded)
assert.Less(t, time.Since(start), 3*time.Second,
"the call waited for the child holding gpg's output to exit")
}
// TestBuildPassesContextToSigning checks that a caller can cancel the gpg
// runs that sign a manifest through the context given to Build.
func TestBuildPassesContextToSigning(t *testing.T) {
t.Parallel()
b := NewBuilder()
b.SetSigningOptions(&SigningOptions{KeyID: "any"})
ctx, cancel := context.WithCancel(context.Background())
cancel()
require.ErrorIs(t, b.Build(ctx, io.Discard), context.Canceled)
}
+2 -1
View File
@@ -283,7 +283,8 @@ func (s *Scanner) ToManifest(
}
// Build and write manifest
return builder.Build(ctx, w)
//nolint:contextcheck // Build's GPG signing exec is not cancellable by design
return builder.Build(w)
}
// configureBuilder constructs a manifest builder configured from the
+8 -9
View File
@@ -2,7 +2,6 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"errors"
"fmt"
@@ -51,13 +50,13 @@ func newTimestampFromTime(t time.Time) *Timestamp {
}
}
func (m *manifest) generate(ctx context.Context) error {
func (m *manifest) generate() error {
if m.pbInner == nil {
return errInnerNotSet
}
if m.pbOuter == nil {
e := m.generateOuter(ctx)
e := m.generateOuter()
if e != nil {
return e
}
@@ -78,7 +77,7 @@ func (m *manifest) generate(ctx context.Context) error {
return nil
}
func (m *manifest) generateOuter(ctx context.Context) error {
func (m *manifest) generateOuter() error {
if m.pbInner == nil {
return errInternal
}
@@ -136,7 +135,7 @@ func (m *manifest) generateOuter(ctx context.Context) error {
// Sign the manifest if signing options are provided
if m.signingOptions != nil && m.signingOptions.KeyID != "" {
return m.signOuter(ctx)
return m.signOuter()
}
return nil
@@ -144,27 +143,27 @@ func (m *manifest) generateOuter(ctx context.Context) error {
// signOuter signs the outer message with the configured GPG key and
// embeds the signature, signer fingerprint, and public key.
func (m *manifest) signOuter(ctx context.Context) error {
func (m *manifest) signOuter() error {
sigString, err := m.signatureString()
if err != nil {
return fmt.Errorf("failed to generate signature string: %w", err)
}
sig, err := gpgSign(ctx, []byte(sigString), m.signingOptions.KeyID)
sig, err := gpgSign([]byte(sigString), m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to sign manifest: %w", err)
}
m.pbOuter.Signature = sig
fingerprint, err := gpgGetKeyFingerprint(ctx, m.signingOptions.KeyID)
fingerprint, err := gpgGetKeyFingerprint(m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to get key fingerprint: %w", err)
}
m.pbOuter.Signer = fingerprint
pubKey, err := gpgExportPublicKey(ctx, m.signingOptions.KeyID)
pubKey, err := gpgExportPublicKey(m.signingOptions.KeyID)
if err != nil {
return fmt.Errorf("failed to export public key: %w", err)
}