Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
19a57e30b3 |
@@ -1,12 +1,12 @@
|
||||
# mfer
|
||||
|
||||
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
|
||||
thin wrapper command-line utility written in [Go](https://golang.org) and first
|
||||
published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
|
||||
It specifies and generates `.mf` manifest files over a directory tree of files
|
||||
to encapsulate metadata about them (such as cryptographic checksums or
|
||||
signatures over same) to aid in archiving, downloading, and streaming, or
|
||||
mirroring. The manifest files' data is serialized with Google's
|
||||
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
|
||||
(public domain) [Go](https://golang.org) library and command-line tool by
|
||||
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
|
||||
over a directory tree to encapsulate metadata about the files — such as
|
||||
cryptographic checksums and signatures over same — to aid in archiving,
|
||||
downloading, streaming, and mirroring. It was first published in 2022. The
|
||||
manifest files' data is serialized with Google's
|
||||
[protobuf serialization format](https://developers.google.com/protocol-buffers).
|
||||
The structure of these files can be found
|
||||
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
|
||||
@@ -21,6 +21,36 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
|
||||
as a de-facto standard and be incorporated into other software. A compatible
|
||||
javascript library is planned.
|
||||
|
||||
# Getting Started
|
||||
|
||||
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
|
||||
is committed, so no `protoc` toolchain is required:
|
||||
|
||||
```sh
|
||||
git clone https://git.eeqj.de/sneak/mfer.git
|
||||
cd mfer
|
||||
go build -o bin/mfer ./cmd/mfer
|
||||
```
|
||||
|
||||
Generate a manifest for a directory tree, verify it later, and fetch a published
|
||||
tree by URL:
|
||||
|
||||
```sh
|
||||
# Write .index.mf, a manifest of the files under the current directory.
|
||||
bin/mfer gen .
|
||||
|
||||
# Verify the files on disk against the manifest. Exits nonzero if any file
|
||||
# is missing or corrupted.
|
||||
bin/mfer check .index.mf
|
||||
|
||||
# Download and cryptographically verify a tree published over HTTP: mfer
|
||||
# fetches <url>/index.mf, then downloads every file it lists.
|
||||
bin/mfer fetch https://example.com/tree/
|
||||
```
|
||||
|
||||
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
|
||||
for a single command's options.
|
||||
|
||||
# Build Status
|
||||
|
||||
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
|
||||
@@ -82,7 +112,9 @@ Any changes submitted to this project must also be
|
||||
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
|
||||
tooling requirements, and workflow conventions.
|
||||
|
||||
# Problem Statement
|
||||
# Rationale
|
||||
|
||||
## The problem
|
||||
|
||||
Given a plain URL, there is no standard way to safely and programmatically
|
||||
download everything "under" that URL path. `wget -r` can traverse directory
|
||||
@@ -106,7 +138,7 @@ Real issues I face:
|
||||
- when I download a large file via HTTP, I have no way of knowing if the file
|
||||
content is what it's supposed to be
|
||||
|
||||
# Proposed Solution
|
||||
## The solution
|
||||
|
||||
A standard, a manifest file format, and a tool for generating same.
|
||||
|
||||
@@ -138,6 +170,28 @@ The manifest file would do several important things:
|
||||
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
|
||||
top-level infohash for the whole manifest?
|
||||
|
||||
# Design
|
||||
|
||||
The repository is split into a reusable library and a thin command-line wrapper
|
||||
around it.
|
||||
|
||||
- `mfer/` is the reusable library and the heart of the project: it defines the
|
||||
manifest format and implements building, scanning, checking, serialization,
|
||||
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
|
||||
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
|
||||
with `go get` and needs no `protoc` toolchain.
|
||||
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
|
||||
`check`, `freshen`, `export`, `list`, `fetch`, and `version` — that wire the
|
||||
library to the command-line interface.
|
||||
- `internal/log/` provides the logging used by the commands and the library.
|
||||
- `internal/bork/` defines the errors the library returns when reading a
|
||||
manifest that is truncated or lacks the magic bytes.
|
||||
- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
|
||||
into the `mfer` binary.
|
||||
|
||||
Everything under `internal/` is private to this repository; only the `mfer/`
|
||||
package is intended for import by other software.
|
||||
|
||||
# Design Goals
|
||||
|
||||
- Replace SHASUMS/SHASUMS.asc files
|
||||
@@ -271,9 +325,9 @@ Open work, open design questions included, is tracked in this repo's issues:
|
||||
- Issues:
|
||||
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
|
||||
|
||||
# Authors
|
||||
# Author
|
||||
|
||||
- [@sneak <sneak@sneak.berlin>](mailto:sneak@sneak.berlin)
|
||||
- [@sneak](https://sneak.berlin)
|
||||
|
||||
# License
|
||||
|
||||
|
||||
@@ -2,7 +2,6 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -127,9 +126,7 @@ func (mfa *CLIApp) fetchManifestToTemp(url string) (string, error) {
|
||||
|
||||
// verifyRequiredSigner enforces the --require-signature fingerprint
|
||||
// against the manifest's embedded signing key.
|
||||
func verifyRequiredSigner(
|
||||
ctx context.Context, chk *mfer.Checker, requiredSigner string,
|
||||
) error {
|
||||
func verifyRequiredSigner(chk *mfer.Checker, requiredSigner string) error {
|
||||
// Validate fingerprint format: must be exactly 40 hex characters
|
||||
if len(requiredSigner) != fingerprintHexLen {
|
||||
return fmt.Errorf("%w, got %d", errInvalidFingerprint, len(requiredSigner))
|
||||
@@ -148,7 +145,7 @@ func verifyRequiredSigner(
|
||||
// Extract fingerprint from the embedded public key (not from the
|
||||
// signer field). This validates the key is importable and gets its
|
||||
// actual fingerprint.
|
||||
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(ctx)
|
||||
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
|
||||
if err != nil {
|
||||
return fmt.Errorf(
|
||||
"failed to extract fingerprint from embedded signing key: %w", err)
|
||||
@@ -306,7 +303,7 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
|
||||
// Check signature requirement
|
||||
requiredSigner := ctx.String("require-signature")
|
||||
if requiredSigner != "" {
|
||||
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
|
||||
err = verifyRequiredSigner(chk, requiredSigner)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -83,8 +83,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
|
||||
t.Run("invalid fingerprint length", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := verifyRequiredSigner(context.Background(),
|
||||
unsignedChecker(t), "12345678")
|
||||
err := verifyRequiredSigner(unsignedChecker(t), "12345678")
|
||||
require.ErrorIs(t, err, errInvalidFingerprint)
|
||||
assert.EqualError(t, err,
|
||||
"invalid fingerprint: must be exactly 40 hex characters, got 8")
|
||||
@@ -93,8 +92,7 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
|
||||
t.Run("manifest not signed", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
err := verifyRequiredSigner(context.Background(),
|
||||
unsignedChecker(t), msgFpA)
|
||||
err := verifyRequiredSigner(unsignedChecker(t), msgFpA)
|
||||
require.ErrorIs(t, err, errManifestNotSigned)
|
||||
assert.EqualError(t, err,
|
||||
"manifest is not signed, but signature from "+msgFpA+" is required")
|
||||
@@ -111,10 +109,10 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
|
||||
func TestSignerMismatchMessage(t *testing.T) {
|
||||
chk := signedChecker(t)
|
||||
|
||||
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
|
||||
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
|
||||
require.NoError(t, err)
|
||||
|
||||
err = verifyRequiredSigner(context.Background(), chk, msgFpB)
|
||||
err = verifyRequiredSigner(chk, msgFpB)
|
||||
require.ErrorIs(t, err, errSignerMismatch)
|
||||
assert.EqualError(t, err,
|
||||
"embedded signing key fingerprint "+embeddedFP+
|
||||
@@ -162,7 +160,7 @@ func signedChecker(t *testing.T) *mfer.Checker {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
fs := afero.NewMemMapFs()
|
||||
require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -305,7 +304,7 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
|
||||
// writeFreshenedManifest writes the manifest atomically (write to a
|
||||
// temp file, then rename over the target).
|
||||
func writeFreshenedManifest(
|
||||
ctx context.Context, afs afero.Fs, builder *mfer.Builder, manifestPath string,
|
||||
afs afero.Fs, builder *mfer.Builder, manifestPath string,
|
||||
) error {
|
||||
tmpPath := manifestPath + ".tmp"
|
||||
|
||||
@@ -314,7 +313,7 @@ func writeFreshenedManifest(
|
||||
return fmt.Errorf("failed to create temp file: %w", err)
|
||||
}
|
||||
|
||||
err = builder.Build(ctx, outFile)
|
||||
err = builder.Build(outFile)
|
||||
_ = outFile.Close()
|
||||
|
||||
if err != nil {
|
||||
@@ -531,7 +530,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
|
||||
}
|
||||
|
||||
// Write updated manifest atomically (write to temp, then rename)
|
||||
err = writeFreshenedManifest(ctx.Context, mfa.Fs, hasher.builder, manifestPath)
|
||||
err = writeFreshenedManifest(mfa.Fs, hasher.builder, manifestPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
+4
-6
@@ -3,7 +3,6 @@
|
||||
package mfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -282,9 +281,8 @@ func (b *Builder) SetSigningOptions(opts *SigningOptions) {
|
||||
b.signingOptions = opts
|
||||
}
|
||||
|
||||
// Build finalizes the manifest and writes it to the writer. ctx bounds the
|
||||
// gpg runs that sign the manifest when signing options are set.
|
||||
func (b *Builder) Build(ctx context.Context, w io.Writer) error {
|
||||
// Build finalizes the manifest and writes it to the writer.
|
||||
func (b *Builder) Build(w io.Writer) error {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
|
||||
@@ -310,13 +308,13 @@ func (b *Builder) Build(ctx context.Context, w io.Writer) error {
|
||||
}
|
||||
|
||||
// Generate outer wrapper
|
||||
err := m.generateOuter(ctx)
|
||||
err := m.generateOuter()
|
||||
if err != nil {
|
||||
return fmt.Errorf("build: generate outer: %w", err)
|
||||
}
|
||||
|
||||
// Generate final output
|
||||
err = m.generate(ctx)
|
||||
err = m.generate()
|
||||
if err != nil {
|
||||
return fmt.Errorf("build: generate: %w", err)
|
||||
}
|
||||
|
||||
@@ -3,7 +3,6 @@ package mfer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -114,7 +113,7 @@ func TestBuilderBuild(t *testing.T) {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err = b.Build(context.Background(), &buf)
|
||||
err = b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should have magic bytes
|
||||
@@ -178,7 +177,7 @@ func TestBuilderDeterministicOutput(t *testing.T) {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err := b.Build(context.Background(), &buf)
|
||||
err := b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
return buf.Bytes()
|
||||
@@ -326,7 +325,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
m, err := NewManifestFromReader(&buf)
|
||||
require.NoError(t, err)
|
||||
@@ -384,7 +383,7 @@ func TestManifestString(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
m, err := NewManifestFromReader(&buf)
|
||||
require.NoError(t, err)
|
||||
@@ -398,7 +397,7 @@ func TestBuilderBuildEmpty(t *testing.T) {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err := b.Build(context.Background(), &buf)
|
||||
err := b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Should still produce valid manifest with 0 files
|
||||
@@ -417,7 +416,7 @@ func TestBuilderOmitsCreatedAtByDefault(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
m, err := NewManifestFromReader(&buf)
|
||||
require.NoError(t, err)
|
||||
@@ -439,7 +438,7 @@ func TestBuilderIncludesCreatedAtWhenRequested(t *testing.T) {
|
||||
require.NoError(t, err)
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
m, err := NewManifestFromReader(&buf)
|
||||
require.NoError(t, err)
|
||||
@@ -465,7 +464,7 @@ func TestBuilderDeterministicFileOrder(t *testing.T) {
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
m, err := NewManifestFromReader(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
|
||||
+2
-2
@@ -164,12 +164,12 @@ func (c *Checker) SigningPubKey() []byte {
|
||||
// ExtractEmbeddedSigningKeyFP imports the manifest's embedded public key into a
|
||||
// temporary keyring and extracts its fingerprint. This validates the key and
|
||||
// returns its actual fingerprint from the key material itself.
|
||||
func (c *Checker) ExtractEmbeddedSigningKeyFP(ctx context.Context) (string, error) {
|
||||
func (c *Checker) ExtractEmbeddedSigningKeyFP() (string, error) {
|
||||
if len(c.signingPubKey) == 0 {
|
||||
return "", errNoSigningPubKey
|
||||
}
|
||||
|
||||
return gpgExtractPubKeyFingerprint(ctx, c.signingPubKey)
|
||||
return gpgExtractPubKeyFingerprint(c.signingPubKey)
|
||||
}
|
||||
|
||||
// Check verifies all files against the manifest.
|
||||
|
||||
@@ -61,7 +61,7 @@ func createTestManifest(
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, builder.Build(context.Background(), &buf))
|
||||
require.NoError(t, builder.Build(&buf))
|
||||
require.NoError(t, afero.WriteFile(fs, manifestPath, buf.Bytes(), 0o644))
|
||||
}
|
||||
|
||||
|
||||
@@ -2,7 +2,6 @@ package mfer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -93,9 +92,7 @@ func (m *manifest) verifyOuterIntegrity() error {
|
||||
)
|
||||
}
|
||||
|
||||
// Loading a manifest takes no context; gpgTimeout still bounds gpg.
|
||||
err = gpgVerify(
|
||||
context.Background(),
|
||||
[]byte(sigString),
|
||||
m.pbOuter.GetSignature(),
|
||||
m.pbOuter.GetSigningPubKey(),
|
||||
|
||||
@@ -3,7 +3,6 @@ package mfer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"testing"
|
||||
@@ -123,7 +122,7 @@ func TestDeserializeValidManifestRoundTrips(t *testing.T) {
|
||||
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
|
||||
|
||||
var buf bytes.Buffer
|
||||
require.NoError(t, b.Build(context.Background(), &buf))
|
||||
require.NoError(t, b.Build(&buf))
|
||||
|
||||
m, err := NewManifestFromReader(bytes.NewReader(buf.Bytes()))
|
||||
require.NoError(t, err)
|
||||
|
||||
+2
-4
@@ -2,7 +2,6 @@
|
||||
package mfer
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -81,7 +80,6 @@ func TestSerializeInternalErrorMessagesVerbatim(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
m := &manifest{}
|
||||
require.EqualError(t, m.generate(context.Background()),
|
||||
"internal error: pbInner not set")
|
||||
require.EqualError(t, m.generateOuter(context.Background()), "internal error")
|
||||
require.EqualError(t, m.generate(), "internal error: pbInner not set")
|
||||
require.EqualError(t, m.generateOuter(), "internal error")
|
||||
}
|
||||
|
||||
+15
-48
@@ -10,21 +10,9 @@ import (
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const (
|
||||
// gpgTimeout bounds every gpg run, which can otherwise wait forever on
|
||||
// a passphrase prompt or a stalled gpg-agent. A minute leaves a person
|
||||
// time to type a passphrase or touch a smartcard.
|
||||
gpgTimeout = time.Minute
|
||||
|
||||
// gpgWaitDelay is how long a gpg run keeps waiting for gpg's stdout
|
||||
// and stderr to close once gpg has been killed or has exited. Reading
|
||||
// what gpg itself wrote takes far less; only a process gpg left behind
|
||||
// holds them open longer.
|
||||
gpgWaitDelay = time.Second
|
||||
|
||||
// privateDirPerms is the permission mode for temporary GPG home
|
||||
// directories.
|
||||
privateDirPerms os.FileMode = 0o700
|
||||
@@ -78,20 +66,8 @@ func gpgArgs(opts []string, positional ...string) []string {
|
||||
}
|
||||
|
||||
// runGPG runs the gpg binary in batch mode with the given arguments and
|
||||
// optional stdin, returning captured stdout and stderr. gpg is killed when
|
||||
// ctx ends or gpgTimeout passes, whichever comes first.
|
||||
func runGPG(
|
||||
ctx context.Context, stdin io.Reader, args ...string,
|
||||
) (*bytes.Buffer, *bytes.Buffer, error) {
|
||||
// exec.CommandContext kills only gpg itself. A gpg-agent that gpg
|
||||
// starts runs detached and holds none of gpg's output, but another
|
||||
// process gpg leaves behind (a wrapper script that runs the real gpg
|
||||
// without exec, for example) can keep gpg's stdout or stderr open, and
|
||||
// Run would wait for it to exit. WaitDelay stops that wait
|
||||
// gpgWaitDelay after the kill; that process is left running.
|
||||
ctx, cancel := context.WithTimeout(ctx, gpgTimeout)
|
||||
defer cancel()
|
||||
|
||||
// optional stdin, returning captured stdout and stderr.
|
||||
func runGPG(stdin io.Reader, args ...string) (*bytes.Buffer, *bytes.Buffer, error) {
|
||||
fullArgs := append([]string{"--batch", "--no-tty"}, args...)
|
||||
|
||||
// G204: the executable name is a compile-time constant. The arguments
|
||||
@@ -100,8 +76,7 @@ func runGPG(
|
||||
// option or after the "--" end-of-options marker inserted by gpgArgs,
|
||||
// and therefore cannot be reinterpreted by gpg as an option.
|
||||
cmd := exec.CommandContext( //nolint:gosec // G204: see comment above
|
||||
ctx, "gpg", fullArgs...)
|
||||
cmd.WaitDelay = gpgWaitDelay
|
||||
context.Background(), "gpg", fullArgs...)
|
||||
cmd.Stdin = stdin
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
@@ -110,14 +85,6 @@ func runGPG(
|
||||
cmd.Stderr = &stderr
|
||||
|
||||
err := cmd.Run()
|
||||
if err != nil && ctx.Err() != nil {
|
||||
// gpg was killed because ctx ended, which Run reports only as
|
||||
// "signal: killed"; return the reason instead.
|
||||
err = ctx.Err()
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
err = fmt.Errorf("gpg timed out: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return &stdout, &stderr, err
|
||||
}
|
||||
@@ -138,8 +105,8 @@ func parseFingerprint(colonOutput string) (string, bool) {
|
||||
|
||||
// gpgSign creates a detached signature of the data using the specified key.
|
||||
// Returns the armored detached signature.
|
||||
func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(ctx, bytes.NewReader(data),
|
||||
func gpgSign(data []byte, keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(bytes.NewReader(data),
|
||||
"--detach-sign",
|
||||
gpgOptArmor,
|
||||
"--local-user", string(keyID),
|
||||
@@ -153,8 +120,8 @@ func gpgSign(ctx context.Context, data []byte, keyID GPGKeyID) ([]byte, error) {
|
||||
|
||||
// gpgExportPublicKey exports the public key for the specified key ID.
|
||||
// Returns the armored public key.
|
||||
func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(ctx, nil,
|
||||
func gpgExportPublicKey(keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(nil,
|
||||
gpgArgs([]string{"--export", gpgOptArmor}, string(keyID))...,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -169,8 +136,8 @@ func gpgExportPublicKey(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
|
||||
}
|
||||
|
||||
// gpgGetKeyFingerprint gets the full fingerprint for a key ID.
|
||||
func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(ctx, nil,
|
||||
func gpgGetKeyFingerprint(keyID GPGKeyID) ([]byte, error) {
|
||||
stdout, stderr, err := runGPG(nil,
|
||||
gpgArgs([]string{"--with-colons", "--fingerprint"}, string(keyID))...,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -190,7 +157,7 @@ func gpgGetKeyFingerprint(ctx context.Context, keyID GPGKeyID) ([]byte, error) {
|
||||
// gpgExtractPubKeyFingerprint imports a public key into a temporary keyring
|
||||
// and extracts its fingerprint. This verifies the key is valid and returns
|
||||
// the actual fingerprint from the key material.
|
||||
func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, error) {
|
||||
func gpgExtractPubKeyFingerprint(pubKey []byte) (string, error) {
|
||||
// Create temporary directory for GPG operations
|
||||
tmpDir, err := os.MkdirTemp("", "mfer-gpg-fingerprint-*")
|
||||
if err != nil {
|
||||
@@ -214,7 +181,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
|
||||
}
|
||||
|
||||
// Import the public key into the temporary keyring
|
||||
_, importStderr, err := runGPG(ctx, nil,
|
||||
_, importStderr, err := runGPG(nil,
|
||||
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -224,7 +191,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
|
||||
}
|
||||
|
||||
// List keys to get fingerprint
|
||||
listStdout, listStderr, err := runGPG(ctx, nil,
|
||||
listStdout, listStderr, err := runGPG(nil,
|
||||
"--homedir", tmpDir,
|
||||
"--with-colons",
|
||||
"--fingerprint",
|
||||
@@ -245,7 +212,7 @@ func gpgExtractPubKeyFingerprint(ctx context.Context, pubKey []byte) (string, er
|
||||
|
||||
// gpgVerify verifies a detached signature against data using the provided public key.
|
||||
// It creates a temporary keyring to import the public key for verification.
|
||||
func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
|
||||
func gpgVerify(data, signature, pubKey []byte) error {
|
||||
// Create temporary directory for GPG operations
|
||||
tmpDir, err := os.MkdirTemp("", "mfer-gpg-verify-*")
|
||||
if err != nil {
|
||||
@@ -285,7 +252,7 @@ func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
|
||||
}
|
||||
|
||||
// Import the public key into the temporary keyring
|
||||
_, importStderr, err := runGPG(ctx, nil,
|
||||
_, importStderr, err := runGPG(nil,
|
||||
gpgArgs([]string{gpgOptHomedir, tmpDir, "--import"}, pubKeyFile)...,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -295,7 +262,7 @@ func gpgVerify(ctx context.Context, data, signature, pubKey []byte) error {
|
||||
}
|
||||
|
||||
// Verify the signature
|
||||
_, verifyStderr, err := runGPG(ctx, nil,
|
||||
_, verifyStderr, err := runGPG(nil,
|
||||
gpgArgs([]string{gpgOptHomedir, tmpDir, gpgOptVerify},
|
||||
sigFile, dataFile)...,
|
||||
)
|
||||
|
||||
+20
-100
@@ -4,14 +4,11 @@ package mfer
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/afero"
|
||||
"github.com/stretchr/testify/assert"
|
||||
@@ -46,11 +43,8 @@ Expire-Date: 0
|
||||
paramsFile := filepath.Join(gpgHome, "key-params")
|
||||
require.NoError(t, os.WriteFile(paramsFile, []byte(keyParams), 0o600))
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), gpgTimeout)
|
||||
defer cancel()
|
||||
|
||||
//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
|
||||
cmd := exec.CommandContext(ctx, "gpg",
|
||||
cmd := exec.CommandContext(context.Background(), "gpg",
|
||||
"--batch", "--gen-key", paramsFile)
|
||||
|
||||
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
|
||||
@@ -61,7 +55,7 @@ Expire-Date: 0
|
||||
}
|
||||
|
||||
// Get the key fingerprint
|
||||
cmd = exec.CommandContext(ctx, "gpg",
|
||||
cmd = exec.CommandContext(context.Background(), "gpg",
|
||||
"--list-keys", "--with-colons", "test@mfer.test")
|
||||
|
||||
cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
|
||||
@@ -96,7 +90,7 @@ func TestGPGSign(t *testing.T) {
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
data := []byte("test data to sign")
|
||||
sig, err := gpgSign(context.Background(), data, keyID)
|
||||
sig, err := gpgSign(data, keyID)
|
||||
require.NoError(t, err)
|
||||
assert.NotEmpty(t, sig)
|
||||
assert.Contains(t, string(sig), "-----BEGIN PGP SIGNATURE-----")
|
||||
@@ -107,7 +101,7 @@ func TestGPGExportPublicKey(t *testing.T) {
|
||||
keyID, gpgHome := testGPGEnv(t)
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
|
||||
pubKey, err := gpgExportPublicKey(keyID)
|
||||
require.NoError(t, err)
|
||||
assert.NotEmpty(t, pubKey)
|
||||
assert.Contains(t, string(pubKey), "-----BEGIN PGP PUBLIC KEY BLOCK-----")
|
||||
@@ -118,7 +112,7 @@ func TestGPGGetKeyFingerprint(t *testing.T) {
|
||||
keyID, gpgHome := testGPGEnv(t)
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
fingerprint, err := gpgGetKeyFingerprint(context.Background(), keyID)
|
||||
fingerprint, err := gpgGetKeyFingerprint(keyID)
|
||||
require.NoError(t, err)
|
||||
assert.NotEmpty(t, fingerprint)
|
||||
// The fingerprint should be 40 hex chars
|
||||
@@ -152,12 +146,12 @@ func TestGPGOptionLikeKeyIDIsNotAnOption(t *testing.T) {
|
||||
_, gpgHome := testGPGEnv(t)
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
pubKey, err := gpgExportPublicKey(context.Background(), GPGKeyID("--version"))
|
||||
pubKey, err := gpgExportPublicKey(GPGKeyID("--version"))
|
||||
require.Error(t, err)
|
||||
require.ErrorIs(t, err, errGPGKeyNotFound)
|
||||
assert.NotContains(t, string(pubKey), "gpg (GnuPG)")
|
||||
|
||||
fpr, err := gpgGetKeyFingerprint(context.Background(), GPGKeyID("--version"))
|
||||
fpr, err := gpgGetKeyFingerprint(GPGKeyID("--version"))
|
||||
require.Error(t, err)
|
||||
assert.NotContains(t, string(fpr), "gpg (GnuPG)")
|
||||
}
|
||||
@@ -168,8 +162,7 @@ func TestGPGSignInvalidKey(t *testing.T) {
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
data := []byte("test data")
|
||||
_, err := gpgSign(context.Background(), data,
|
||||
GPGKeyID("NONEXISTENT_KEY_ID_12345"))
|
||||
_, err := gpgSign(data, GPGKeyID("NONEXISTENT_KEY_ID_12345"))
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -192,7 +185,7 @@ func TestBuilderWithSigning(t *testing.T) {
|
||||
// Build the manifest
|
||||
var buf bytes.Buffer
|
||||
|
||||
err = b.Build(context.Background(), &buf)
|
||||
err = b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Parse the manifest and verify signature fields are populated
|
||||
@@ -258,14 +251,14 @@ func TestGPGVerify(t *testing.T) {
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
data := []byte("test data to sign and verify")
|
||||
sig, err := gpgSign(context.Background(), data, keyID)
|
||||
sig, err := gpgSign(data, keyID)
|
||||
require.NoError(t, err)
|
||||
|
||||
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
|
||||
pubKey, err := gpgExportPublicKey(keyID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Verify the signature
|
||||
err = gpgVerify(context.Background(), data, sig, pubKey)
|
||||
err = gpgVerify(data, sig, pubKey)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
@@ -274,15 +267,15 @@ func TestGPGVerifyInvalidSignature(t *testing.T) {
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
data := []byte("test data to sign")
|
||||
sig, err := gpgSign(context.Background(), data, keyID)
|
||||
sig, err := gpgSign(data, keyID)
|
||||
require.NoError(t, err)
|
||||
|
||||
pubKey, err := gpgExportPublicKey(context.Background(), keyID)
|
||||
pubKey, err := gpgExportPublicKey(keyID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Try to verify with different data - should fail
|
||||
wrongData := []byte("different data")
|
||||
err = gpgVerify(context.Background(), wrongData, sig, pubKey)
|
||||
err = gpgVerify(wrongData, sig, pubKey)
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -291,12 +284,12 @@ func TestGPGVerifyBadPublicKey(t *testing.T) {
|
||||
t.Setenv("GNUPGHOME", gpgHome)
|
||||
|
||||
data := []byte("test data")
|
||||
sig, err := gpgSign(context.Background(), data, keyID)
|
||||
sig, err := gpgSign(data, keyID)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Try to verify with invalid public key - should fail
|
||||
badPubKey := []byte("not a valid public key")
|
||||
err = gpgVerify(context.Background(), data, sig, badPubKey)
|
||||
err = gpgVerify(data, sig, badPubKey)
|
||||
assert.Error(t, err)
|
||||
}
|
||||
|
||||
@@ -319,7 +312,7 @@ func TestManifestSignatureVerification(t *testing.T) {
|
||||
// Build the manifest
|
||||
var buf bytes.Buffer
|
||||
|
||||
err = b.Build(context.Background(), &buf)
|
||||
err = b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Parse the manifest - signature should be verified during load
|
||||
@@ -348,7 +341,7 @@ func TestManifestTamperedSignatureFails(t *testing.T) {
|
||||
|
||||
var buf bytes.Buffer
|
||||
|
||||
err = b.Build(context.Background(), &buf)
|
||||
err = b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Tamper with the signature by replacing some bytes
|
||||
@@ -382,7 +375,7 @@ func TestBuilderWithoutSigning(t *testing.T) {
|
||||
// Build the manifest
|
||||
var buf bytes.Buffer
|
||||
|
||||
err = b.Build(context.Background(), &buf)
|
||||
err = b.Build(&buf)
|
||||
require.NoError(t, err)
|
||||
|
||||
// Parse the manifest and verify signature fields are empty
|
||||
@@ -397,76 +390,3 @@ func TestBuilderWithoutSigning(t *testing.T) {
|
||||
assert.Empty(t, manifest.pbOuter.GetSigningPubKey(),
|
||||
"signing public key should be empty when not signing")
|
||||
}
|
||||
|
||||
// fakeGPGPath writes script as an executable named gpg into a temporary
|
||||
// directory and returns a PATH value with that directory first.
|
||||
func fakeGPGPath(t *testing.T, script string) string {
|
||||
t.Helper()
|
||||
|
||||
binDir := t.TempDir()
|
||||
//nolint:gosec // G306: the fake gpg has to be executable
|
||||
require.NoError(t, os.WriteFile(filepath.Join(binDir, "gpg"),
|
||||
[]byte(script), 0o700))
|
||||
|
||||
return binDir + string(os.PathListSeparator) + os.Getenv("PATH")
|
||||
}
|
||||
|
||||
// TestGPGTimeoutKillsGPG puts a fake gpg that never finishes first on
|
||||
// PATH and checks that a run past its deadline is killed and reported as
|
||||
// a timeout of the named operation, instead of hanging.
|
||||
func TestGPGTimeoutKillsGPG(t *testing.T) {
|
||||
t.Setenv("PATH", fakeGPGPath(t, "#!/bin/sh\nexec sleep 10\n"))
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
|
||||
require.ErrorIs(t, err, context.DeadlineExceeded)
|
||||
assert.Contains(t, err.Error(), "gpg sign failed: gpg timed out")
|
||||
}
|
||||
|
||||
// TestGPGTimeoutWhenChildHoldsOutput uses a fake gpg that runs sleep as a
|
||||
// child instead of exec-ing it, the way a wrapper script around the real
|
||||
// gpg might. Killing the fake gpg leaves sleep holding its stdout and
|
||||
// stderr open; the call must still return shortly after the deadline
|
||||
// instead of waiting for sleep to exit. The fake gpg writes the process ID
|
||||
// of sleep to a file so that the test can kill it before returning.
|
||||
func TestGPGTimeoutWhenChildHoldsOutput(t *testing.T) {
|
||||
pidFile := filepath.Join(t.TempDir(), "sleep.pid")
|
||||
t.Setenv("PATH", fakeGPGPath(t,
|
||||
"#!/bin/sh\nsleep 3 &\necho $! >'"+pidFile+"'\nwait\n"))
|
||||
t.Cleanup(func() {
|
||||
pid, err := os.ReadFile(pidFile) //nolint:gosec // G304: path inside t.TempDir()
|
||||
require.NoError(t, err)
|
||||
|
||||
n, err := strconv.Atoi(strings.TrimSpace(string(pid)))
|
||||
require.NoError(t, err)
|
||||
|
||||
sleep, err := os.FindProcess(n)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, sleep.Kill())
|
||||
})
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
_, err := gpgSign(ctx, []byte("data"), GPGKeyID("any"))
|
||||
require.ErrorIs(t, err, context.DeadlineExceeded)
|
||||
assert.Less(t, time.Since(start), 3*time.Second,
|
||||
"the call waited for the child holding gpg's output to exit")
|
||||
}
|
||||
|
||||
// TestBuildPassesContextToSigning checks that a caller can cancel the gpg
|
||||
// runs that sign a manifest through the context given to Build.
|
||||
func TestBuildPassesContextToSigning(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
b := NewBuilder()
|
||||
b.SetSigningOptions(&SigningOptions{KeyID: "any"})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
|
||||
require.ErrorIs(t, b.Build(ctx, io.Discard), context.Canceled)
|
||||
}
|
||||
|
||||
+2
-1
@@ -283,7 +283,8 @@ func (s *Scanner) ToManifest(
|
||||
}
|
||||
|
||||
// Build and write manifest
|
||||
return builder.Build(ctx, w)
|
||||
//nolint:contextcheck // Build's GPG signing exec is not cancellable by design
|
||||
return builder.Build(w)
|
||||
}
|
||||
|
||||
// configureBuilder constructs a manifest builder configured from the
|
||||
|
||||
+8
-9
@@ -2,7 +2,6 @@ package mfer
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -51,13 +50,13 @@ func newTimestampFromTime(t time.Time) *Timestamp {
|
||||
}
|
||||
}
|
||||
|
||||
func (m *manifest) generate(ctx context.Context) error {
|
||||
func (m *manifest) generate() error {
|
||||
if m.pbInner == nil {
|
||||
return errInnerNotSet
|
||||
}
|
||||
|
||||
if m.pbOuter == nil {
|
||||
e := m.generateOuter(ctx)
|
||||
e := m.generateOuter()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
@@ -78,7 +77,7 @@ func (m *manifest) generate(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *manifest) generateOuter(ctx context.Context) error {
|
||||
func (m *manifest) generateOuter() error {
|
||||
if m.pbInner == nil {
|
||||
return errInternal
|
||||
}
|
||||
@@ -136,7 +135,7 @@ func (m *manifest) generateOuter(ctx context.Context) error {
|
||||
|
||||
// Sign the manifest if signing options are provided
|
||||
if m.signingOptions != nil && m.signingOptions.KeyID != "" {
|
||||
return m.signOuter(ctx)
|
||||
return m.signOuter()
|
||||
}
|
||||
|
||||
return nil
|
||||
@@ -144,27 +143,27 @@ func (m *manifest) generateOuter(ctx context.Context) error {
|
||||
|
||||
// signOuter signs the outer message with the configured GPG key and
|
||||
// embeds the signature, signer fingerprint, and public key.
|
||||
func (m *manifest) signOuter(ctx context.Context) error {
|
||||
func (m *manifest) signOuter() error {
|
||||
sigString, err := m.signatureString()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to generate signature string: %w", err)
|
||||
}
|
||||
|
||||
sig, err := gpgSign(ctx, []byte(sigString), m.signingOptions.KeyID)
|
||||
sig, err := gpgSign([]byte(sigString), m.signingOptions.KeyID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to sign manifest: %w", err)
|
||||
}
|
||||
|
||||
m.pbOuter.Signature = sig
|
||||
|
||||
fingerprint, err := gpgGetKeyFingerprint(ctx, m.signingOptions.KeyID)
|
||||
fingerprint, err := gpgGetKeyFingerprint(m.signingOptions.KeyID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to get key fingerprint: %w", err)
|
||||
}
|
||||
|
||||
m.pbOuter.Signer = fingerprint
|
||||
|
||||
pubKey, err := gpgExportPublicKey(ctx, m.signingOptions.KeyID)
|
||||
pubKey, err := gpgExportPublicKey(m.signingOptions.KeyID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to export public key: %w", err)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user