Validate manifest entry paths on deserialize (closes #61)

Untrusted .mf files were parsed with no path validation, so an entry
like ../../etc/passwd flowed into filepath.Join against the checker's
base path. ValidatePath ran only on the write side (Builder). Enforce it
on every entry in deserializeInner, so all consumers — Checker now, any
restore/extract path later — are protected. Reject the whole manifest on
the first offender rather than dropping entries, which could hide files
from a check; failures wrap a new errInvalidManifestPath sentinel and
name the path.

A path that is not valid UTF-8 is refused at load time by the proto3
string decoder before validation runs; the test covers that plus the
traversal, absolute, backslash, and empty cases, and a valid round-trip.

Model: opus-4-8

.editorconfig

@@ -1,12 +0,0 @@
-root = true
-
-[*]
-indent_style = space
-indent_size = 4
-end_of_line = lf
-charset = utf-8
-trim_trailing_whitespace = true
-insert_final_newline = true
-
-[Makefile]
-indent_style = tab

.gitignore

@@ -10,24 +10,5 @@ modcache.tzst
 # Generated manifest files
 .index.mf
 
-# Secrets
+# Stale files
-.env
+.drone.yml
-.env.*
-*.key
-*.pem
-
-# OS files
-.DS_Store
-Thumbs.db
-
-# Editor files
-*.swp
-*.swo
-*~
-.idea/
-.vscode/
-
-# Go build artifacts
-*.log
-*.out
-*.test

README.md

@@ -1,12 +1,12 @@
 # mfer
 
-[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
+[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
-(public domain) [Go](https://golang.org) library and command-line tool by
+thin wrapper command-line utility written in [Go](https://golang.org) and first
-[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
+published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
-over a directory tree to encapsulate metadata about the files — such as
+It specifies and generates `.mf` manifest files over a directory tree of files
-cryptographic checksums and signatures over same — to aid in archiving,
+to encapsulate metadata about them (such as cryptographic checksums or
-downloading, streaming, and mirroring. It was first published in 2022. The
+signatures over same) to aid in archiving, downloading, and streaming, or
-manifest files' data is serialized with Google's
+mirroring. The manifest files' data is serialized with Google's
 [protobuf serialization format](https://developers.google.com/protocol-buffers).
 The structure of these files can be found
 [in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
@@ -21,36 +21,6 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
 as a de-facto standard and be incorporated into other software. A compatible
 javascript library is planned.
 
-# Getting Started
-
-`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
-is committed, so no `protoc` toolchain is required:
-
-```sh
-git clone https://git.eeqj.de/sneak/mfer.git
-cd mfer
-go build -o bin/mfer ./cmd/mfer
-```
-
-Generate a manifest for a directory tree, verify it later, and fetch a published
-tree by URL:
-
-```sh
-# Write .index.mf describing every file under the current directory.
-bin/mfer gen .
-
-# Verify the files on disk against the manifest. Exits nonzero if any file
-# is missing or corrupted.
-bin/mfer check .index.mf
-
-# Download and cryptographically verify a tree published over HTTP: mfer
-# fetches <url>/index.mf, then downloads every file it lists.
-bin/mfer fetch https://example.com/tree/
-```
-
-Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
-for a single command's options.
-
 # Build Status
 
 CI runs via `script/cibuild` (`docker build .`), which executes `make check`
@@ -111,9 +81,7 @@ Any changes submitted to this project must also be
 See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
 tooling requirements, and workflow conventions.
 
-# Rationale
+# Problem Statement
-
-## The problem
 
 Given a plain URL, there is no standard way to safely and programmatically
 download everything "under" that URL path. `wget -r` can traverse directory
@@ -137,7 +105,7 @@ Real issues I face:
 - when I download a large file via HTTP, I have no way of knowing if the file
   content is what it's supposed to be
 
-## The solution
+# Proposed Solution
 
 A standard, a manifest file format, and a tool for generating same.
 
@@ -169,27 +137,6 @@ The manifest file would do several important things:
     - maybe a bittorrent chunklist for torrent client compatibility? perhaps a
       top-level infohash for the whole manifest?
 
-# Design
-
-The repository is split into a reusable library and a thin command-line wrapper
-around it.
-
-- `mfer/` is the reusable library and the heart of the project: it defines the
-  manifest format and implements building, scanning, checking, serialization,
-  and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
-  produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
-  with `go get` and needs no `protoc` toolchain.
-- `internal/cli/` holds the command implementations — `gen`, `check`, `freshen`,
-  `export`, `list`, and `fetch` — that wire the library to the command-line
-  interface.
-- `internal/log/` provides the logging used across the commands.
-- `internal/bork/` provides error-handling support.
-- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
-  into the `mfer` binary.
-
-Everything under `internal/` is private to this repository; only the `mfer/`
-package is intended for import by other software.
-
 # Design Goals
 
 - Replace SHASUMS/SHASUMS.asc files
@@ -515,9 +462,9 @@ proto `go_package` option. Which is canonical?
 - Issues:
   [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
 
-# Author
+# Authors
 
-- [@sneak](https://sneak.berlin)
+- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
 
 # License
 

TODO.md

@@ -24,14 +24,8 @@ only thing left of the `chore/align-repo-policies` branch is the list below.
 
 # Completed Steps
 
-- 2026-09-21: added the required README sections — named author, license, and
+- 2026-09-21: validate manifest entry paths on deserialize so untrusted `.mf`
-  category in the Description first line; added Getting Started (verified
+  files cannot make `Checker` stat or read outside `basePath` (#61)
-  install/usage block), Rationale (folding in Problem Statement and Proposed
-  Solution), and a Design section for the package layout; renamed Authors to
-  Author (#75)
-- 2026-09-21: added the canonical `.editorconfig`, made `.gitignore` cover
-  secrets, OS, editor, and Go artifacts, and removed the dead Drone CI
-  references from `.gitignore` and `bin/gitrev.sh` (#72)
 - 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and
   `script/fmt-check` one shared prettier file set via `script/prettier`, dropped
   the `|| true` that hid prettier failures, and added a node-based Dockerfile

bin/gitrev.sh

@@ -1,5 +1,10 @@
 #!/bin/bash
 #
+if [[ ! -z "$DRONE_COMMIT_SHA" ]]; then
+    echo "${DRONE_COMMIT_SHA:0:7}"
+    exit 0
+fi
+
 if [[ ! -z "$GITREV" ]]; then
     echo $GITREV
 else

mfer/checker.go

@@ -312,6 +312,10 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
 }
 
 func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
+	// entry.GetPath() is safe to join here: a manifest's entry paths are
+	// validated against the path invariants when it is loaded (see
+	// deserializeInner) or built (see Builder.AddFile), so a traversal or
+	// absolute path can never reach this point.
 	absPath := filepath.Join(string(c.basePath), entry.GetPath())
 	relPath := RelFilePath(entry.GetPath())
 

mfer/deserialize.go

@@ -25,6 +25,7 @@ var (
 	errDecompressedTooLarge = errors.New("decompressed data exceeds maximum allowed size")
 	errUUIDMismatch         = errors.New("outer and inner UUID mismatch")
 	errInvalidFileFormat    = errors.New("invalid file format")
+	errInvalidManifestPath  = errors.New("manifest contains invalid path")
 )
 
 // validateUUID checks that the byte slice is a valid UUID (16 bytes, parseable).
@@ -181,6 +182,19 @@ func (m *manifest) deserializeInner() error {
 		return errUUIDMismatch
 	}
 
+	// Enforce the manifest path invariants on every entry as it is loaded,
+	// so that no consumer of a manifest — Checker today, any restore or
+	// extract path tomorrow — acts on a traversal or absolute path from an
+	// untrusted .mf. Reject loudly on the first offender rather than
+	// dropping entries, which would let a hostile manifest hide files from a
+	// check.
+	for _, f := range m.pbInner.GetFiles() {
+		err = ValidatePath(f.GetPath())
+		if err != nil {
+			return fmt.Errorf("%w: %w", errInvalidManifestPath, err)
+		}
+	}
+
 	log.Infof("loaded manifest with %d files", len(m.pbInner.GetFiles()))
 
 	return nil

mfer/deserialize_path_test.go

@@ -0,0 +1,145 @@
+//nolint:testpackage // white-box tests exercise unexported internals
+package mfer
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"fmt"
+	"testing"
+
+	"github.com/google/uuid"
+	"github.com/klauspost/compress/zstd"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/protobuf/encoding/protowire"
+	"google.golang.org/protobuf/proto"
+)
+
+// craftInnerBytes builds the wire bytes of an inner MFFile holding a single
+// file entry whose path is exactly pathBytes. It writes the wire form by hand
+// so a hostile path — including one that is not valid UTF-8 — can be embedded
+// without proto.Marshal's own UTF-8 enforcement rejecting it first.
+func craftInnerBytes(id uuid.UUID, pathBytes string) []byte {
+	entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
+	entry = protowire.AppendString(entry, pathBytes)
+
+	inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
+	inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
+	inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
+	inner = protowire.AppendBytes(inner, entry)
+	inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
+	inner = protowire.AppendBytes(inner, id[:])
+
+	return inner
+}
+
+// wrapInner wraps inner MFFile wire bytes in a complete, well-formed .mf
+// envelope (magic prefix, zstd-compressed payload, matching hash and UUID) so
+// that deserialization reaches path validation rather than failing earlier on
+// an integrity check.
+func wrapInner(t *testing.T, id uuid.UUID, innerData []byte) []byte {
+	t.Helper()
+
+	var cbuf bytes.Buffer
+
+	zw, err := zstd.NewWriter(&cbuf, zstd.WithEncoderLevel(zstd.SpeedBestCompression))
+	require.NoError(t, err)
+
+	_, err = zw.Write(innerData)
+	require.NoError(t, err)
+	require.NoError(t, zw.Close())
+
+	compressed := cbuf.Bytes()
+	sum := sha256.Sum256(compressed)
+
+	outer := &MFFileOuter{
+		InnerMessage:    compressed,
+		Size:            int64(len(innerData)),
+		Sha256:          sum[:],
+		Uuid:            id[:],
+		Version:         MFFileOuter_VERSION_ONE,
+		CompressionType: MFFileOuter_COMPRESSION_ZSTD,
+	}
+
+	ob, err := proto.Marshal(outer)
+	require.NoError(t, err)
+
+	return append([]byte(MAGIC), ob...)
+}
+
+func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{"parent traversal", "../escape"},
+		{"interior traversal", "a/../../escape"},
+		{"absolute path", "/etc/passwd"},
+		{"backslash path", `a\b`},
+		{"double slash", "a//b"},
+		{"empty path", ""},
+		{"invalid utf-8", "abc\xff"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			id := uuid.New()
+			data := wrapInner(t, id, craftInnerBytes(id, tt.path))
+
+			_, err := NewManifestFromReader(bytes.NewReader(data))
+			require.Error(t, err)
+
+			if tt.path == "abc\xff" {
+				// A path that is not valid UTF-8 cannot survive the proto3
+				// string decoder, which rejects it before path validation
+				// runs; the manifest is still refused at load time.
+				return
+			}
+
+			require.ErrorIs(t, err, errInvalidManifestPath)
+
+			if tt.path != "" {
+				// ValidatePath quotes the path with %q; assert against the
+				// same rendering so escaped characters (e.g. a backslash)
+				// still match.
+				assert.Contains(t, err.Error(), fmt.Sprintf("%q", tt.path),
+					"error must name the offending path")
+			}
+		})
+	}
+}
+
+func TestDeserializeValidManifestRoundTrips(t *testing.T) {
+	t.Parallel()
+
+	hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
+
+	b := NewBuilder()
+	require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
+
+	var buf bytes.Buffer
+	require.NoError(t, b.Build(&buf))
+
+	m, err := NewManifestFromReader(bytes.NewReader(buf.Bytes()))
+	require.NoError(t, err)
+
+	files := m.Files()
+	require.Len(t, files, 1)
+	assert.Equal(t, "dir/file.txt", files[0].GetPath())
+	assert.Equal(t, int64(123), files[0].GetSize())
+}
+
+// TestValidatePathRejectsInvalidUTF8 pins the ValidatePath rule that a manifest
+// path must be valid UTF-8, independent of the proto decoder that also enforces
+// it on the wire.
+func TestValidatePathRejectsInvalidUTF8(t *testing.T) {
+	t.Parallel()
+
+	err := ValidatePath("abc\xff")
+	require.ErrorIs(t, err, errPathNotUTF8)
+	assert.Contains(t, err.Error(), "UTF-8")
+}