mfer/docs/FORMAT.md

.mf File Format Specification

Version 1.0

Overview

An .mf file is a binary manifest that describes a directory tree of files, including their paths, sizes, and cryptographic checksums. It supports optional OpenPGP signatures for integrity verification and optional timestamps and file permissions for metadata preservation.

Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is reserved or kept for later use.

File Structure

An .mf file consists of two parts, concatenated:

1. Magic bytes (8 bytes): the ASCII string ZNAVSRFG
2. Outer message: a Protocol Buffers serialized MFFileOuter message

There is no length prefix or version byte between the magic and the protobuf message. The protobuf message extends to the end of the file.

See mfer/mf.proto for exact field numbers and types.

Outer Message (MFFileOuter)

The outer message contains:

Field	Number	Type	Description
version	101	enum	Must be VERSION_ONE (1)
compressionType	102	enum	Compression of innerMessage; must be COMPRESSION_ZSTD (1)
size	103	int64	Uncompressed size of innerMessage (corruption detection)
sha256	104	bytes	SHA-256 hash of the compressed innerMessage (corruption detection)
uuid	105	bytes	Random v4 UUID; must match the inner message UUID
innerMessage	199	bytes	Zstd-compressed serialized MFFile message
signature	201	bytes (optional)	One OpenPGP signature over the canonical string, armored or binary
signer	202	bytes (optional)	Fingerprint of the signing primary key, in upper-case hex
signingPubKey	203	bytes (optional)	Full OpenPGP public key of the signing key (ASCII-armored or binary)

SHA-256 Hash

The sha256 field (104) covers the compressed innerMessage bytes. This allows verifying data integrity before decompression.

Compression

The innerMessage field is compressed with Zstandard (zstd). Implementations must enforce a decompression size limit to prevent decompression bombs. The reference implementation limits decompressed size to 256 MiB. It writes zstd frames with a window of at most 8 MiB, the largest window the zstd format recommends decoders support, and refuses frames that ask for a larger one. It also refuses an inner message whose file entries, hashes, timestamps and MIME types, counted at 176, 112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a manifest file larger than 258 MiB without reading the rest of it: zstd's worst case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room for the signature, the signing key and the other outer fields.

Inner Message (MFFile)

After decompressing innerMessage, the result is a serialized MFFile (referred to as the manifest):

Field	Number	Type	Description
version	100	enum	Must be VERSION_ONE (1)
files	101	repeated MFFilePath	List of files in the manifest
uuid	102	bytes	Random v4 UUID; must match outer UUID
createdAt	201	Timestamp (optional)	When the manifest was created

File Entries (MFFilePath)

Each file entry contains:

Field	Number	Type	Description
path	1	string	Relative file path (see Path Rules)
size	2	int64	File size in bytes
hashes	3	repeated MFFileChecksum	At least one hash required
mimeType	301	string (optional)	MIME type
mtime	302	Timestamp (optional)	Modification time
ctime	303	Timestamp (optional)	Change time (inode metadata change)
mode	304	uint32	Permission bits (see File Mode)

File Mode

mode holds a file's Unix permission bits, the nine rwx bits, so it is never above 0777 (octal); the setuid, setgid and sticky bits are never recorded. Writers record 0000 unless whoever creates the manifest asks for permissions. 0000, the proto3 default, means no mode was recorded: readers never check or apply it.

The reference implementation records modes when gen or freshen is given --include-permissions. check fails a file whose permission bits differ from a recorded mode other than 0000. fetch sets a recorded mode other than 0000 on each file it writes, and refuses a manifest that records a mode above 0777 before it requests any file.

Path Rules

All path values must satisfy these invariants:

• UTF-8: paths must be valid UTF-8
• Forward slashes: use / as the path separator (never \)
• Relative only: no leading /
• No parent traversal: no .. path segments
• No empty segments: no // sequences
• No trailing slash: paths refer to files, not directories
• Listed once: each path appears at most once in a manifest, compared byte for byte, so A.txt and a.txt are two paths

Implementations must validate these invariants when reading and writing manifests. Paths that violate these rules must be rejected, and a reader must reject a manifest that lists a path more than once.

Hash Format (MFFileChecksum)

Each checksum is a single bytes multiHash field containing a multihash-encoded value. Multihash is self-describing: the encoded bytes include a varint algorithm identifier followed by a varint digest length followed by the digest itself.

The 1.0 implementation writes SHA-256 multihashes (0x12 algorithm code). Implementations must be able to verify SHA-256 multihashes at minimum.

Signature Scheme

Signing is optional. A signature is stored only in the outer message's signature field, never in a separate file. It is one OpenPGP signature over the canonical string below, armored or binary, and does not contain the string, which a verifier builds again from the outer message.

The canonical string is three parts separated by hyphens (-):

ZNAVSRFG-<UUID>-<SHA256>

• ZNAVSRFG is the magic, in ASCII.
• <UUID> is the 16 bytes of the outer message's uuid as 32 lower-case hex digits, with no hyphens.
• <SHA256> is the lower-case hex of the outer message's sha256 encoded as a SHA-256 multihash (see Hash Format). That multihash is the algorithm code 0x12, the digest length 0x20 and the 32 bytes of the hash, so this part is 68 hex digits starting 1220.

The signature covers exactly the 110 ASCII characters of the string, with no line ending. For example, the UUID b0fb6918-6298-443b-ada2-1b162f4dc0b3 and a sha256 of e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in hex give this canonical string:

ZNAVSRFG-b0fb69186298443bada21b162f4dc0b3-1220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The signature covers innerMessage only through sha256, so a verifier checks sha256 against innerMessage (see SHA-256 Hash) as well as the signature.

The signing key's public key goes in signingPubKey and the fingerprint of its primary key, in upper-case hex, in signer. The reference implementation signs only with version 4 keys, whose fingerprints are 40 hex digits, and writes both the signature and the public key armored.

A verifier accepts a signed manifest only if signingPubKey holds exactly one primary key, signature is one good signature over the canonical string made by that key (or one of its subkeys), and signer is that primary key's fingerprint, compared without regard to case. The reference implementation refuses to load a manifest that fails these checks; check and fetch given --require-signature then compare the required fingerprint with signer. It also refuses a manifest whose signingPubKey holds a DSA key or subkey, or any secret key or subkey, since checking the self-signatures of a DSA key or the numbers of a secret key can take hours when those numbers are very large.

Deterministic Serialization

By default, manifests are generated deterministically:

• File entries are sorted by path in lexicographic byte order
• createdAt is omitted unless explicitly requested
• mode is 0000 unless explicitly requested

This ensures that two independent runs over the same directory tree produce byte-identical .mf files (assuming file contents and metadata have not changed).

MIME Type

The recommended MIME type for .mf files is application/octet-stream. The .mf file extension is the canonical identifier.

Reference

• Proto definition: mfer/mf.proto
• Reference implementation: git.eeqj.de/sneak/mfer