mfer/docs/FORMAT.md

.mf File Format Specification

Version 1.0

Overview

An .mf file is a binary manifest that describes a directory tree of files, including their paths, sizes, and cryptographic checksums. It supports optional OpenPGP signatures for integrity verification and optional timestamps and file permissions for metadata preservation.

Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is reserved or kept for later use.

File Structure

An .mf file consists of two parts, concatenated:

1. Magic bytes (8 bytes): the ASCII string ZNAVSRFG
2. Outer message: a Protocol Buffers serialized MFFileOuter message

There is no length prefix or version byte between the magic and the protobuf message. The protobuf message extends to the end of the file, so a .mf is always a whole file; a different framing, such as one with a length prefix, would need a different magic. The format's version is the outer message's version field.

See mfer/mf.proto for exact field numbers and types. Readers ignore any field they do not know, in any message, so a later version can add a field without breaking 1.0 readers.

Outer Message (MFFileOuter)

The outer message contains:

Field	Number	Type	Description
version	101	enum	Must be VERSION_ONE (1)
compressionType	102	enum	Compression of innerMessage; must be COMPRESSION_ZSTD (1)
size	103	int64	Uncompressed size of innerMessage (corruption detection)
sha256	104	bytes	SHA-256 hash of the compressed innerMessage (corruption detection)
uuid	105	bytes	16-byte UUID; must match the inner message UUID
innerMessage	199	bytes	Zstd-compressed serialized MFFile message
signature	201	bytes (optional)	One OpenPGP signature over the canonical string, armored or binary
signer	202	bytes (optional)	Fingerprint of the signing primary key, in upper-case hex
signingPubKey	203	bytes (optional)	Full OpenPGP public key of the signing key (ASCII-armored or binary)

SHA-256 Hash

The sha256 field (104) covers the compressed innerMessage bytes. This allows verifying data integrity before decompression.

Compression

The innerMessage field is compressed with Zstandard (zstd). Implementations must enforce a decompression size limit to prevent decompression bombs. The reference implementation limits decompressed size to 256 MiB. It writes zstd frames with a window of at most 8 MiB, the largest window the zstd format recommends decoders support, and refuses frames that ask for a larger one. It also refuses an inner message whose file entries, hashes, timestamps and MIME types, counted at 176, 112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a manifest file larger than 258 MiB without reading the rest of it: zstd's worst case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room for the signature, the signing key and the other outer fields.

Inner Message (MFFile)

After decompressing innerMessage, the result is a serialized MFFile (referred to as the manifest):

Field	Number	Type	Description
version	100	enum	Must be VERSION_ONE (1)
files	101	repeated MFFilePath	List of files in the manifest
uuid	102	bytes	16-byte UUID; must match outer UUID
createdAt	201	Timestamp (optional)	When the manifest was created

File Entries (MFFilePath)

Each file entry contains:

Field	Number	Type	Description
path	1	string	Relative file path (see Path Rules)
size	2	int64	File size in bytes
hashes	3	repeated MFFileChecksum	At least one hash required; readers reject an entry with none
mimeType	301	string (optional)	MIME type
mtime	302	Timestamp (optional)	Modification time
ctime	303	Timestamp (optional)	Change time (inode metadata change)
mode	304	uint32	Permission bits (see File Mode)

File Mode

mode holds a file's Unix permission bits, the nine rwx bits, so it is never above 0777 (octal); the setuid, setgid and sticky bits are never recorded. Writers record 0000 unless whoever creates the manifest asks for permissions. 0000, the proto3 default, means no mode was recorded: readers never check or apply it.

The reference implementation records modes when gen or freshen is given --include-permissions. check fails a file whose permission bits differ from a recorded mode other than 0000. fetch sets a recorded mode other than 0000 on each file it writes, and refuses a manifest that records a mode above 0777 before it requests any file.

Path Rules

All path values must satisfy these invariants:

• UTF-8: paths must be valid UTF-8
• Forward slashes: use / as the path separator (never \)
• Relative only: no leading /
• No parent traversal: no .. path segments
• No empty segments: no // sequences
• No trailing slash: paths refer to files, not directories
• Listed once: each path appears at most once in a manifest, compared byte for byte, so A.txt and a.txt are two paths

Implementations must validate these invariants when reading and writing manifests. Paths that violate these rules must be rejected, and a reader must reject a manifest that lists a path more than once.

A path holds each name exactly as the file system gives it, with no Unicode normalization, and readers compare paths byte for byte. So é stored as one code point and é stored as e followed by a combining accent are two paths.

Hash Format (MFFileChecksum)

Each checksum is a single bytes multiHash field containing a multihash-encoded value. Multihash is self-describing: the encoded bytes include a varint algorithm identifier followed by a varint digest length followed by the digest itself.

The 1.0 implementation writes SHA-256 multihashes (0x12 algorithm code). Implementations must be able to verify SHA-256 multihashes at minimum.

Signature Scheme

Signing is optional. A signature is stored only in the outer message's signature field, never in a separate file. It is one OpenPGP signature over the canonical string below, armored or binary, and does not contain the string, which a verifier builds again from the outer message.

The canonical string is three parts separated by hyphens (-):

ZNAVSRFG-<UUID>-<SHA256>

• ZNAVSRFG is the magic, in ASCII.
• <UUID> is the 16 bytes of the outer message's uuid as 32 lower-case hex digits, with no hyphens.
• <SHA256> is the lower-case hex of the outer message's sha256 encoded as a SHA-256 multihash (see Hash Format). That multihash is the algorithm code 0x12, the digest length 0x20 and the 32 bytes of the hash, so this part is 68 hex digits starting 1220.

The signature covers exactly the 110 ASCII characters of the string, with no line ending. For example, the UUID b0fb6918-6298-443b-ada2-1b162f4dc0b3 and a sha256 of e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 in hex give this canonical string:

ZNAVSRFG-b0fb69186298443bada21b162f4dc0b3-1220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

The signature covers innerMessage only through sha256, so a verifier checks sha256 against innerMessage (see SHA-256 Hash) as well as the signature.

The signing key's public key goes in signingPubKey and the fingerprint of its primary key, in upper-case hex, in signer. The reference implementation signs only with version 4 keys, whose fingerprints are 40 hex digits, and writes both the signature and the public key armored.

A verifier accepts a signed manifest only if signingPubKey holds exactly one primary key, signature is one good signature over the canonical string made by that key (or one of its subkeys), and signer is that primary key's fingerprint, compared without regard to case. The reference implementation refuses to load a manifest that fails these checks; check and fetch given --require-signature then compare the required fingerprint with signer. It also refuses a manifest whose signingPubKey holds a DSA key or subkey, or any secret key or subkey, since checking the self-signatures of a DSA key or the numbers of a secret key can take hours when those numbers are very large.

A signing key that has expired does not make the signature bad, because a manifest outlives its key. A signature past its own expiry time is not good, whether or not the key has expired too.

Deterministic Serialization

The reference implementation writes the same inner message for the same files, UUID and options, and compresses the same inner message to the same bytes:

• File entries are sorted by path in byte order, so B.txt comes before a.txt.
• Each message is encoded by the Go protobuf library with deterministic marshaling, which writes fields in field-number order and leaves out the fields that are not set. A field not declared optional counts as not set when it holds its default value, zero or empty.
• Each entry's mtime is the file's modification time as the file system gives it, to the nanosecond. createdAt is left out unless gen or freshen is given --include-timestamps, and mode is 0000 unless it is given --include-permissions.
• The inner message is compressed into one zstd frame, with a checksum of its content, by the encoder of the Go library github.com/klauspost/compress at its SpeedBestCompression level and otherwise its default settings. That encoder gives the same bytes for the same input only within one version of the library. Another zstd encoder gives other bytes for the same inner message, and so another sha256.
• gen and freshen choose a new random version 4 UUID on every run, except that gen given --seed takes the first 16 bytes of the SHA-256 of the seed string instead. Those bytes are not a version 4 UUID; readers accept any 16 bytes.

So two runs of gen over an unchanged tree write byte-identical files only when both are given the same --seed and neither signs the manifest: a signature records the time it was made, and the OpenPGP library adds a random salt to each one.

Test Vector

mfer/testdata/test-vector.mf is a signed manifest the reference implementation made once, with the UUID 92197fd5-32bc-4090-8aa9-f86e5518b56c and an Ed25519 key that the OpenPGP library made for this file and nothing else; its secret key was not kept. TestFormatTestVector in mfer/deserialize_test.go loads the file, verifies its signature and checks the signed string and signer shown here.

A reader should load the file and accept its signature. A writer given the same files and UUID should give the same inner message; its compressed bytes, and so its sha256 and signature, depend on its zstd encoder (see Deterministic Serialization).

The manifest has no createdAt and lists two files, each with mode 0000 and the modification time 2026-10-13 12:00:00.5 UTC (seconds 1791892800, nanos 500000000):

path	size	Content
dir/world.txt	6	world and a line feed
hello.txt	6	hello and a line feed

The inner message, 164 bytes before compression:

00000000  a0 06 01 aa 06 46 0a 0d  64 69 72 2f 77 6f 72 6c  |.....F..dir/worl|
00000010  64 2e 74 78 74 10 06 1a  24 0a 22 12 20 e2 58 d2  |d.txt...$.". .X.|
00000020  48 fd a9 4c 63 75 36 07  f7 c4 49 4e e0 fc be 92  |H..Lcu6...IN....|
00000030  f1 a7 6b fd ac 79 5c 9d  84 10 1e b3 17 f2 12 0c  |..k..y\.........|
00000040  08 c0 ba b8 d6 06 10 80  ca b5 ee 01 aa 06 42 0a  |..............B.|
00000050  09 68 65 6c 6c 6f 2e 74  78 74 10 06 1a 24 0a 22  |.hello.txt...$."|
00000060  12 20 58 91 b5 b5 22 d5  df 08 6d 0f f0 b1 10 fb  |. X..."...m.....|
00000070  d9 d2 1b b4 fc 71 63 af  34 d0 82 86 a2 e8 46 f6  |.....qc.4.....F.|
00000080  be 03 f2 12 0c 08 c0 ba  b8 d6 06 10 80 ca b5 ee  |................|
00000090  01 b2 06 10 92 19 7f d5  32 bc 40 90 8a a9 f8 6e  |........2.@....n|
000000a0  55 18 b5 6c                                       |U..l|

Its compressed form is hashed into sha256:

2353553205f036a88cefee3d0d9049ccd8e68bb24011a0b462b5867f1fcc6352

So the signed string is:

ZNAVSRFG-92197fd532bc40908aa9f86e5518b56c-12202353553205f036a88cefee3d0d9049ccd8e68bb24011a0b462b5867f1fcc6352

signer is 5E962A37590D855B467D20294947A60319611E6F, and signature holds:

-----BEGIN PGP SIGNATURE-----

wqsEABYIAF0FgmrOejEJEElHpgMZYR5vNRQAAAAAABwAEHNhbHRAbm90YXRpb25z
Lm9wZW5wZ3Bqcy5vcmddxfykLLTqmTD7G/jHGzY+FiEEXpYqN1kNhVtGfSApSUem
AxlhHm8AAH0hAQDI6oGxvk1sGqIcu3Abvs7DM/7EalxkND614N75FCg+ywEApP6L
oIvrZUgEw8DUb9JeK7TChOi6PUAHUFY75Fp6hAE=
=nfX5
-----END PGP SIGNATURE-----

The whole file, 1056 bytes:

00000000  5a 4e 41 56 53 52 46 47  a8 06 01 b0 06 01 b8 06  |ZNAVSRFG........|
00000010  a4 01 c2 06 20 23 53 55  32 05 f0 36 a8 8c ef ee  |.... #SU2..6....|
00000020  3d 0d 90 49 cc d8 e6 8b  b2 40 11 a0 b4 62 b5 86  |=..I.....@...b..|
00000030  7f 1f cc 63 52 ca 06 10  92 19 7f d5 32 bc 40 90  |...cR.......2.@.|
00000040  8a a9 f8 6e 55 18 b5 6c  ba 0c a1 01 28 b5 2f fd  |...nU..l....(./.|
00000050  04 00 a5 04 00 94 08 a0  06 01 aa 06 46 0a 0d 64  |............F..d|
00000060  69 72 2f 77 6f 72 6c 64  2e 74 78 74 10 06 1a 24  |ir/world.txt...$|
00000070  0a 22 12 20 e2 58 d2 48  fd a9 4c 63 75 36 07 f7  |.". .X.H..Lcu6..|
00000080  c4 49 4e e0 fc be 92 f1  a7 6b fd ac 79 5c 9d 84  |.IN......k..y\..|
00000090  10 1e b3 17 f2 12 0c 08  c0 ba b8 d6 06 10 80 ca  |................|
000000a0  b5 ee 01 aa 06 42 0a 09  68 65 6c 6c 6f 58 91 b5  |.....B..helloX..|
000000b0  b5 22 d5 df 08 6d 0f f0  b1 10 fb d9 d2 1b b4 fc  |."...m..........|
000000c0  71 63 af 34 d0 82 86 a2  e8 46 f6 be 03 b2 06 10  |qc.4.....F......|
000000d0  92 19 7f d5 32 bc 40 90  8a a9 f8 6e 55 18 b5 6c  |....2.@....nU..l|
000000e0  02 10 06 40 ba b7 10 ce  0c f8 c2 24 13 ca 0c ac  |...@.......$....|
000000f0  02 2d 2d 2d 2d 2d 42 45  47 49 4e 20 50 47 50 20  |.-----BEGIN PGP |
00000100  53 49 47 4e 41 54 55 52  45 2d 2d 2d 2d 2d 0a 0a  |SIGNATURE-----..|
00000110  77 71 73 45 41 42 59 49  41 46 30 46 67 6d 72 4f  |wqsEABYIAF0FgmrO|
00000120  65 6a 45 4a 45 45 6c 48  70 67 4d 5a 59 52 35 76  |ejEJEElHpgMZYR5v|
00000130  4e 52 51 41 41 41 41 41  41 42 77 41 45 48 4e 68  |NRQAAAAAABwAEHNh|
00000140  62 48 52 41 62 6d 39 30  59 58 52 70 62 32 35 7a  |bHRAbm90YXRpb25z|
00000150  0a 4c 6d 39 77 5a 57 35  77 5a 33 42 71 63 79 35  |.Lm9wZW5wZ3Bqcy5|
00000160  76 63 6d 64 64 78 66 79  6b 4c 4c 54 71 6d 54 44  |vcmddxfykLLTqmTD|
00000170  37 47 2f 6a 48 47 7a 59  2b 46 69 45 45 58 70 59  |7G/jHGzY+FiEEXpY|
00000180  71 4e 31 6b 4e 68 56 74  47 66 53 41 70 53 55 65  |qN1kNhVtGfSApSUe|
00000190  6d 0a 41 78 6c 68 48 6d  38 41 41 48 30 68 41 51  |m.AxlhHm8AAH0hAQ|
000001a0  44 49 36 6f 47 78 76 6b  31 73 47 71 49 63 75 33  |DI6oGxvk1sGqIcu3|
000001b0  41 62 76 73 37 44 4d 2f  37 45 61 6c 78 6b 4e 44  |Abvs7DM/7EalxkND|
000001c0  36 31 34 4e 37 35 46 43  67 2b 79 77 45 41 70 50  |614N75FCg+ywEApP|
000001d0  36 4c 0a 6f 49 76 72 5a  55 67 45 77 38 44 55 62  |6L.oIvrZUgEw8DUb|
000001e0  39 4a 65 4b 37 54 43 68  4f 69 36 50 55 41 48 55  |9JeK7TChOi6PUAHU|
000001f0  46 59 37 35 46 70 36 68  41 45 3d 0a 3d 6e 66 58  |FY75Fp6hAE=.=nfX|
00000200  35 0a 2d 2d 2d 2d 2d 45  4e 44 20 50 47 50 20 53  |5.-----END PGP S|
00000210  49 47 4e 41 54 55 52 45  2d 2d 2d 2d 2d d2 0c 28  |IGNATURE-----..(|
00000220  35 45 39 36 32 41 33 37  35 39 30 44 38 35 35 42  |5E962A37590D855B|
00000230  34 36 37 44 32 30 32 39  34 39 34 37 41 36 30 33  |467D20294947A603|
00000240  31 39 36 31 31 45 36 46  da 0c d4 03 2d 2d 2d 2d  |19611E6F....----|
00000250  2d 42 45 47 49 4e 20 50  47 50 20 50 55 42 4c 49  |-BEGIN PGP PUBLI|
00000260  43 20 4b 45 59 20 42 4c  4f 43 4b 2d 2d 2d 2d 2d  |C KEY BLOCK-----|
00000270  0a 0a 78 6a 4d 45 61 73  31 30 67 42 59 4a 4b 77  |..xjMEas10gBYJKw|
00000280  59 42 42 41 48 61 52 77  38 42 41 51 64 41 70 71  |YBBAHaRw8BAQdApq|
00000290  33 32 53 79 4f 72 4e 75  76 31 4d 71 53 55 66 45  |32SyOrNuv1MqSUfE|
000002a0  44 67 74 42 7a 6e 4b 59  32 39 38 2f 30 47 46 55  |DgtBznKY298/0GFU|
000002b0  76 6b 0a 6c 6f 42 71 35  69 6e 4e 4b 47 31 6d 5a  |vk.loBq5inNKG1mZ|
000002c0  58 49 67 64 47 56 7a 64  43 42 32 5a 57 4e 30 62  |XIgdGVzdCB2ZWN0b|
000002d0  33 49 67 50 48 52 6c 63  33 51 74 64 6d 56 6a 64  |3IgPHRlc3QtdmVjd|
000002e0  47 39 79 51 47 31 6d 5a  58 49 75 64 47 56 7a 64  |G9yQG1mZXIudGVzd|
000002f0  44 37 43 0a 76 51 51 54  46 67 67 41 62 77 57 43  |D7C.vQQTFggAbwWC|
00000300  61 73 31 30 67 41 49 4c  42 77 6b 51 53 55 65 6d  |as10gAILBwkQSUem|
00000310  41 78 6c 68 48 6d 38 31  46 41 41 41 41 41 41 41  |AxlhHm81FAAAAAAA|
00000320  48 41 41 51 63 32 46 73  64 45 42 75 62 33 52 68  |HAAQc2FsdEBub3Rh|
00000330  64 47 6c 76 0a 62 6e 4d  75 62 33 42 6c 62 6e 42  |dGlv.bnMub3BlbnB|
00000340  6e 63 47 70 7a 4c 6d 39  79 5a 30 79 62 6c 73 34  |ncGpzLm9yZ0ybls4|
00000350  71 53 43 6d 77 4a 64 45  49 45 6d 4c 72 54 55 45  |qSCmwJdEIEmLrTUE|
00000360  43 46 51 67 43 46 67 41  43 47 51 45 43 6d 77 4d  |CFQgCFgACGQECmwM|
00000370  43 48 67 45 57 0a 49 51  52 65 6c 69 6f 33 57 51  |CHgEW.IQRelio3WQ|
00000380  32 46 57 30 5a 39 49 43  6c 4a 52 36 59 44 47 57  |2FW0Z9IClJR6YDGW|
00000390  45 65 62 77 41 41 31 67  73 41 2f 32 61 2f 50 58  |EebwAA1gsA/2a/PX|
000003a0  68 6c 44 49 4b 78 67 32  79 4e 30 4c 58 39 48 68  |hlDIKxg2yN0LX9Hh|
000003b0  49 44 43 54 7a 69 0a 49  32 43 53 71 44 74 2b 34  |IDCTzi.I2CSqDt+4|
000003c0  62 2f 73 7a 6d 59 64 41  51 43 66 2b 63 6a 6b 4f  |b/szmYdAQCf+cjkO|
000003d0  71 6c 34 4c 47 5a 41 43  6c 2f 73 2f 2f 6d 31 71  |ql4LGZACl/s//m1q|
000003e0  62 74 6e 2f 4a 31 46 45  53 52 56 73 4e 68 65 2f  |btn/J1FESRVsNhe/|
000003f0  76 6d 69 41 67 3d 3d 0a  3d 56 6b 4f 33 0a 2d 2d  |vmiAg==.=VkO3.--|
00000400  2d 2d 2d 45 4e 44 20 50  47 50 20 50 55 42 4c 49  |---END PGP PUBLI|
00000410  43 20 4b 45 59 20 42 4c  4f 43 4b 2d 2d 2d 2d 2d  |C KEY BLOCK-----|

MIME Type

The recommended MIME type for .mf files is application/octet-stream. The .mf file extension is the canonical identifier.

Reference

• Proto definition: mfer/mf.proto
• Reference implementation: git.eeqj.de/sneak/mfer