sneak
d08867ae56
check / check (push) Waiting to run
The Signature Scheme section now states what signatureString builds: the magic, the outer uuid as lower-case hex with no hyphens, and the outer sha256 as the lower-case hex of its multihash, which starts 1220, with an example string taken from a run of the code. It says signer is the upper-case hex fingerprint of the signing primary key and that a signature is stored only in the manifest. The mf.proto comments on fields 201 to 203 say the same, and mf.pb.go is regenerated. A new test pins the example string. Nothing about what is signed or how changes. Model: opus-5-5
199 lines
9.5 KiB
Markdown
199 lines
9.5 KiB
Markdown
# .mf File Format Specification
|
|
|
|
Version 1.0
|
|
|
|
## Overview
|
|
|
|
An `.mf` file is a binary manifest that describes a directory tree of files,
|
|
including their paths, sizes, and cryptographic checksums. It supports optional
|
|
OpenPGP signatures for integrity verification and optional timestamps and file
|
|
permissions for metadata preservation.
|
|
|
|
Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
|
|
reserved or kept for later use.
|
|
|
|
## File Structure
|
|
|
|
An `.mf` file consists of two parts, concatenated:
|
|
|
|
1. **Magic bytes** (8 bytes): the ASCII string `ZNAVSRFG`
|
|
2. **Outer message**: a Protocol Buffers serialized `MFFileOuter` message
|
|
|
|
There is no length prefix or version byte between the magic and the protobuf
|
|
message. The protobuf message extends to the end of the file.
|
|
|
|
See [`mfer/mf.proto`](../mfer/mf.proto) for exact field numbers and types.
|
|
|
|
## Outer Message (`MFFileOuter`)
|
|
|
|
The outer message contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------------- | ------ | ---------------- | ------------------------------------------------------------------------ |
|
|
| `version` | 101 | enum | Must be `VERSION_ONE` (1) |
|
|
| `compressionType` | 102 | enum | Compression of `innerMessage`; must be `COMPRESSION_ZSTD` (1) |
|
|
| `size` | 103 | int64 | Uncompressed size of `innerMessage` (corruption detection) |
|
|
| `sha256` | 104 | bytes | SHA-256 hash of the **compressed** `innerMessage` (corruption detection) |
|
|
| `uuid` | 105 | bytes | Random v4 UUID; must match the inner message UUID |
|
|
| `innerMessage` | 199 | bytes | Zstd-compressed serialized `MFFile` message |
|
|
| `signature` | 201 | bytes (optional) | One OpenPGP signature over the canonical string, armored or binary |
|
|
| `signer` | 202 | bytes (optional) | Fingerprint of the signing primary key, in upper-case hex |
|
|
| `signingPubKey` | 203 | bytes (optional) | Full OpenPGP public key of the signing key (ASCII-armored or binary) |
|
|
|
|
### SHA-256 Hash
|
|
|
|
The `sha256` field (104) covers the **compressed** `innerMessage` bytes. This
|
|
allows verifying data integrity before decompression.
|
|
|
|
## Compression
|
|
|
|
The `innerMessage` field is compressed with
|
|
[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must
|
|
enforce a decompression size limit to prevent decompression bombs. The reference
|
|
implementation limits decompressed size to 256 MiB. It writes zstd frames with a
|
|
window of at most 8 MiB, the largest window the zstd format recommends decoders
|
|
support, and refuses frames that ask for a larger one. It also refuses an inner
|
|
message whose file entries, hashes, timestamps and MIME types, counted at 176,
|
|
112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
|
|
manifest file larger than 258 MiB without reading the rest of it: zstd's worst
|
|
case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
|
|
for the signature, the signing key and the other outer fields.
|
|
|
|
## Inner Message (`MFFile`)
|
|
|
|
After decompressing `innerMessage`, the result is a serialized `MFFile`
|
|
(referred to as the manifest):
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------- | ------ | --------------------- | ------------------------------------- |
|
|
| `version` | 100 | enum | Must be `VERSION_ONE` (1) |
|
|
| `files` | 101 | repeated `MFFilePath` | List of files in the manifest |
|
|
| `uuid` | 102 | bytes | Random v4 UUID; must match outer UUID |
|
|
| `createdAt` | 201 | Timestamp (optional) | When the manifest was created |
|
|
|
|
## File Entries (`MFFilePath`)
|
|
|
|
Each file entry contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ---------- | ------ | ------------------------- | ----------------------------------- |
|
|
| `path` | 1 | string | Relative file path (see Path Rules) |
|
|
| `size` | 2 | int64 | File size in bytes |
|
|
| `hashes` | 3 | repeated `MFFileChecksum` | At least one hash required |
|
|
| `mimeType` | 301 | string (optional) | MIME type |
|
|
| `mtime` | 302 | Timestamp (optional) | Modification time |
|
|
| `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) |
|
|
| `mode` | 304 | uint32 | Permission bits (see File Mode) |
|
|
|
|
## File Mode
|
|
|
|
`mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
|
|
above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
|
|
Writers record `0000` unless whoever creates the manifest asks for permissions.
|
|
`0000`, the proto3 default, means no mode was recorded: readers never check or
|
|
apply it.
|
|
|
|
The reference implementation records modes when `gen` or `freshen` is given
|
|
`--include-permissions`. `check` fails a file whose permission bits differ from
|
|
a recorded mode other than `0000`. `fetch` sets a recorded mode other than
|
|
`0000` on each file it writes, and refuses a manifest that records a mode above
|
|
`0777` before it requests any file.
|
|
|
|
## Path Rules
|
|
|
|
All `path` values must satisfy these invariants:
|
|
|
|
- **UTF-8**: paths must be valid UTF-8
|
|
- **Forward slashes**: use `/` as the path separator (never `\`)
|
|
- **Relative only**: no leading `/`
|
|
- **No parent traversal**: no `..` path segments
|
|
- **No empty segments**: no `//` sequences
|
|
- **No trailing slash**: paths refer to files, not directories
|
|
- **Listed once**: each path appears at most once in a manifest, compared byte
|
|
for byte, so `A.txt` and `a.txt` are two paths
|
|
|
|
Implementations must validate these invariants when reading and writing
|
|
manifests. Paths that violate these rules must be rejected, and a reader must
|
|
reject a manifest that lists a path more than once.
|
|
|
|
## Hash Format (`MFFileChecksum`)
|
|
|
|
Each checksum is a single `bytes multiHash` field containing a
|
|
[multihash](https://multiformats.io/multihash/)-encoded value. Multihash is
|
|
self-describing: the encoded bytes include a varint algorithm identifier
|
|
followed by a varint digest length followed by the digest itself.
|
|
|
|
The 1.0 implementation writes SHA-256 multihashes (`0x12` algorithm code).
|
|
Implementations must be able to verify SHA-256 multihashes at minimum.
|
|
|
|
## Signature Scheme
|
|
|
|
Signing is optional. A signature is stored only in the outer message's
|
|
`signature` field, never in a separate file. It is one OpenPGP signature over
|
|
the canonical string below, armored or binary, and does not contain the string,
|
|
which a verifier builds again from the outer message.
|
|
|
|
The canonical string is three parts separated by hyphens (`-`):
|
|
|
|
```
|
|
ZNAVSRFG-<UUID>-<SHA256>
|
|
```
|
|
|
|
- `ZNAVSRFG` is the magic, in ASCII.
|
|
- `<UUID>` is the 16 bytes of the outer message's `uuid` as 32 lower-case hex
|
|
digits, with no hyphens.
|
|
- `<SHA256>` is the lower-case hex of the outer message's `sha256` encoded as a
|
|
SHA-256 multihash (see Hash Format). That multihash is the algorithm code
|
|
`0x12`, the digest length `0x20` and the 32 bytes of the hash, so this part is
|
|
68 hex digits starting `1220`.
|
|
|
|
The signature covers exactly the 110 ASCII characters of the string, with no
|
|
line ending. For example, the UUID `b0fb6918-6298-443b-ada2-1b162f4dc0b3` and a
|
|
`sha256` of `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
|
|
in hex give this canonical string:
|
|
|
|
```
|
|
ZNAVSRFG-b0fb69186298443bada21b162f4dc0b3-1220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
|
|
```
|
|
|
|
The signature covers `innerMessage` only through `sha256`, so a verifier checks
|
|
`sha256` against `innerMessage` (see SHA-256 Hash) as well as the signature.
|
|
|
|
The signing key's public key goes in `signingPubKey` and the fingerprint of its
|
|
primary key, in upper-case hex, in `signer`. The reference implementation signs
|
|
only with version 4 keys, whose fingerprints are 40 hex digits, and writes both
|
|
the signature and the public key armored.
|
|
|
|
A verifier accepts a signed manifest only if `signingPubKey` holds exactly one
|
|
primary key, `signature` is one good signature over the canonical string made by
|
|
that key (or one of its subkeys), and `signer` is that primary key's
|
|
fingerprint, compared without regard to case. The reference implementation
|
|
refuses to load a manifest that fails these checks; `check` and `fetch` given
|
|
`--require-signature` then compare the required fingerprint with `signer`. It
|
|
also refuses a manifest whose `signingPubKey` holds a DSA key or subkey, or any
|
|
secret key or subkey, since checking the self-signatures of a DSA key or the
|
|
numbers of a secret key can take hours when those numbers are very large.
|
|
|
|
## Deterministic Serialization
|
|
|
|
By default, manifests are generated deterministically:
|
|
|
|
- File entries are sorted by `path` in **lexicographic byte order**
|
|
- `createdAt` is omitted unless explicitly requested
|
|
- `mode` is `0000` unless explicitly requested
|
|
|
|
This ensures that two independent runs over the same directory tree produce
|
|
byte-identical `.mf` files (assuming file contents and metadata have not
|
|
changed).
|
|
|
|
## MIME Type
|
|
|
|
The recommended MIME type for `.mf` files is `application/octet-stream`. The
|
|
`.mf` file extension is the canonical identifier.
|
|
|
|
## Reference
|
|
|
|
- Proto definition: [`mfer/mf.proto`](../mfer/mf.proto)
|
|
- Reference implementation:
|
|
[git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
|