clawbot
24496880c5
check / check (push) Waiting to run
verifySignature accepts a signature whose key has expired, since a manifest outlives its key. The OpenPGP library stops at the expired key without checking the signature's own expiry time, so a signature made with one was refused while the key was live and accepted once the key had expired as well. verifySignature now gets the signature packet from openpgp.VerifyDetachedSignature and refuses it when it has expired. docs/FORMAT.md Signature Scheme states the rule. Model: opus-5-5
203 lines
9.9 KiB
Markdown
203 lines
9.9 KiB
Markdown
# .mf File Format Specification
|
|
|
|
Version 1.0
|
|
|
|
## Overview
|
|
|
|
An `.mf` file is a binary manifest that describes a directory tree of files,
|
|
including their paths, sizes, and cryptographic checksums. It supports optional
|
|
OpenPGP signatures for integrity verification and optional timestamps and file
|
|
permissions for metadata preservation.
|
|
|
|
Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
|
|
reserved or kept for later use.
|
|
|
|
## File Structure
|
|
|
|
An `.mf` file consists of two parts, concatenated:
|
|
|
|
1. **Magic bytes** (8 bytes): the ASCII string `ZNAVSRFG`
|
|
2. **Outer message**: a Protocol Buffers serialized `MFFileOuter` message
|
|
|
|
There is no length prefix or version byte between the magic and the protobuf
|
|
message. The protobuf message extends to the end of the file.
|
|
|
|
See [`mfer/mf.proto`](../mfer/mf.proto) for exact field numbers and types.
|
|
|
|
## Outer Message (`MFFileOuter`)
|
|
|
|
The outer message contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------------- | ------ | ---------------- | ------------------------------------------------------------------------ |
|
|
| `version` | 101 | enum | Must be `VERSION_ONE` (1) |
|
|
| `compressionType` | 102 | enum | Compression of `innerMessage`; must be `COMPRESSION_ZSTD` (1) |
|
|
| `size` | 103 | int64 | Uncompressed size of `innerMessage` (corruption detection) |
|
|
| `sha256` | 104 | bytes | SHA-256 hash of the **compressed** `innerMessage` (corruption detection) |
|
|
| `uuid` | 105 | bytes | Random v4 UUID; must match the inner message UUID |
|
|
| `innerMessage` | 199 | bytes | Zstd-compressed serialized `MFFile` message |
|
|
| `signature` | 201 | bytes (optional) | One OpenPGP signature over the canonical string, armored or binary |
|
|
| `signer` | 202 | bytes (optional) | Fingerprint of the signing primary key, in upper-case hex |
|
|
| `signingPubKey` | 203 | bytes (optional) | Full OpenPGP public key of the signing key (ASCII-armored or binary) |
|
|
|
|
### SHA-256 Hash
|
|
|
|
The `sha256` field (104) covers the **compressed** `innerMessage` bytes. This
|
|
allows verifying data integrity before decompression.
|
|
|
|
## Compression
|
|
|
|
The `innerMessage` field is compressed with
|
|
[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must
|
|
enforce a decompression size limit to prevent decompression bombs. The reference
|
|
implementation limits decompressed size to 256 MiB. It writes zstd frames with a
|
|
window of at most 8 MiB, the largest window the zstd format recommends decoders
|
|
support, and refuses frames that ask for a larger one. It also refuses an inner
|
|
message whose file entries, hashes, timestamps and MIME types, counted at 176,
|
|
112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
|
|
manifest file larger than 258 MiB without reading the rest of it: zstd's worst
|
|
case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
|
|
for the signature, the signing key and the other outer fields.
|
|
|
|
## Inner Message (`MFFile`)
|
|
|
|
After decompressing `innerMessage`, the result is a serialized `MFFile`
|
|
(referred to as the manifest):
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------- | ------ | --------------------- | ------------------------------------- |
|
|
| `version` | 100 | enum | Must be `VERSION_ONE` (1) |
|
|
| `files` | 101 | repeated `MFFilePath` | List of files in the manifest |
|
|
| `uuid` | 102 | bytes | Random v4 UUID; must match outer UUID |
|
|
| `createdAt` | 201 | Timestamp (optional) | When the manifest was created |
|
|
|
|
## File Entries (`MFFilePath`)
|
|
|
|
Each file entry contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ---------- | ------ | ------------------------- | ------------------------------------------------------------- |
|
|
| `path` | 1 | string | Relative file path (see Path Rules) |
|
|
| `size` | 2 | int64 | File size in bytes |
|
|
| `hashes` | 3 | repeated `MFFileChecksum` | At least one hash required; readers reject an entry with none |
|
|
| `mimeType` | 301 | string (optional) | MIME type |
|
|
| `mtime` | 302 | Timestamp (optional) | Modification time |
|
|
| `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) |
|
|
| `mode` | 304 | uint32 | Permission bits (see File Mode) |
|
|
|
|
## File Mode
|
|
|
|
`mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
|
|
above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
|
|
Writers record `0000` unless whoever creates the manifest asks for permissions.
|
|
`0000`, the proto3 default, means no mode was recorded: readers never check or
|
|
apply it.
|
|
|
|
The reference implementation records modes when `gen` or `freshen` is given
|
|
`--include-permissions`. `check` fails a file whose permission bits differ from
|
|
a recorded mode other than `0000`. `fetch` sets a recorded mode other than
|
|
`0000` on each file it writes, and refuses a manifest that records a mode above
|
|
`0777` before it requests any file.
|
|
|
|
## Path Rules
|
|
|
|
All `path` values must satisfy these invariants:
|
|
|
|
- **UTF-8**: paths must be valid UTF-8
|
|
- **Forward slashes**: use `/` as the path separator (never `\`)
|
|
- **Relative only**: no leading `/`
|
|
- **No parent traversal**: no `..` path segments
|
|
- **No empty segments**: no `//` sequences
|
|
- **No trailing slash**: paths refer to files, not directories
|
|
- **Listed once**: each path appears at most once in a manifest, compared byte
|
|
for byte, so `A.txt` and `a.txt` are two paths
|
|
|
|
Implementations must validate these invariants when reading and writing
|
|
manifests. Paths that violate these rules must be rejected, and a reader must
|
|
reject a manifest that lists a path more than once.
|
|
|
|
## Hash Format (`MFFileChecksum`)
|
|
|
|
Each checksum is a single `bytes multiHash` field containing a
|
|
[multihash](https://multiformats.io/multihash/)-encoded value. Multihash is
|
|
self-describing: the encoded bytes include a varint algorithm identifier
|
|
followed by a varint digest length followed by the digest itself.
|
|
|
|
The 1.0 implementation writes SHA-256 multihashes (`0x12` algorithm code).
|
|
Implementations must be able to verify SHA-256 multihashes at minimum.
|
|
|
|
## Signature Scheme
|
|
|
|
Signing is optional. A signature is stored only in the outer message's
|
|
`signature` field, never in a separate file. It is one OpenPGP signature over
|
|
the canonical string below, armored or binary, and does not contain the string,
|
|
which a verifier builds again from the outer message.
|
|
|
|
The canonical string is three parts separated by hyphens (`-`):
|
|
|
|
```
|
|
ZNAVSRFG-<UUID>-<SHA256>
|
|
```
|
|
|
|
- `ZNAVSRFG` is the magic, in ASCII.
|
|
- `<UUID>` is the 16 bytes of the outer message's `uuid` as 32 lower-case hex
|
|
digits, with no hyphens.
|
|
- `<SHA256>` is the lower-case hex of the outer message's `sha256` encoded as a
|
|
SHA-256 multihash (see Hash Format). That multihash is the algorithm code
|
|
`0x12`, the digest length `0x20` and the 32 bytes of the hash, so this part is
|
|
68 hex digits starting `1220`.
|
|
|
|
The signature covers exactly the 110 ASCII characters of the string, with no
|
|
line ending. For example, the UUID `b0fb6918-6298-443b-ada2-1b162f4dc0b3` and a
|
|
`sha256` of `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`
|
|
in hex give this canonical string:
|
|
|
|
```
|
|
ZNAVSRFG-b0fb69186298443bada21b162f4dc0b3-1220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
|
|
```
|
|
|
|
The signature covers `innerMessage` only through `sha256`, so a verifier checks
|
|
`sha256` against `innerMessage` (see SHA-256 Hash) as well as the signature.
|
|
|
|
The signing key's public key goes in `signingPubKey` and the fingerprint of its
|
|
primary key, in upper-case hex, in `signer`. The reference implementation signs
|
|
only with version 4 keys, whose fingerprints are 40 hex digits, and writes both
|
|
the signature and the public key armored.
|
|
|
|
A verifier accepts a signed manifest only if `signingPubKey` holds exactly one
|
|
primary key, `signature` is one good signature over the canonical string made by
|
|
that key (or one of its subkeys), and `signer` is that primary key's
|
|
fingerprint, compared without regard to case. The reference implementation
|
|
refuses to load a manifest that fails these checks; `check` and `fetch` given
|
|
`--require-signature` then compare the required fingerprint with `signer`. It
|
|
also refuses a manifest whose `signingPubKey` holds a DSA key or subkey, or any
|
|
secret key or subkey, since checking the self-signatures of a DSA key or the
|
|
numbers of a secret key can take hours when those numbers are very large.
|
|
|
|
A signing key that has expired does not make the signature bad, because a
|
|
manifest outlives its key. A signature past its own expiry time is not good,
|
|
whether or not the key has expired too.
|
|
|
|
## Deterministic Serialization
|
|
|
|
By default, manifests are generated deterministically:
|
|
|
|
- File entries are sorted by `path` in **lexicographic byte order**
|
|
- `createdAt` is omitted unless explicitly requested
|
|
- `mode` is `0000` unless explicitly requested
|
|
|
|
This ensures that two independent runs over the same directory tree produce
|
|
byte-identical `.mf` files (assuming file contents and metadata have not
|
|
changed).
|
|
|
|
## MIME Type
|
|
|
|
The recommended MIME type for `.mf` files is `application/octet-stream`. The
|
|
`.mf` file extension is the canonical identifier.
|
|
|
|
## Reference
|
|
|
|
- Proto definition: [`mfer/mf.proto`](../mfer/mf.proto)
|
|
- Reference implementation:
|
|
[git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
|