clawbot
ae0599c0cb
check / check (push) Waiting to run
gen given arguments whose files share a path, such as gen a b with a.txt in both, or gen . ., now fails while listing the files, before hashing any, naming the path and both files. Builder.AddFile and Builder.AddFileWithHash refuse a path already added. Loading refuses a manifest that lists a path twice, compared byte for byte; fetch keeps its own letter-case check. The Path Rules in docs/FORMAT.md say each path appears at most once. The decode-size test listed one path 1000 times; each entry now has its own path of the same length. Model: opus-5-5
179 lines
8.2 KiB
Markdown
179 lines
8.2 KiB
Markdown
# .mf File Format Specification
|
|
|
|
Version 1.0
|
|
|
|
## Overview
|
|
|
|
An `.mf` file is a binary manifest that describes a directory tree of files,
|
|
including their paths, sizes, and cryptographic checksums. It supports optional
|
|
GPG signatures for integrity verification and optional timestamps and file
|
|
permissions for metadata preservation.
|
|
|
|
Nothing goes in the 1.0 manifest that 1.0 does not read or write: no field is
|
|
reserved or kept for later use.
|
|
|
|
## File Structure
|
|
|
|
An `.mf` file consists of two parts, concatenated:
|
|
|
|
1. **Magic bytes** (8 bytes): the ASCII string `ZNAVSRFG`
|
|
2. **Outer message**: a Protocol Buffers serialized `MFFileOuter` message
|
|
|
|
There is no length prefix or version byte between the magic and the protobuf
|
|
message. The protobuf message extends to the end of the file.
|
|
|
|
See [`mfer/mf.proto`](../mfer/mf.proto) for exact field numbers and types.
|
|
|
|
## Outer Message (`MFFileOuter`)
|
|
|
|
The outer message contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------------- | ------ | ---------------- | ------------------------------------------------------------------------ |
|
|
| `version` | 101 | enum | Must be `VERSION_ONE` (1) |
|
|
| `compressionType` | 102 | enum | Compression of `innerMessage`; must be `COMPRESSION_ZSTD` (1) |
|
|
| `size` | 103 | int64 | Uncompressed size of `innerMessage` (corruption detection) |
|
|
| `sha256` | 104 | bytes | SHA-256 hash of the **compressed** `innerMessage` (corruption detection) |
|
|
| `uuid` | 105 | bytes | Random v4 UUID; must match the inner message UUID |
|
|
| `innerMessage` | 199 | bytes | Zstd-compressed serialized `MFFile` message |
|
|
| `signature` | 201 | bytes (optional) | GPG signature (ASCII-armored or binary) |
|
|
| `signer` | 202 | bytes (optional) | Fingerprint of the signing key |
|
|
| `signingPubKey` | 203 | bytes (optional) | Full GPG signing public key |
|
|
|
|
### SHA-256 Hash
|
|
|
|
The `sha256` field (104) covers the **compressed** `innerMessage` bytes. This
|
|
allows verifying data integrity before decompression.
|
|
|
|
## Compression
|
|
|
|
The `innerMessage` field is compressed with
|
|
[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must
|
|
enforce a decompression size limit to prevent decompression bombs. The reference
|
|
implementation limits decompressed size to 256 MiB. It writes zstd frames with a
|
|
window of at most 8 MiB, the largest window the zstd format recommends decoders
|
|
support, and refuses frames that ask for a larger one. It also refuses an inner
|
|
message whose file entries, hashes, timestamps and MIME types, counted at 176,
|
|
112, 64 and 16 bytes each, add up to more than 8 times its size. It refuses a
|
|
manifest file larger than 258 MiB without reading the rest of it: zstd's worst
|
|
case grows a 256 MiB inner message by 1/256 to 257 MiB, and the last MiB is room
|
|
for the signature, the signing key and the other outer fields.
|
|
|
|
## Inner Message (`MFFile`)
|
|
|
|
After decompressing `innerMessage`, the result is a serialized `MFFile`
|
|
(referred to as the manifest):
|
|
|
|
| Field | Number | Type | Description |
|
|
| ----------- | ------ | --------------------- | ------------------------------------- |
|
|
| `version` | 100 | enum | Must be `VERSION_ONE` (1) |
|
|
| `files` | 101 | repeated `MFFilePath` | List of files in the manifest |
|
|
| `uuid` | 102 | bytes | Random v4 UUID; must match outer UUID |
|
|
| `createdAt` | 201 | Timestamp (optional) | When the manifest was created |
|
|
|
|
## File Entries (`MFFilePath`)
|
|
|
|
Each file entry contains:
|
|
|
|
| Field | Number | Type | Description |
|
|
| ---------- | ------ | ------------------------- | ----------------------------------- |
|
|
| `path` | 1 | string | Relative file path (see Path Rules) |
|
|
| `size` | 2 | int64 | File size in bytes |
|
|
| `hashes` | 3 | repeated `MFFileChecksum` | At least one hash required |
|
|
| `mimeType` | 301 | string (optional) | MIME type |
|
|
| `mtime` | 302 | Timestamp (optional) | Modification time |
|
|
| `ctime` | 303 | Timestamp (optional) | Change time (inode metadata change) |
|
|
| `mode` | 304 | uint32 | Permission bits (see File Mode) |
|
|
|
|
## File Mode
|
|
|
|
`mode` holds a file's Unix permission bits, the nine `rwx` bits, so it is never
|
|
above `0777` (octal); the setuid, setgid and sticky bits are never recorded.
|
|
Writers record `0000` unless whoever creates the manifest asks for permissions.
|
|
`0000`, the proto3 default, means no mode was recorded: readers never check or
|
|
apply it.
|
|
|
|
The reference implementation records modes when `gen` or `freshen` is given
|
|
`--include-permissions`. `check` fails a file whose permission bits differ from
|
|
a recorded mode other than `0000`. `fetch` sets a recorded mode other than
|
|
`0000` on each file it writes, and refuses a manifest that records a mode above
|
|
`0777` before it requests any file.
|
|
|
|
## Path Rules
|
|
|
|
All `path` values must satisfy these invariants:
|
|
|
|
- **UTF-8**: paths must be valid UTF-8
|
|
- **Forward slashes**: use `/` as the path separator (never `\`)
|
|
- **Relative only**: no leading `/`
|
|
- **No parent traversal**: no `..` path segments
|
|
- **No empty segments**: no `//` sequences
|
|
- **No trailing slash**: paths refer to files, not directories
|
|
- **Listed once**: each path appears at most once in a manifest, compared byte
|
|
for byte, so `A.txt` and `a.txt` are two paths
|
|
|
|
Implementations must validate these invariants when reading and writing
|
|
manifests. Paths that violate these rules must be rejected, and a reader must
|
|
reject a manifest that lists a path more than once.
|
|
|
|
## Hash Format (`MFFileChecksum`)
|
|
|
|
Each checksum is a single `bytes multiHash` field containing a
|
|
[multihash](https://multiformats.io/multihash/)-encoded value. Multihash is
|
|
self-describing: the encoded bytes include a varint algorithm identifier
|
|
followed by a varint digest length followed by the digest itself.
|
|
|
|
The 1.0 implementation writes SHA-256 multihashes (`0x12` algorithm code).
|
|
Implementations must be able to verify SHA-256 multihashes at minimum.
|
|
|
|
## Signature Scheme
|
|
|
|
Signing is optional. When present, the signature covers a canonical string
|
|
constructed as:
|
|
|
|
```
|
|
ZNAVSRFG-<UUID>-<SHA256>
|
|
```
|
|
|
|
Where:
|
|
|
|
- `ZNAVSRFG` is the magic bytes string (literal ASCII)
|
|
- `<UUID>` is the hex-encoded UUID from the outer message
|
|
- `<SHA256>` is the hex-encoded SHA-256 hash from the outer message (covering
|
|
compressed data)
|
|
|
|
Components are separated by hyphens. The signature is produced by GPG over this
|
|
canonical string and stored in the `signature` field of the outer message. The
|
|
signing key's public key goes in `signingPubKey` and its fingerprint, in hex, in
|
|
`signer`.
|
|
|
|
A verifier accepts a signed manifest only if `signingPubKey` holds exactly one
|
|
primary key, `signature` is one good signature over the canonical string made by
|
|
that key (or one of its subkeys), and `signer` is that key's fingerprint. The
|
|
reference implementation refuses to load a manifest that fails these checks;
|
|
`check` and `fetch` given `--require-signature` then compare the required
|
|
fingerprint with `signer`.
|
|
|
|
## Deterministic Serialization
|
|
|
|
By default, manifests are generated deterministically:
|
|
|
|
- File entries are sorted by `path` in **lexicographic byte order**
|
|
- `createdAt` is omitted unless explicitly requested
|
|
- `mode` is `0000` unless explicitly requested
|
|
|
|
This ensures that two independent runs over the same directory tree produce
|
|
byte-identical `.mf` files (assuming file contents and metadata have not
|
|
changed).
|
|
|
|
## MIME Type
|
|
|
|
The recommended MIME type for `.mf` files is `application/octet-stream`. The
|
|
`.mf` file extension is the canonical identifier.
|
|
|
|
## Reference
|
|
|
|
- Proto definition: [`mfer/mf.proto`](../mfer/mf.proto)
|
|
- Reference implementation:
|
|
[git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
|