DECISION: wire format questions blocking the 1.0 format freeze #81

New Issue

clawbot · 2026-08-09T03:44:03+02:00

clawbot commented

2026-08-09 03:44:03 +02:00

Owner decisions required. These six questions are currently inlined in the
README's 1.0 roadmap with empty > _answer:_ slots. Moving them here so
answers land in one place.

Every one of these changes the on-disk format. After 1.0 they are frozen,
because .mf files written by 1.0 must stay readable forever. Answer them
now or accept them as-is permanently.

1. Simplify MFFileChecksum? It is a message wrapping a single
bytes multiHash field. Multihash already self-describes its algorithm, so
repeated bytes hashes directly on MFFilePath would be simpler and cut
per-file protobuf overhead. Is the extra message layer intentional — e.g.
reserved for per-hash metadata like verified_at?

Recommendation: keep the wrapper. The overhead is a few bytes per file
and the format's stated non-goal is small manifests. Removing an
extension point to save bytes you have explicitly said you do not care
about is the wrong trade.

2. Store Unix file mode? The format stores mtime/ctime but not
permissions. Irrelevant for pure archival, a real gap for software
distribution and filesystem restore.

Recommendation: reserve optional uint32 mode = 305 now without
populating it. Reserving a field number costs nothing; adding one after
1.0 to a format with deployed readers costs a version negotiation.

3. Remove atime? Access time is volatile, frequently disabled
(noatime), and non-deterministic — two manifests of an unchanged directory
will differ, which directly conflicts with the determinism goal.

Recommendation: remove it from the schema. It is actively harmful to the
stated goal and there is no use case in the README that needs it. If
removal is too aggressive, mark it reserved and never populate it — but do
not ship 1.0 writing a field that breaks reproducibility.

4. What are the path normalization rules? string path has no
specification: forward-slash always? relative always? .. forbidden? UTF-8
NFC vs NFD (macOS writes NFD, Linux NFC — the same filename produces
different bytes)? Maximum length? This is both a security question (#61) and
a cross-platform correctness question.

Recommendation: mandate relative, forward-slash, valid UTF-8, no ..,
no leading /, no empty segments — which is what ValidatePath already
enforces on write. The genuinely open part is Unicode normalization; NFC
is the usual choice, but it means macOS-generated manifests need
normalizing at write time, which is a real behavior change worth deciding
deliberately.

5. Add a version byte after the magic? Currently ZNAVSRFG is followed
immediately by protobuf. A version byte (ZNAVSRFG\x01) would allow future
framing changes without parsing protobuf first. MFFileOuter.Version exists
but requires successful deserialization to read.

Recommendation: add it. One byte, and it is the difference between being
able to change the framing later and not. This is the cheapest option
value in the list.

6. Add a length prefix after the magic? Protobuf is not
self-delimiting, so the current framing cannot support concatenating
manifests or appending data.

Recommendation: add a varint length prefix if there is any intent to
support concatenation or embedding; skip it if .mf is always a whole
file. This one genuinely depends on intent — worth stating the intent
either way so the spec can say so.

Definition of done

Each question is answered inline in a comment here.
Each answer becomes either a tracked implementation issue or an explicit
"no change" recorded in docs/FORMAT.md.
The format specification documents the final answers, including the ones
that resulted in no change — a spec that is silent on normalization is a
spec that will be implemented inconsistently by the planned JavaScript
library.

Please answer and reassign to clawbot.

Owner decisions required. These six questions are currently inlined in the README's 1.0 roadmap with empty `> _answer:_` slots. Moving them here so answers land in one place. Every one of these changes the on-disk format. After 1.0 they are frozen, because `.mf` files written by 1.0 must stay readable forever. Answer them now or accept them as-is permanently. --- **1. Simplify `MFFileChecksum`?** It is a message wrapping a single `bytes multiHash` field. Multihash already self-describes its algorithm, so `repeated bytes hashes` directly on `MFFilePath` would be simpler and cut per-file protobuf overhead. Is the extra message layer intentional — e.g. reserved for per-hash metadata like `verified_at`? > _Recommendation:_ keep the wrapper. The overhead is a few bytes per file > and the format's stated non-goal is small manifests. Removing an > extension point to save bytes you have explicitly said you do not care > about is the wrong trade. **2. Store Unix file mode?** The format stores mtime/ctime but not permissions. Irrelevant for pure archival, a real gap for software distribution and filesystem restore. > _Recommendation:_ reserve `optional uint32 mode = 305` now without > populating it. Reserving a field number costs nothing; adding one after > 1.0 to a format with deployed readers costs a version negotiation. **3. Remove `atime`?** Access time is volatile, frequently disabled (`noatime`), and non-deterministic — two manifests of an unchanged directory will differ, which directly conflicts with the determinism goal. > _Recommendation:_ remove it from the schema. It is actively harmful to the > stated goal and there is no use case in the README that needs it. If > removal is too aggressive, mark it reserved and never populate it — but do > not ship 1.0 writing a field that breaks reproducibility. **4. What are the path normalization rules?** `string path` has no specification: forward-slash always? relative always? `..` forbidden? UTF-8 NFC vs NFD (macOS writes NFD, Linux NFC — the same filename produces different bytes)? Maximum length? This is both a security question (#61) and a cross-platform correctness question. > _Recommendation:_ mandate relative, forward-slash, valid UTF-8, no `..`, > no leading `/`, no empty segments — which is what `ValidatePath` already > enforces on write. The genuinely open part is Unicode normalization; NFC > is the usual choice, but it means macOS-generated manifests need > normalizing at write time, which is a real behavior change worth deciding > deliberately. **5. Add a version byte after the magic?** Currently `ZNAVSRFG` is followed immediately by protobuf. A version byte (`ZNAVSRFG\x01`) would allow future framing changes without parsing protobuf first. `MFFileOuter.Version` exists but requires successful deserialization to read. > _Recommendation:_ add it. One byte, and it is the difference between being > able to change the framing later and not. This is the cheapest option > value in the list. **6. Add a length prefix after the magic?** Protobuf is not self-delimiting, so the current framing cannot support concatenating manifests or appending data. > _Recommendation:_ add a varint length prefix if there is any intent to > support concatenation or embedding; skip it if `.mf` is always a whole > file. This one genuinely depends on intent — worth stating the intent > either way so the spec can say so. --- ## Definition of done - Each question is answered inline in a comment here. - Each answer becomes either a tracked implementation issue or an explicit "no change" recorded in `docs/FORMAT.md`. - The format specification documents the final answers, including the ones that resulted in no change — a spec that is silent on normalization is a spec that will be implemented inconsistently by the planned JavaScript library. Please answer and reassign to `clawbot`.

clawbot added this to the 1.0.0 milestone 2026-08-09 03:44:03 +02:00

sneak was assigned by clawbot

2026-08-09 03:44:03 +02:00

clawbot referenced this issue

2026-08-09 03:44:51 +02:00

Make docs/FORMAT.md a complete, independently-implementable specification #84

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/mfer#81