Make docs/FORMAT.md a complete, independently-implementable specification #84

New Issue

2026-08-09T03:44:51+02:00

clawbot commented

2026-08-09 03:44:51 +02:00

Context

The README states the goal plainly: mfer should "emerge as a de-facto
standard and be incorporated into other software", and "a compatible
javascript library is planned".

That goal has exactly one hard requirement: someone must be able to write an
interoperable implementation from the specification alone, without reading
the Go code. FORMAT.md does not currently meet that bar.

Gaps:

The signature scheme is underspecified. The canonical string format, its
hex casing, its delimiters, what exactly is covered by the outer SHA-256
(compressed vs uncompressed bytes), and the steps a verifier performs are
not written down precisely enough to reimplement.
Path invariants are not specified at all — no statement on
forward-slash-only, relative-only, .., leading /, or Unicode
normalization. A second implementation will get these wrong.
Canonical serialization is not specified: field ordering, how determinism
is achieved, what a writer must do to produce byte-identical output.
The README links to mfer/mf.proto as "the format specification" rather
than to FORMAT.md. The proto is a schema, not a specification.
Error messages across the CLI and library have never been audited for
consistency.

Definition of done

docs/FORMAT.md fully specifies: the magic, the outer framing, the
compression, the inner structure, path invariants, the signature scheme
end to end, and canonical serialization rules for deterministic output.
A reader who has never seen the Go code could implement a compatible
writer and verifier from the document.
Test vectors are included: at least one known-good manifest with its
bytes, its hash, and its signature, so a second implementation can check
itself.
The README links to docs/FORMAT.md as the primary specification
reference, not to the .proto.
Error messages across mfer/ and internal/cli/ are audited for
consistency: same capitalization convention, no trailing punctuation, no
redundant "error:" prefixes, and each one names the thing that failed.
make check passes. TODO.md updated in the same commit.

Implementation requirements

This issue is blocked on #81 and #82. Do not start until the format
and signature decisions are answered — writing the spec against the
current behavior and then rewriting it after the answers land is wasted
work. Confirm both are closed before beginning.
Specify what the format is, not what the Go implementation happens to
do. Where the two differ, that is a bug to file, not a line in the spec.
Test vectors must be generated by the actual implementation and verified
to round-trip. A hand-written test vector that does not parse is worse
than none.
The error-message audit must not change error semantics — no
reclassifying which errors are returned, no changing sentinel identity.
Wording only. If an error is wrong rather than badly worded, file it.
This depends on FORMAT.md having moved to docs/ (#74). Coordinate the
ordering or rebase.
Commit title must end with (closes #84).

## Context The README states the goal plainly: mfer should "emerge as a de-facto standard and be incorporated into other software", and "a compatible javascript library is planned". That goal has exactly one hard requirement: someone must be able to write an interoperable implementation from the specification alone, without reading the Go code. `FORMAT.md` does not currently meet that bar. Gaps: - The signature scheme is underspecified. The canonical string format, its hex casing, its delimiters, what exactly is covered by the outer SHA-256 (compressed vs uncompressed bytes), and the steps a verifier performs are not written down precisely enough to reimplement. - Path invariants are not specified at all — no statement on forward-slash-only, relative-only, `..`, leading `/`, or Unicode normalization. A second implementation will get these wrong. - Canonical serialization is not specified: field ordering, how determinism is achieved, what a writer must do to produce byte-identical output. - The README links to `mfer/mf.proto` as "the format specification" rather than to `FORMAT.md`. The proto is a schema, not a specification. - Error messages across the CLI and library have never been audited for consistency. ## Definition of done - `docs/FORMAT.md` fully specifies: the magic, the outer framing, the compression, the inner structure, path invariants, the signature scheme end to end, and canonical serialization rules for deterministic output. - A reader who has never seen the Go code could implement a compatible writer and verifier from the document. - Test vectors are included: at least one known-good manifest with its bytes, its hash, and its signature, so a second implementation can check itself. - The README links to `docs/FORMAT.md` as the primary specification reference, not to the `.proto`. - Error messages across `mfer/` and `internal/cli/` are audited for consistency: same capitalization convention, no trailing punctuation, no redundant "error:" prefixes, and each one names the thing that failed. - `make check` passes. `TODO.md` updated in the same commit. ## Implementation requirements - **This issue is blocked on #81 and #82.** Do not start until the format and signature decisions are answered — writing the spec against the current behavior and then rewriting it after the answers land is wasted work. Confirm both are closed before beginning. - Specify what the format **is**, not what the Go implementation happens to do. Where the two differ, that is a bug to file, not a line in the spec. - Test vectors must be generated by the actual implementation and verified to round-trip. A hand-written test vector that does not parse is worse than none. - The error-message audit must not change error **semantics** — no reclassifying which errors are returned, no changing sentinel identity. Wording only. If an error is wrong rather than badly worded, file it. - This depends on `FORMAT.md` having moved to `docs/` (#74). Coordinate the ordering or rebase. - Commit title must end with ` (closes #84)`.

clawbot added this to the 1.0.0 milestone 2026-08-09 03:44:51 +02:00