sneak/mfer
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Releases Wiki Activity

Make docs/FORMAT.md a complete, independently-implementable specification #84

New Issue
Closed
opened 2026-08-09 03:44:51 +02:00 by clawbot · 4 comments
clawbot commented 2026-08-09 03:44:51 +02:00
Collaborator
Copy Link
Copy Source

Context

The README states the goal plainly: mfer should "emerge as a de-facto
standard and be incorporated into other software", and "a compatible
javascript library is planned".

That goal has exactly one hard requirement: someone must be able to write an
interoperable implementation from the specification alone, without reading
the Go code. FORMAT.md does not currently meet that bar.

Gaps:

  • The signature scheme is underspecified. The canonical string format, its
    hex casing, its delimiters, what exactly is covered by the outer SHA-256
    (compressed vs uncompressed bytes), and the steps a verifier performs are
    not written down precisely enough to reimplement.
  • Path invariants are not specified at all — no statement on
    forward-slash-only, relative-only, .., leading /, or Unicode
    normalization. A second implementation will get these wrong.
  • Canonical serialization is not specified: field ordering, how determinism
    is achieved, what a writer must do to produce byte-identical output.
  • The README links to mfer/mf.proto as "the format specification" rather
    than to FORMAT.md. The proto is a schema, not a specification.
  • Error messages across the CLI and library have never been audited for
    consistency.

Definition of done

  • docs/FORMAT.md fully specifies: the magic, the outer framing, the
    compression, the inner structure, path invariants, the signature scheme
    end to end, and canonical serialization rules for deterministic output.
  • A reader who has never seen the Go code could implement a compatible
    writer and verifier from the document.
  • Test vectors are included: at least one known-good manifest with its
    bytes, its hash, and its signature, so a second implementation can check
    itself.
  • The README links to docs/FORMAT.md as the primary specification
    reference, not to the .proto.
  • Error messages across mfer/ and internal/cli/ are audited for
    consistency: same capitalization convention, no trailing punctuation, no
    redundant "error:" prefixes, and each one names the thing that failed.
  • make check passes. TODO.md updated in the same commit.

Implementation requirements

  • This issue is blocked on #81 and #82. Do not start until the format
    and signature decisions are answered — writing the spec against the
    current behavior and then rewriting it after the answers land is wasted
    work. Confirm both are closed before beginning.
  • Specify what the format is, not what the Go implementation happens to
    do. Where the two differ, that is a bug to file, not a line in the spec.
  • Test vectors must be generated by the actual implementation and verified
    to round-trip. A hand-written test vector that does not parse is worse
    than none.
  • The error-message audit must not change error semantics — no
    reclassifying which errors are returned, no changing sentinel identity.
    Wording only. If an error is wrong rather than badly worded, file it.
  • This depends on FORMAT.md having moved to docs/ (#74). Coordinate the
    ordering or rebase.
  • Commit title must end with (closes #84).
## Context The README states the goal plainly: mfer should "emerge as a de-facto standard and be incorporated into other software", and "a compatible javascript library is planned". That goal has exactly one hard requirement: someone must be able to write an interoperable implementation from the specification alone, without reading the Go code. `FORMAT.md` does not currently meet that bar. Gaps: - The signature scheme is underspecified. The canonical string format, its hex casing, its delimiters, what exactly is covered by the outer SHA-256 (compressed vs uncompressed bytes), and the steps a verifier performs are not written down precisely enough to reimplement. - Path invariants are not specified at all — no statement on forward-slash-only, relative-only, `..`, leading `/`, or Unicode normalization. A second implementation will get these wrong. - Canonical serialization is not specified: field ordering, how determinism is achieved, what a writer must do to produce byte-identical output. - The README links to `mfer/mf.proto` as "the format specification" rather than to `FORMAT.md`. The proto is a schema, not a specification. - Error messages across the CLI and library have never been audited for consistency. ## Definition of done - `docs/FORMAT.md` fully specifies: the magic, the outer framing, the compression, the inner structure, path invariants, the signature scheme end to end, and canonical serialization rules for deterministic output. - A reader who has never seen the Go code could implement a compatible writer and verifier from the document. - Test vectors are included: at least one known-good manifest with its bytes, its hash, and its signature, so a second implementation can check itself. - The README links to `docs/FORMAT.md` as the primary specification reference, not to the `.proto`. - Error messages across `mfer/` and `internal/cli/` are audited for consistency: same capitalization convention, no trailing punctuation, no redundant "error:" prefixes, and each one names the thing that failed. - `make check` passes. `TODO.md` updated in the same commit. ## Implementation requirements - **This issue is blocked on #81 and #82.** Do not start until the format and signature decisions are answered — writing the spec against the current behavior and then rewriting it after the answers land is wasted work. Confirm both are closed before beginning. - Specify what the format **is**, not what the Go implementation happens to do. Where the two differ, that is a bug to file, not a line in the spec. - Test vectors must be generated by the actual implementation and verified to round-trip. A hand-written test vector that does not parse is worse than none. - The error-message audit must not change error **semantics** — no reclassifying which errors are returned, no changing sentinel identity. Wording only. If an error is wrong rather than badly worded, file it. - This depends on `FORMAT.md` having moved to `docs/` (#74). Coordinate the ordering or rebase. - Commit title must end with ` (closes #84)`.
clawbot added this to the 1.0.0 milestone 2026-08-09 03:44:51 +02:00
clawbot commented 2026-10-07 10:35:00 +02:00
Author
Collaborator
Copy Link
Copy Source

Two parts of this issue wait on no open decision and now have their own issues: the error-message audit (#165) and the README link to the specification (#166). The rest of this issue still waits on #81 and #82.

Model: opus-5-5

Two parts of this issue wait on no open decision and now have their own issues: the error-message audit (https://git.eeqj.de/sneak/mfer/issues/165) and the README link to the specification (https://git.eeqj.de/sneak/mfer/issues/166). The rest of this issue still waits on https://git.eeqj.de/sneak/mfer/issues/81 and https://git.eeqj.de/sneak/mfer/issues/82. Model: opus-5-5
clawbot commented 2026-10-07 23:19:47 +02:00
Author
Collaborator
Copy Link
Copy Source

Added to this issue's scope by sneak's ruling on the signature scheme decisions (#82 (comment)): the Signature Scheme section of docs/FORMAT.md is corrected to describe the code, which is the format.

  • The outer sha256 covers the compressed innerMessage bytes.
  • The canonical string's third part is the lower-case hex of the multihash encoding of that SHA-256, so it starts 1220, as signatureString in mfer/manifest.go builds it; the UUID is lower-case hex too.
  • Signatures are embedded only; there is no detached signature file.
  • Signing uses OpenPGP in Go, not GnuPG, once #181 lands.

The signature parts no longer wait on anything. The rest of this issue still waits on the wire format decisions (#81).

Model: opus-5-5

Added to this issue's scope by sneak's ruling on the signature scheme decisions (https://git.eeqj.de/sneak/mfer/issues/82#issuecomment-132656): the Signature Scheme section of `docs/FORMAT.md` is corrected to describe the code, which is the format. - The outer `sha256` covers the compressed `innerMessage` bytes. - The canonical string's third part is the lower-case hex of the multihash encoding of that SHA-256, so it starts `1220`, as `signatureString` in `mfer/manifest.go` builds it; the UUID is lower-case hex too. - Signatures are embedded only; there is no detached signature file. - Signing uses OpenPGP in Go, not GnuPG, once https://git.eeqj.de/sneak/mfer/issues/181 lands. The signature parts no longer wait on anything. The rest of this issue still waits on the wire format decisions (https://git.eeqj.de/sneak/mfer/issues/81). Model: opus-5-5
clawbot commented 2026-10-13 13:00:05 +02:00
Author
Collaborator
Copy Link
Copy Source

Remaining scope, 2026-10-13

The wire format answers are settled (#81 (comment)) except whether mimeType and ctime stay; the signature part is #187; the error wording and the README link landed in #165 and #166. This starts after #188, #193 and #194 land, since all three edit docs/FORMAT.md.

Definition of done

docs/FORMAT.md, describing what the code does (where the code is wrong, file an issue instead):

  • Path Rules: a path holds the name exactly as the file system gives it, with no Unicode normalization; readers compare bytes.
  • File Structure: a .mf is always a whole file; a different framing would need a different magic.
  • Readers ignore fields they do not know, so a later version can add a field without breaking 1.0 readers.
  • Deterministic Serialization: what a writer does to give the same bytes for the same tree, as the code does it (entry order, protobuf encoding, zstd settings, how the UUID is chosen). The section now says two runs give byte-identical files; check that against how gen picks the UUID without --seed and correct it.
  • Test vector: one small signed manifest, made once by the code from a fixed UUID and a test key, checked in with its bytes, its sha256, its signed string and its signature shown in the document, and a test that loads it and verifies the signature, so the document cannot drift from the code.
  • The mimeType and ctime rows stay as they are until sneak answers question B on #81.
  • make check passes; markdown formatted with make fmt.

Model: opus-5-5

## Remaining scope, 2026-10-13 The wire format answers are settled (https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-136431) except whether `mimeType` and `ctime` stay; the signature part is https://git.eeqj.de/sneak/mfer/issues/187; the error wording and the README link landed in https://git.eeqj.de/sneak/mfer/issues/165 and https://git.eeqj.de/sneak/mfer/issues/166. This starts after https://git.eeqj.de/sneak/mfer/pulls/188, https://git.eeqj.de/sneak/mfer/issues/193 and https://git.eeqj.de/sneak/mfer/issues/194 land, since all three edit `docs/FORMAT.md`. ## Definition of done `docs/FORMAT.md`, describing what the code does (where the code is wrong, file an issue instead): - Path Rules: a path holds the name exactly as the file system gives it, with no Unicode normalization; readers compare bytes. - File Structure: a `.mf` is always a whole file; a different framing would need a different magic. - Readers ignore fields they do not know, so a later version can add a field without breaking 1.0 readers. - Deterministic Serialization: what a writer does to give the same bytes for the same tree, as the code does it (entry order, protobuf encoding, zstd settings, how the UUID is chosen). The section now says two runs give byte-identical files; check that against how `gen` picks the UUID without `--seed` and correct it. - Test vector: one small signed manifest, made once by the code from a fixed UUID and a test key, checked in with its bytes, its `sha256`, its signed string and its signature shown in the document, and a test that loads it and verifies the signature, so the document cannot drift from the code. - The `mimeType` and `ctime` rows stay as they are until sneak answers question B on https://git.eeqj.de/sneak/mfer/issues/81. - `make check` passes; markdown formatted with `make fmt`. Model: opus-5-5
clawbot commented 2026-10-13 21:11:53 +02:00
Author
Collaborator
Copy Link
Copy Source

#217 finishes docs/FORMAT.md as described in #84 (comment): whole-file framing, unknown fields, paths without normalization, what the writer does to give the same bytes, and a signed test vector in mfer/testdata/ that a test loads and verifies.

Model: opus-5-5

https://git.eeqj.de/sneak/mfer/pulls/217 finishes `docs/FORMAT.md` as described in https://git.eeqj.de/sneak/mfer/issues/84#issuecomment-136437: whole-file framing, unknown fields, paths without normalization, what the writer does to give the same bytes, and a signed test vector in `mfer/testdata/` that a test loads and verifies. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
All checks pass, reviewed, rebased, ready for merge
 needs-checks
make check does not pass cleanly
 needs-rebase
PR has merge conflicts or is behind main
 needs-review
Code review not yet done
 needs-rework
Code review found issues to fix
No labels
Milestone
No items
No Milestone 1.0.0
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/mfer#84