mfer/TODO.md

TODO: mfer 1.0

Design Questions

These need answers before implementation. Respond inline.

1. Should atime be removed from the proto?

Access time is volatile, non-deterministic, and often disabled (noatime mount option). Including it means two manifests of the same unchanged directory tree generated at different times will differ. This hurts deterministic/reproducible manifests. Recommendation: remove it or document it as "never set by default, opt-in only."

Answer:

2. Should MFFileChecksum stay as a wrapper message or simplify to repeated bytes?

Currently MFFilePath.hashes is repeated MFFileChecksum where MFFileChecksum contains a single bytes multiHash field. Since multihash already self-describes the algorithm, repeated bytes hashes = 3 on MFFilePath would be equivalent and save protobuf overhead (one fewer tag+length per hash per file). The wrapper is justified if you plan to add per-hash metadata later (e.g. verified_at timestamp). Otherwise it's unnecessary indirection.

Answer:

3. Should file permissions/mode be supported?

The proto has no field for Unix permission bits. For the archival use case (ExFAT drives, filesystem-independent checksums), this may not matter. But for software distribution or filesystem restoration, the absence of permission bits is a gap. Recommendation: reserve optional uint32 mode = 305 in MFFilePath now, even if it's not populated yet.

Answer:

4. Path semantics: what are the rules?

The proto has string path with no specified invariants. These need to be documented and enforced:

• Always forward-slash separated? (even on Windows)
• Must be relative? (no leading /)
• No .. components? (path traversal prevention)
• UTF-8 NFC normalized? (macOS uses NFD, Linux uses NFC — same filename looks different in bytes)
• Max path length?

Answer:

5. Should the sha256 field on MFFileOuter hash compressed or uncompressed data?

Currently it hashes the compressed inner bytes. This is actually the better choice — it lets you verify integrity before decompression (preventing decompression bombs from untrusted sources). But it's a non-obvious design decision that should be documented. Confirm this is intentional.

Answer:

6. GPG vs signify (Ed25519) for signatures?

The current implementation shells out to gpg, which is fragile (binary might not be installed, output format changes between versions). Options:

• Keep GPG: Use github.com/ProtonMail/go-crypto for pure-Go OpenPGP (no subprocess, cross-platform, testable).
• Switch to signify/Ed25519: Simpler, smaller keys, no key management complexity. github.com/frankbraun/gosignify exists. README already mentions this as an option.
• Support both: More work, but maximum flexibility.

Answer:

7. Should manifests be deterministic by default?

If createdAt is populated and file order depends on filesystem walk order, two runs of mfer gen on the same directory produce different manifests. Recommendation: sort files lexicographically by path, omit createdAt unless --timestamp is passed. Deterministic by default.

Answer:

8. Should the magic bytes include a version byte?

Currently: ZNAVSRFG (8 bytes), then protobuf. If the framing ever needs to change (e.g. different compression framing, post-quantum signatures), you'd need to parse the protobuf to discover the version. Adding a version byte after the magic (ZNAVSRFG\x01) allows future format changes without requiring protobuf parsing for version detection.

Answer:

9. Detached signature support?

For HTTP distribution, a detached .mf.sig file alongside index.mf would let servers serve both without modifying the manifest itself. This follows the SHASUMS + SHASUMS.asc pattern. The embedded signature is better for single-file distribution. Worth supporting both modes?

Answer:

10. What about the duplicate internal packages?

There are two parallel implementations:

• mfer/scanner.go + mfer/checker.go (newer, better-typed with FileSize, RelFilePath)
• internal/scanner/ + internal/checker/ (older, raw types)

The mfer/ versions are superior. Should the internal/ versions be deleted and CLI updated to use mfer/ package?

Answer:

---

Implementation Plan

Ordered by dependency and priority. Each item should be a PR.

Phase 1: Foundation (format correctness)

• Delete internal/scanner/ and internal/checker/ — consolidate on the mfer/ package versions. Update CLI code to use mfer.Scanner and mfer.Checker.
• Add deterministic file ordering — sort file entries by path (lexicographic, byte-order) before serialization in Builder.Build(). Add a test that generates a manifest twice and asserts byte-identical output.
• Add decompression size limit — use io.LimitReader in deserializeInner() with m.pbOuter.Size as the bound.
• Fix errors.Is dead code in checker — replace with os.IsNotExist(err) or errors.Is(err, fs.ErrNotExist).
• Fix AddFile to verify size — after reading, check totalRead == size and return an error on mismatch.
• Specify path invariants — add comments to proto and validate in Builder.AddFile / Builder.AddFileWithHash.

Phase 2: CLI polish

• Fix flag naming — ensure all CLI flags use kebab-case as primary names (--include-dotfiles, --follow-symlinks).
• Fix URL construction in fetch — use url.JoinPath() instead of string concatenation.
• Add progress rate-limiting to Checker — throttle to once per second, matching Scanner behavior.
• Add --deterministic flag (or make it default) — omit createdAt timestamp, sort files.

Phase 3: Robustness

• Replace GPG subprocess with pure-Go crypto — use github.com/ProtonMail/go-crypto or switch to Ed25519/signify.
• Add timeout to any remaining subprocess calls.
• Add fuzzing tests for NewManifestFromReader — parses untrusted input.
• Add retry logic to fetch — exponential backoff for transient HTTP errors.

Phase 4: Format finalization

• Remove or deprecate atime from proto (pending design question #1).
• Reserve optional uint32 mode = 305 in MFFilePath (pending design question #3).
• Add a version byte after magic (pending design question #8).
• Write a format specification document — separate from README.

Phase 5: Release prep

• Reconcile module path (sneak.berlin/go/mfer vs git.eeqj.de/sneak/mfer).
• Audit all error messages for consistency.
• Ensure --version output matches SemVer format.
• Tag v1.0.0.