diff --git a/AGENTS.md b/AGENTS.md index fb82780..bea12f1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,5 +27,8 @@ source for coding standards, formatting, linting, and workflow rules. - The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are committed (required for `go get` compatibility). - The format specification is in `FORMAT.md`. -- See the TODO section in `README.md` for the 1.0 implementation plan and open - design questions. +- Open work, open design questions included, is tracked only in the repo's + issues: https://git.eeqj.de/sneak/mfer/issues. There is no `TODO.md` and no + TODO list in `README.md`; do not add either. For this repo this overrides the + `REPO_POLICIES.md` rule to put the todo list in the README, per sneak's + ruling: https://git.eeqj.de/sneak/mfer/issues/76#issuecomment-118130. diff --git a/README.md b/README.md index 4dabec8..015bf48 100644 --- a/README.md +++ b/README.md @@ -157,18 +157,23 @@ The manifest file would do several important things: - metadata size should not be used as an excuse to sacrifice utility (such as providing checksums over each chunk of a large file) -# Open Questions +# Original Design Questions + +These were the open questions when the project started; open design questions +are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues). - Should the manifest file include checksums of individual file chunks, or just - for the whole assembled file? - - - If so, should the chunksize be fixed or dynamic? + for the whole assembled file? If so, should the chunk size be fixed or + dynamic? Still open, on + [issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698). - Should the manifest signature format be GnuPG signatures, or those from OpenBSD's signify (of which there is a good - [golang implementation](https://github.com/frankbraun/gosignify)? + [golang implementation](https://github.com/frankbraun/gosignify))? Still open, + as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82). -- Should the on-disk serialization format be proto3 or json? +- Should the on-disk serialization format be proto3 or json? Settled: it is + proto3, see `FORMAT.md` and `mfer/mf.proto`. # Tool Examples @@ -246,207 +251,10 @@ regardless of filesystem format. Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired username for an account on this Gitea instance. -# TODO: Remaining Work for 1.0 +# TODO -## Design Questions (Owner Decision Required) - -These require @sneak's input before implementation. Answers should be added -inline below each question. - -### Format Design - -**1. Should `MFFileChecksum` be simplified?** Currently it's a separate message -wrapping a single `bytes multiHash` field. Since multihash already -self-describes the algorithm, `repeated bytes hashes` directly on `MFFilePath` -would be simpler and reduce per-file protobuf overhead. Is the extra message -layer intentional (e.g. planning to add per-hash metadata like `verified_at`)? - -> _answer:_ - -**2. Should file permissions/mode be stored?** The format stores mtime/ctime but -not Unix file permissions. For archival use this may not matter, but for -software distribution or filesystem restoration it's a gap. Should we reserve a -field now (e.g. `optional uint32 mode = 305`) even if we don't populate it yet? - -> _answer:_ - -**3. Should `atime` be removed from the schema?** Access time is volatile, -non-deterministic, and often disabled (`noatime`). Including it means two -manifests of the same directory at different times will differ, which conflicts -with the determinism goal. Remove it, or document it as "never set by default"? - -> _answer:_ - -**4. What are the path normalization rules?** The proto has `string path` with -no specification about: always forward-slash? Must be relative? No `..` -components allowed? UTF-8 NFC vs NFD normalization (macOS vs Linux)? Max path -length? This is a security issue (path traversal) and a cross-platform -compatibility issue. What rules should the spec mandate? - -> _answer:_ - -**5. Should we add a version byte after the magic?** Currently `ZNAVSRFG` is -followed immediately by protobuf. Adding a version byte (`ZNAVSRFG\x01`) would -allow future framing changes without requiring protobuf parsing to detect the -version. `MFFileOuter.Version` serves this purpose but requires successful -deserialization to read. Worth the extra byte? - -> _answer:_ - -**6. Should we add a length-prefix after the magic?** Protobuf is not -self-delimiting. If we ever want to concatenate manifests or append data after -the protobuf, the current framing is insufficient. Add a varint or fixed-width -length-prefix? - -> _answer:_ - -### Signature Design - -**7. What does the outer SHA-256 hash cover — compressed or uncompressed data?** -The code currently hashes compressed data (good for verifying before -decompression), but this should be explicitly documented. Which is the intended -behavior? - -> _answer:_ - -**8. Should `signatureString()` sign raw bytes instead of a hex-encoded -string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with hex -encoding, which adds a transformation layer. Signing the raw `sha256` bytes (or -compressed `innerMessage` directly) would be simpler. Keep the string format or -switch to raw bytes? - -> _answer:_ - -**9. Should we support detached signature files (`.mf.sig`)?** Embedded -signatures are better for single-file distribution. Detached `.mf.sig` files -follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are simpler for HTTP -serving. Support both modes? - -> _answer:_ - -**10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is fragile -(may not be installed, version-dependent output). -`github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could use -Ed25519/signify (simpler, no key management). Which direction? - -> _answer:_ - -### Implementation Design - -**11. Should manifests be deterministic by default?** This means: sort file -entries by path, omit `createdAt` timestamp (or make it opt-in), no `atime`. -Should determinism be the default, with a `--include-timestamps` flag to opt in? - -> _answer:_ - -**12. Should we consolidate or keep both scanner/checker implementations?** -There are two parallel implementations: `mfer/scanner.go` + `mfer/checker.go` -(typed with `FileSize`, `RelFilePath`) and `internal/scanner/` + -`internal/checker/` (raw `int64`, `string`). The `mfer/` versions are superior. -Delete the `internal/` versions? - -> _answer:_ - -**13. Should the `manifest` type be exported?** Currently unexported with -exported constructors (`NewManifestFromReader`, `NewManifestFromFile`). -Consumers can't declare `var m *mfer.manifest`. Export the type, or define an -interface? - -> _answer:_ - -**14. What should the Go module path be for 1.0?** Currently -`sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in the -proto `go_package` option. Which is canonical? - -> _answer:_ - -## Implementation Tasks - -### Repo Infrastructure - -- [ ] Add `.golangci.yml` (fetch from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`) -- [ ] Add `.editorconfig` -- [ ] Add `.gitea/workflows/check.yml` that runs `docker build .` - -### Format & Correctness - -- [ ] Resolve proto `go_package` path inconsistency - (`git.eeqj.de/sneak/mfer/mfer` vs `sneak.berlin/go/mfer`) -- [ ] Specify path invariants — add proto comments requiring UTF-8, - forward-slash, relative paths, no `..`, no leading `/`; validate in - `Builder.AddFile` and `Builder.AddFileWithHash` (pending design question - answer) -- [ ] Remove or deprecate `atime` from proto (pending design question answer) -- [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future file - permissions (pending design question answer) -- [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version 1 - (pending design question answer) -- [ ] Write format specification document — separate from README: magic, outer - structure, compression, inner structure, path invariants, signature - scheme, canonical serialization - -### Library - -- [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate on - `mfer/` package versions; update CLI code (pending design question answer) -- [ ] Add deterministic file ordering — sort entries by path (lexicographic, - byte-order) in `Builder.Build()`; add test asserting byte-identical output - from two runs -- [ ] Add decompression size limit — `io.LimitReader` in `deserializeInner()` - with `m.pbOuter.Size` as bound -- [ ] Fix `errors.Is` dead code in checker — replace with `os.IsNotExist(err)` - or `errors.Is(err, fs.ErrNotExist)` -- [ ] Fix `AddFile` to verify size — check `totalRead == size` after reading, - return error on mismatch -- [ ] Export the `manifest` type or define a public interface (pending design - question answer) — currently consumers cannot hold a reference to a loaded - manifest in their own type declarations -- [ ] Replace GPG subprocess calls with pure-Go crypto (pending design question - answer) — current implementation shells out to `gpg` which may not be - installed -- [ ] Add timeout to any remaining subprocess calls - -### CLI - -- [ ] Fix flag naming — all CLI flags should use kebab-case as primary - (`--include-dotfiles`, `--follow-symlinks`) -- [ ] Fix URL construction in fetch — use `BaseURL.JoinPath()` or - `url.JoinPath()` instead of string concatenation -- [ ] Add progress rate-limiting to Checker — throttle to once per second, - matching Scanner -- [ ] Add `--deterministic` flag or make it default — omit `createdAt`, sort - files (pending design question answer) -- [ ] Wire `--version` flag properly (currently only a `version` subcommand - exists; top-level `--version` shows urfave/cli generic output) -- [ ] Add retry logic to `fetch` — currently no retries on transient HTTP - errors; needs exponential backoff -- [ ] `fetch` command uses bare `http.Get` with no timeout — needs `http.Client` - with configurable timeout - -### Testing & Robustness - -- [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf deserialization - of untrusted input needs fuzz coverage -- [ ] Add integration test for `freshen` CLI command — current tests only verify - setup, not the actual freshen operation end-to-end -- [ ] Add test for `fetch` CLI command end-to-end (currently only `downloadFile` - is tested) - -### Documentation - -- [ ] Promote `FORMAT.md` as primary spec reference; README should link to it - more prominently -- [ ] Audit and update all error messages for consistency and helpfulness -- [ ] Document the signature scheme more thoroughly (canonical string format, - verification steps) - -### Release - -- [ ] Finalize Go module path -- [ ] Update version constant in `mfer/constants.go` -- [ ] Add `--version` output matching SemVer -- [ ] Tag `v1.0.0` +Open work, open design questions included, is tracked in this repo's issues: +[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues). # See Also diff --git a/TODO.md b/TODO.md deleted file mode 100644 index c4e97e2..0000000 --- a/TODO.md +++ /dev/null @@ -1,135 +0,0 @@ -# Workflow - -- branch (from `main`) -- do the work in Next Step -- move Next Step to the top of Completed Steps -- move the top item of Future Steps into Next Step -- commit (`TODO.md` changes in the same commit as the work) -- merge to `main` if the branch is not protected, otherwise open a PR -- push - -# Status - -pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists open -design questions and implementation tasks; policy compliance work is in flight -and unmerged. - -# Next Step - -Work through the remaining compliance items folded from the 2026-07-02 audit -(the first group under Future Steps): `.editorconfig`, `.gitignore` coverage, -gofumpt-based `fmt-check`, README "Getting Started", and the rest. -`.golangci.yml` and `TODO.md` are tracked and committed as of 2026-08-07, so the -only thing left of the `chore/align-repo-policies` branch is the list below. - -# Completed Steps - -- 2026-10-03: pinned the CLI error messages by driving the functions that emit - them in `internal/cli/errmsg_test.go`, and made the freshen mtime-presence - test distinguish an absent mtime from the epoch (#87) -- 2026-10-03: `script/cibuild` builds the image with the same command as - `script/docker`, `--no-cache` included, so the checks in the Dockerfile run on - every build, also on an unchanged tree (#89) -- 2026-10-03: `fetch` removes whatever sits at a file's temp name and then - creates the temp file only if that name is free, so a hard link left there - cannot make it write into a file outside the destination directory (#115) -- 2026-10-03: `fetch` refuses any manifest path that runs through a symlink - already in the destination directory, checked before each of its writes - (directories, temp file, rename), so such a symlink cannot send a write - outside it (#86) -- 2026-10-02: a plain `docker build .` of a clone now stamps the tag or short - commit into `mfer version` instead of nothing: `.dockerignore` sends `.git` - (not `.git/config`), and the build stage takes the `VERSION` build argument, - otherwise `git describe --tags --always`, failing if `.git` is present and no - version comes out. `script/docker` is the canonical copy, which passes - `VERSION`; `bin/gitrev.sh` uses `--tags` too (#112) -- 2026-09-21: validate manifest entry paths on deserialize so untrusted `.mf` - files cannot make `Checker` stat or read outside `basePath` (#61) -- 2026-09-21: rewrote `script/test` to the canonical pattern (30s timeout, - `-race -cover`, quiet-first with verbose-on-failure rerun) and fixed the - process-global logger data race it surfaced (#67) -- 2026-09-21: added the canonical `.editorconfig`, made `.gitignore` cover - secrets, OS, editor, and Go artifacts, and removed the dead Drone CI - references from `.gitignore` and `bin/gitrev.sh` (#72) -- 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and - `script/fmt-check` one shared prettier file set via `script/prettier`, dropped - the `|| true` that hid prettier failures, and added a node-based Dockerfile - stage so a markdown formatting violation fails `docker build .` (#69) -- 2026-08-07: updated golangci-lint to v2.12.2 everywhere it is pinned - (`Makefile`, `Dockerfile`), added the canonical `.golangci.yml` - (`default: all`), and fixed all resulting lint findings across the codebase -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile - shims, README Entrypoints section -- 2026-07-03: aligned repo tooling, docs, and config with standardized policies - (7d9a138, on chore/align-repo-policies, unmerged) -- 2026-06-28: moved to standardized repo policies (#56, on main) -- 2026-04-07: added 1.0 roadmap as README TODO section, removed old TODO.md - (#54) -- 2026-03-20: added Gitea Actions CI workflow (#53) -- 2026-03-17: added REPO_POLICIES.md, renamed CLAUDE.md to AGENTS.md (#51); - removed committed .index.mf (#52) -- 2026-03-15: split Dockerfile with pre-built golangci-lint stage for faster CI - (#45) -- 2026-03-01: 1.0 quality polish: code review, tests, bug fixes, docs (#32) -- 2026-02-20: deterministic file ordering in Builder.Build() (#28); removed - committed vendor/modcache archives (#35) -- 2026-02-08: added --seed flag for deterministic manifest UUID - -# Future Steps - -- Compliance (fold of TODO.md audit 2026-07-02; verify which items the in-flight - branch already closes, then check off): - - Add .editorconfig (canonical copy from sneak/prompts) - - Make .gitignore cover secrets (.env, _.key, _.pem), OS files (.DS_Store), - and editor files (_.swp, _~) - - Make fmt-check/lint verify with gofumpt, not gofmt -l, so `make check` - matches what `make fmt` writes - - Add README "Getting Started" section with copy-pasteable install/usage - block - - Move FORMAT.md from repo root to docs/ and update the AGENTS.md reference - - Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`, - `golangci-lint@v2.12.2`) by module hash, not mutable tag - - Add explicit README "Rationale" heading (content exists under other - names); name the author in the README Description first line - - Reconcile root-level AGENTS.md with directory-hygiene policy (keep or - relocate) - - Add a `make build` target - - Rewrite `make hooks` to use printf or a heredoc instead of non-portable - `echo '...\n...'` -- Answer the 14 owner design questions in the README 1.0 roadmap: - - Format: simplify MFFileChecksum; store file mode; drop atime; specify path - normalization rules; version byte after magic; length-prefix after magic - - Signatures: hash covers compressed or uncompressed data; sign raw bytes vs - hex canonical string; detached .mf.sig support; GPG subprocess vs pure-Go - crypto - - Implementation: deterministic manifests by default; consolidate duplicate - scanner/checker implementations; export the manifest type; canonical Go - module path for 1.0 -- Format and correctness: - - Resolve proto go_package vs go.mod module path inconsistency - - Specify and validate path invariants (UTF-8, forward-slash, relative, no - .., no leading /) - - Remove or deprecate atime; reserve mode field; add version byte (all - pending design answers) - - Write a standalone format specification document -- Library: - - Delete internal/scanner and internal/checker; consolidate on the mfer/ - package versions (pending design answer) - - Add decompression size limit via io.LimitReader in deserializeInner() - - Fix errors.Is dead code in checker; make AddFile verify totalRead == size - - Export manifest type or define a public interface (pending) - - Replace GPG subprocess with pure-Go crypto (pending); add timeouts to - remaining subprocess calls -- CLI: - - Kebab-case primary flag names; fix fetch URL construction with - url.JoinPath; add http.Client timeout and retry with backoff to fetch; - rate-limit Checker progress output; add --deterministic flag or default; - wire top-level --version properly -- Testing: - - Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch -- Documentation: - - Promote docs/FORMAT.md as primary spec reference; audit error messages; - document the signature scheme fully -- Release: - - Finalize module path, bump version constant, SemVer --version output, tag - v1.0.0