Remove TODO.md; track open work in the issues (closes #76)

TODO.md is deleted and the README TODO section now only points to the issue tracker, which is where open work and open design questions live from now on. The section heading stays because the shared repo policy requires it. AGENTS.md points at the tracker too. Every open item in both lists was checked against the code and the tracker: each was either already done or already owned by an issue, apart from one, now filed as #121. The 14 design questions are all on #81, #82, #83 and #79. Model: opus-5-5
2026-10-03 23:46:30 +00:00
parent c31796998f
commit 3660173120
3 changed files with 6 additions and 337 deletions
@@ -27,5 +27,6 @@ source for coding standards, formatting, linting, and workflow rules.
 - The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are
  committed (required for `go get` compatibility).
 - The format specification is in `FORMAT.md`.
- See the TODO section in `README.md` for the 1.0 implementation plan and open
+- Open work, open design questions included, is tracked only in the repo's
-  design questions.
+  issues: https://git.eeqj.de/sneak/mfer/issues. There is no `TODO.md` and no
  TODO list in `README.md`; do not add either.
@@ -246,207 +246,10 @@ regardless of filesystem format.
 Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired
 username for an account on this Gitea instance.
-# TODO: Remaining Work for 1.0
+# TODO
-## Design Questions (Owner Decision Required)
+Open work, open design questions included, is tracked in this repo's issues:
-
+[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues).
 These require @sneak's input before implementation. Answers should be added
 inline below each question.
 ### Format Design
 **1. Should `MFFileChecksum` be simplified?** Currently it's a separate message
 wrapping a single `bytes multiHash` field. Since multihash already
 self-describes the algorithm, `repeated bytes hashes` directly on `MFFilePath`
 would be simpler and reduce per-file protobuf overhead. Is the extra message
 layer intentional (e.g. planning to add per-hash metadata like `verified_at`)?
 > _answer:_
 **2. Should file permissions/mode be stored?** The format stores mtime/ctime but
 not Unix file permissions. For archival use this may not matter, but for
 software distribution or filesystem restoration it's a gap. Should we reserve a
 field now (e.g. `optional uint32 mode = 305`) even if we don't populate it yet?
 > _answer:_
 **3. Should `atime` be removed from the schema?** Access time is volatile,
 non-deterministic, and often disabled (`noatime`). Including it means two
 manifests of the same directory at different times will differ, which conflicts
 with the determinism goal. Remove it, or document it as "never set by default"?
 > _answer:_
 **4. What are the path normalization rules?** The proto has `string path` with
 no specification about: always forward-slash? Must be relative? No `..`
 components allowed? UTF-8 NFC vs NFD normalization (macOS vs Linux)? Max path
 length? This is a security issue (path traversal) and a cross-platform
 compatibility issue. What rules should the spec mandate?
 > _answer:_
 **5. Should we add a version byte after the magic?** Currently `ZNAVSRFG` is
 followed immediately by protobuf. Adding a version byte (`ZNAVSRFG\x01`) would
 allow future framing changes without requiring protobuf parsing to detect the
 version. `MFFileOuter.Version` serves this purpose but requires successful
 deserialization to read. Worth the extra byte?
 > _answer:_
 **6. Should we add a length-prefix after the magic?** Protobuf is not
 self-delimiting. If we ever want to concatenate manifests or append data after
 the protobuf, the current framing is insufficient. Add a varint or fixed-width
 length-prefix?
 > _answer:_
 ### Signature Design
 **7. What does the outer SHA-256 hash cover — compressed or uncompressed data?**
 The code currently hashes compressed data (good for verifying before
 decompression), but this should be explicitly documented. Which is the intended
 behavior?
 > _answer:_
 **8. Should `signatureString()` sign raw bytes instead of a hex-encoded
 string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with hex
 encoding, which adds a transformation layer. Signing the raw `sha256` bytes (or
 compressed `innerMessage` directly) would be simpler. Keep the string format or
 switch to raw bytes?
 > _answer:_
 **9. Should we support detached signature files (`.mf.sig`)?** Embedded
 signatures are better for single-file distribution. Detached `.mf.sig` files
 follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are simpler for HTTP
 serving. Support both modes?
 > _answer:_
 **10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is fragile
 (may not be installed, version-dependent output).
 `github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could use
 Ed25519/signify (simpler, no key management). Which direction?
 > _answer:_
 ### Implementation Design
 **11. Should manifests be deterministic by default?** This means: sort file
 entries by path, omit `createdAt` timestamp (or make it opt-in), no `atime`.
 Should determinism be the default, with a `--include-timestamps` flag to opt in?
 > _answer:_
 **12. Should we consolidate or keep both scanner/checker implementations?**
 There are two parallel implementations: `mfer/scanner.go` + `mfer/checker.go`
 (typed with `FileSize`, `RelFilePath`) and `internal/scanner/` +
 `internal/checker/` (raw `int64`, `string`). The `mfer/` versions are superior.
 Delete the `internal/` versions?
 > _answer:_
 **13. Should the `manifest` type be exported?** Currently unexported with
 exported constructors (`NewManifestFromReader`, `NewManifestFromFile`).
 Consumers can't declare `var m *mfer.manifest`. Export the type, or define an
 interface?
 > _answer:_
 **14. What should the Go module path be for 1.0?** Currently
 `sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in the
 proto `go_package` option. Which is canonical?
 > _answer:_
 ## Implementation Tasks
 ### Repo Infrastructure
 - [ ] Add `.golangci.yml` (fetch from
      `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`)
 - [ ] Add `.editorconfig`
 - [ ] Add `.gitea/workflows/check.yml` that runs `docker build .`
 ### Format & Correctness
 - [ ] Resolve proto `go_package` path inconsistency
      (`git.eeqj.de/sneak/mfer/mfer` vs `sneak.berlin/go/mfer`)
 - [ ] Specify path invariants — add proto comments requiring UTF-8,
      forward-slash, relative paths, no `..`, no leading `/`; validate in
      `Builder.AddFile` and `Builder.AddFileWithHash` (pending design question
      answer)
 - [ ] Remove or deprecate `atime` from proto (pending design question answer)
 - [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future file
      permissions (pending design question answer)
 - [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version 1
      (pending design question answer)
 - [ ] Write format specification document — separate from README: magic, outer
      structure, compression, inner structure, path invariants, signature
      scheme, canonical serialization
 ### Library
 - [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate on
      `mfer/` package versions; update CLI code (pending design question answer)
 - [ ] Add deterministic file ordering — sort entries by path (lexicographic,
      byte-order) in `Builder.Build()`; add test asserting byte-identical output
      from two runs
 - [ ] Add decompression size limit — `io.LimitReader` in `deserializeInner()`
      with `m.pbOuter.Size` as bound
 - [ ] Fix `errors.Is` dead code in checker — replace with `os.IsNotExist(err)`
      or `errors.Is(err, fs.ErrNotExist)`
 - [ ] Fix `AddFile` to verify size — check `totalRead == size` after reading,
      return error on mismatch
 - [ ] Export the `manifest` type or define a public interface (pending design
      question answer) — currently consumers cannot hold a reference to a loaded
      manifest in their own type declarations
 - [ ] Replace GPG subprocess calls with pure-Go crypto (pending design question
      answer) — current implementation shells out to `gpg` which may not be
      installed
 - [ ] Add timeout to any remaining subprocess calls
 ### CLI
 - [ ] Fix flag naming — all CLI flags should use kebab-case as primary
      (`--include-dotfiles`, `--follow-symlinks`)
 - [ ] Fix URL construction in fetch — use `BaseURL.JoinPath()` or
      `url.JoinPath()` instead of string concatenation
 - [ ] Add progress rate-limiting to Checker — throttle to once per second,
      matching Scanner
 - [ ] Add `--deterministic` flag or make it default — omit `createdAt`, sort
      files (pending design question answer)
 - [ ] Wire `--version` flag properly (currently only a `version` subcommand
      exists; top-level `--version` shows urfave/cli generic output)
 - [ ] Add retry logic to `fetch` — currently no retries on transient HTTP
      errors; needs exponential backoff
 - [ ] `fetch` command uses bare `http.Get` with no timeout — needs `http.Client`
      with configurable timeout
 ### Testing & Robustness
 - [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf deserialization
      of untrusted input needs fuzz coverage
 - [ ] Add integration test for `freshen` CLI command — current tests only verify
      setup, not the actual freshen operation end-to-end
 - [ ] Add test for `fetch` CLI command end-to-end (currently only `downloadFile`
      is tested)
 ### Documentation
 - [ ] Promote `FORMAT.md` as primary spec reference; README should link to it
      more prominently
 - [ ] Audit and update all error messages for consistency and helpfulness
 - [ ] Document the signature scheme more thoroughly (canonical string format,
      verification steps)
 ### Release
 - [ ] Finalize Go module path
 - [ ] Update version constant in `mfer/constants.go`
 - [ ] Add `--version` output matching SemVer
 - [ ] Tag `v1.0.0`
 # See Also
@@ -1,135 +0,0 @@
 # Workflow
 - branch (from `main`)
 - do the work in Next Step
 - move Next Step to the top of Completed Steps
 - move the top item of Future Steps into Next Step
 - commit (`TODO.md` changes in the same commit as the work)
 - merge to `main` if the branch is not protected, otherwise open a PR
 - push
 # Status
 pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists open
 design questions and implementation tasks; policy compliance work is in flight
 and unmerged.
 # Next Step
 Work through the remaining compliance items folded from the 2026-07-02 audit
 (the first group under Future Steps): `.editorconfig`, `.gitignore` coverage,
 gofumpt-based `fmt-check`, README "Getting Started", and the rest.
 `.golangci.yml` and `TODO.md` are tracked and committed as of 2026-08-07, so the
 only thing left of the `chore/align-repo-policies` branch is the list below.
 # Completed Steps
 - 2026-10-03: pinned the CLI error messages by driving the functions that emit
  them in `internal/cli/errmsg_test.go`, and made the freshen mtime-presence
  test distinguish an absent mtime from the epoch (#87)
 - 2026-10-03: `script/cibuild` builds the image with the same command as
  `script/docker`, `--no-cache` included, so the checks in the Dockerfile run on
  every build, also on an unchanged tree (#89)
 - 2026-10-03: `fetch` removes whatever sits at a file's temp name and then
  creates the temp file only if that name is free, so a hard link left there
  cannot make it write into a file outside the destination directory (#115)
 - 2026-10-03: `fetch` refuses any manifest path that runs through a symlink
  already in the destination directory, checked before each of its writes
  (directories, temp file, rename), so such a symlink cannot send a write
  outside it (#86)
 - 2026-10-02: a plain `docker build .` of a clone now stamps the tag or short
  commit into `mfer version` instead of nothing: `.dockerignore` sends `.git`
  (not `.git/config`), and the build stage takes the `VERSION` build argument,
  otherwise `git describe --tags --always`, failing if `.git` is present and no
  version comes out. `script/docker` is the canonical copy, which passes
  `VERSION`; `bin/gitrev.sh` uses `--tags` too (#112)
 - 2026-09-21: validate manifest entry paths on deserialize so untrusted `.mf`
  files cannot make `Checker` stat or read outside `basePath` (#61)
 - 2026-09-21: rewrote `script/test` to the canonical pattern (30s timeout,
  `-race -cover`, quiet-first with verbose-on-failure rerun) and fixed the
  process-global logger data race it surfaced (#67)
 - 2026-09-21: added the canonical `.editorconfig`, made `.gitignore` cover
  secrets, OS, editor, and Go artifacts, and removed the dead Drone CI
  references from `.gitignore` and `bin/gitrev.sh` (#72)
 - 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and
  `script/fmt-check` one shared prettier file set via `script/prettier`, dropped
  the `|| true` that hid prettier failures, and added a node-based Dockerfile
  stage so a markdown formatting violation fails `docker build .` (#69)
 - 2026-08-07: updated golangci-lint to v2.12.2 everywhere it is pinned
  (`Makefile`, `Dockerfile`), added the canonical `.golangci.yml`
  (`default: all`), and fixed all resulting lint findings across the codebase
 - 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
  shims, README Entrypoints section
 - 2026-07-03: aligned repo tooling, docs, and config with standardized policies
  (7d9a138, on chore/align-repo-policies, unmerged)
 - 2026-06-28: moved to standardized repo policies (#56, on main)
 - 2026-04-07: added 1.0 roadmap as README TODO section, removed old TODO.md
  (#54)
 - 2026-03-20: added Gitea Actions CI workflow (#53)
 - 2026-03-17: added REPO_POLICIES.md, renamed CLAUDE.md to AGENTS.md (#51);
  removed committed .index.mf (#52)
 - 2026-03-15: split Dockerfile with pre-built golangci-lint stage for faster CI
  (#45)
 - 2026-03-01: 1.0 quality polish: code review, tests, bug fixes, docs (#32)
 - 2026-02-20: deterministic file ordering in Builder.Build() (#28); removed
  committed vendor/modcache archives (#35)
 - 2026-02-08: added --seed flag for deterministic manifest UUID
 # Future Steps
 - Compliance (fold of TODO.md audit 2026-07-02; verify which items the in-flight
  branch already closes, then check off):
    - Add .editorconfig (canonical copy from sneak/prompts)
    - Make .gitignore cover secrets (.env, _.key, _.pem), OS files (.DS_Store),
      and editor files (_.swp, _~)
    - Make fmt-check/lint verify with gofumpt, not gofmt -l, so `make check`
      matches what `make fmt` writes
    - Add README "Getting Started" section with copy-pasteable install/usage
      block
    - Move FORMAT.md from repo root to docs/ and update the AGENTS.md reference
    - Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`,
      `golangci-lint@v2.12.2`) by module hash, not mutable tag
    - Add explicit README "Rationale" heading (content exists under other
      names); name the author in the README Description first line
    - Reconcile root-level AGENTS.md with directory-hygiene policy (keep or
      relocate)
    - Add a `make build` target
    - Rewrite `make hooks` to use printf or a heredoc instead of non-portable
      `echo '...\n...'`
 - Answer the 14 owner design questions in the README 1.0 roadmap:
    - Format: simplify MFFileChecksum; store file mode; drop atime; specify path
      normalization rules; version byte after magic; length-prefix after magic
    - Signatures: hash covers compressed or uncompressed data; sign raw bytes vs
      hex canonical string; detached .mf.sig support; GPG subprocess vs pure-Go
      crypto
    - Implementation: deterministic manifests by default; consolidate duplicate
      scanner/checker implementations; export the manifest type; canonical Go
      module path for 1.0
 - Format and correctness:
    - Resolve proto go_package vs go.mod module path inconsistency
    - Specify and validate path invariants (UTF-8, forward-slash, relative, no
      .., no leading /)
    - Remove or deprecate atime; reserve mode field; add version byte (all
      pending design answers)
    - Write a standalone format specification document
 - Library:
    - Delete internal/scanner and internal/checker; consolidate on the mfer/
      package versions (pending design answer)
    - Add decompression size limit via io.LimitReader in deserializeInner()
    - Fix errors.Is dead code in checker; make AddFile verify totalRead == size
    - Export manifest type or define a public interface (pending)
    - Replace GPG subprocess with pure-Go crypto (pending); add timeouts to
      remaining subprocess calls
 - CLI:
    - Kebab-case primary flag names; fix fetch URL construction with
      url.JoinPath; add http.Client timeout and retry with backoff to fetch;
      rate-limit Checker progress output; add --deterministic flag or default;
      wire top-level --version properly
 - Testing:
    - Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch
 - Documentation:
    - Promote docs/FORMAT.md as primary spec reference; audit error messages;
      document the signature scheme fully
 - Release:
    - Finalize module path, bump version constant, SemVer --version output, tag
      v1.0.0