Add TODO section to README with 1.0 roadmap, remove TODO.md #54

Open
clawbot wants to merge 2 commits from 47-add-todo-section-to-readme into main
Collaborator

Summary

Performs a design and status review of the codebase and adds a comprehensive TODO section to README.md listing remaining work for a 1.0 release.

What changed

  • README.md: Added a TODO: Remaining Work for 1.0 section covering:
    • 7 design questions requiring @sneak's input before implementation (manifest type export, Go module path, GPG vs pure-Go crypto, format framing, etc.) — each with an answer field for inline decisions
    • Implementation tasks organized by category: repo infrastructure, format & correctness, library, CLI, testing & robustness, documentation, and release checklist
    • Updated build status section (removed stale Drone CI badge, replaced with description of current Docker-based CI)
  • TODO.md: Removed — items integrated into README TODO section
  • AGENTS.md: Updated reference from TODO.md to README TODO section

Design review findings

What works well:

  • Core library (Builder, Scanner, Checker) is solid with good test coverage
  • Format specification is well-designed (protobuf + zstd, multihash, deterministic serialization)
  • CLI covers all major operations (gen, check, list, export, freshen, fetch)
  • Test suite is thorough — builder, scanner, checker, GPG, CLI integration, corruption detection
  • afero abstraction enables clean testing without filesystem side effects

Key gaps for 1.0:

  • Missing repo infrastructure (.golangci.yml, .editorconfig, CI workflow)
  • manifest type is unexported — consumers can't use it in their own type declarations
  • GPG signing shells out to gpg subprocess — fragile and may not be installed
  • Go module path inconsistency between go.mod and proto go_package
  • fetch command lacks retry logic and has no HTTP timeout
  • Missing fuzz tests for untrusted input deserialization
  • Freshen CLI command has incomplete integration test coverage

closes #47
closes #50

## Summary Performs a design and status review of the codebase and adds a comprehensive TODO section to `README.md` listing remaining work for a 1.0 release. ### What changed - **README.md**: Added a `TODO: Remaining Work for 1.0` section covering: - **7 design questions** requiring @sneak's input before implementation (manifest type export, Go module path, GPG vs pure-Go crypto, format framing, etc.) — each with an answer field for inline decisions - **Implementation tasks** organized by category: repo infrastructure, format & correctness, library, CLI, testing & robustness, documentation, and release checklist - Updated build status section (removed stale Drone CI badge, replaced with description of current Docker-based CI) - **TODO.md**: Removed — items integrated into README TODO section - **AGENTS.md**: Updated reference from `TODO.md` to README TODO section ### Design review findings **What works well:** - Core library (Builder, Scanner, Checker) is solid with good test coverage - Format specification is well-designed (protobuf + zstd, multihash, deterministic serialization) - CLI covers all major operations (gen, check, list, export, freshen, fetch) - Test suite is thorough — builder, scanner, checker, GPG, CLI integration, corruption detection - afero abstraction enables clean testing without filesystem side effects **Key gaps for 1.0:** - Missing repo infrastructure (`.golangci.yml`, `.editorconfig`, CI workflow) - `manifest` type is unexported — consumers can't use it in their own type declarations - GPG signing shells out to `gpg` subprocess — fragile and may not be installed - Go module path inconsistency between `go.mod` and proto `go_package` - `fetch` command lacks retry logic and has no HTTP timeout - Missing fuzz tests for untrusted input deserialization - Freshen CLI command has incomplete integration test coverage closes #47 closes #50
clawbot added 1 commit 2026-03-17 09:45:47 +01:00
Perform design and status review of the codebase. Add a comprehensive
TODO section to README.md covering design questions requiring owner
input, implementation tasks (repo infrastructure, format, library, CLI,
testing, documentation), and release checklist.

Integrate items from TODO.md into the README TODO section and remove
TODO.md from the repo. Update AGENTS.md to reference the new location.
clawbot added the needs-reviewbot labels 2026-03-17 09:46:46 +01:00
Author
Collaborator

Code Review: PR #54

Policy Divergences

No REPO_POLICIES.md violations found in the changed files themselves. The Dockerfile base images remain pinned by sha256, no linter/CI config was modified, and no test assertions were weakened.

Requirements Checklist

Issue #47: Add TODO section to README

Requirement Status
Add TODO section to README Met
Missing features for 1.0 Met
Known bugs or incomplete implementations Incomplete — many known bugs/issues from TODO.md were dropped
Architectural issues Incomplete — scanner/checker consolidation question dropped
Documentation gaps Met
Concrete and actionable Met

Issue #50: Integrate ALL items from TODO.md into README, remove TODO.md

Requirement Status
Integrate all items from TODO.md Not met — significant items dropped (see below)
Remove TODO.md Met

Critical Finding: TODO.md Items Dropped

The PR claims closes #50, but issue #50 explicitly requires "Integrate all items from TODO.md." The PR drops roughly half the content.

7 of 14 design questions dropped:

  1. #3 — Should atime be removed from the schema? (volatile, non-deterministic, conflicts with determinism goal)
  2. #4 — What are the path normalization rules? (security issue — path traversal, cross-platform compatibility)
  3. #7 — What does the outer SHA-256 hash cover — compressed or uncompressed?
  4. #8 — Should signatureString() sign raw bytes instead of hex?
  5. #9 — Should we support detached signature files (.mf.sig)?
  6. #11 — Should manifests be deterministic by default?
  7. #12 — Should we consolidate or keep both scanner/checker implementations?

12+ implementation tasks dropped entirely:

Phase 1 (all 6 items missing):

  • Delete internal/scanner/ and internal/checker/ — consolidate on mfer/ package
  • Add deterministic file ordering in Builder.Build()
  • Add decompression size limit (io.LimitReader in deserializeInner())
  • Fix errors.Is dead code in checker
  • Fix AddFile to verify size matches totalRead
  • Specify path invariants (UTF-8, forward-slash, relative, no ..)

Phase 2 (all 4 items missing):

  • Fix flag naming — all CLI flags use kebab-case
  • Fix URL construction in fetch — use url.JoinPath() instead of string concatenation
  • Add progress rate-limiting to Checker
  • Add --deterministic flag or make it default

Phase 3 (1 missing):

  • Add timeout to any remaining subprocess calls

Phase 4 (1 missing):

  • Remove or deprecate atime from proto

Build Result

docker build . passes — all tests, linting, and formatting pass.

Verdict: FAIL

The PR drops approximately half the design questions and a majority of the implementation tasks from TODO.md while claiming closes #50, which requires integrating all items. This is not a minor omission — entire phases of concrete, actionable work items (Phase 1 foundation and Phase 2 CLI polish) are missing from the README TODO section. The items dropped include security-relevant concerns (path normalization rules, decompression size limits) and known bugs (dead code, size verification).

All dropped items must be integrated into the README TODO section, or the closes #50 claim must be removed.

## Code Review: [PR #54](https://git.eeqj.de/sneak/mfer/pulls/54) ### Policy Divergences No REPO_POLICIES.md violations found in the changed files themselves. The Dockerfile base images remain pinned by sha256, no linter/CI config was modified, and no test assertions were weakened. ### Requirements Checklist **[Issue #47](https://git.eeqj.de/sneak/mfer/issues/47): Add TODO section to README** | Requirement | Status | |---|---| | Add TODO section to README | ✅ Met | | Missing features for 1.0 | ✅ Met | | Known bugs or incomplete implementations | ❌ Incomplete — many known bugs/issues from TODO.md were dropped | | Architectural issues | ❌ Incomplete — scanner/checker consolidation question dropped | | Documentation gaps | ✅ Met | | Concrete and actionable | ✅ Met | **[Issue #50](https://git.eeqj.de/sneak/mfer/issues/50): Integrate ALL items from TODO.md into README, remove TODO.md** | Requirement | Status | |---|---| | Integrate all items from TODO.md | ❌ **Not met** — significant items dropped (see below) | | Remove TODO.md | ✅ Met | ### Critical Finding: TODO.md Items Dropped The PR claims `closes #50`, but [issue #50](https://git.eeqj.de/sneak/mfer/issues/50) explicitly requires "Integrate **all** items from `TODO.md`." The PR drops roughly half the content. **7 of 14 design questions dropped:** 1. **#3 — Should `atime` be removed from the schema?** (volatile, non-deterministic, conflicts with determinism goal) 2. **#4 — What are the path normalization rules?** (security issue — path traversal, cross-platform compatibility) 3. **#7 — What does the outer SHA-256 hash cover — compressed or uncompressed?** 4. **#8 — Should `signatureString()` sign raw bytes instead of hex?** 5. **#9 — Should we support detached signature files (`.mf.sig`)?** 6. **#11 — Should manifests be deterministic by default?** 7. **#12 — Should we consolidate or keep both scanner/checker implementations?** **12+ implementation tasks dropped entirely:** Phase 1 (all 6 items missing): - Delete `internal/scanner/` and `internal/checker/` — consolidate on `mfer/` package - Add deterministic file ordering in `Builder.Build()` - Add decompression size limit (`io.LimitReader` in `deserializeInner()`) - Fix `errors.Is` dead code in checker - Fix `AddFile` to verify size matches totalRead - Specify path invariants (UTF-8, forward-slash, relative, no `..`) Phase 2 (all 4 items missing): - Fix flag naming — all CLI flags use kebab-case - Fix URL construction in fetch — use `url.JoinPath()` instead of string concatenation - Add progress rate-limiting to Checker - Add `--deterministic` flag or make it default Phase 3 (1 missing): - Add timeout to any remaining subprocess calls Phase 4 (1 missing): - Remove or deprecate `atime` from proto ### Build Result ✅ `docker build .` passes — all tests, linting, and formatting pass. ### Verdict: **FAIL** The PR drops approximately half the design questions and a majority of the implementation tasks from TODO.md while claiming `closes #50`, which requires integrating **all** items. This is not a minor omission — entire phases of concrete, actionable work items (Phase 1 foundation and Phase 2 CLI polish) are missing from the README TODO section. The items dropped include security-relevant concerns (path normalization rules, decompression size limits) and known bugs (dead code, size verification). All dropped items must be integrated into the README TODO section, or the `closes #50` claim must be removed.
clawbot added needs-rework and removed needs-review labels 2026-03-17 09:50:36 +01:00
clawbot added 1 commit 2026-03-17 09:58:30 +01:00
Add all 14 design questions (7 were missing: atime removal, path
normalization, outer SHA-256 scope, signatureString format, detached
signatures, deterministic-by-default, scanner/checker consolidation).

Add all missing implementation tasks from TODO.md phases 1-4:
- Phase 1 foundation: consolidate internal packages, deterministic
  ordering, decompression size limit, errors.Is fix, AddFile size
  verification, path invariants
- Phase 2 CLI: flag naming, URL construction, progress rate-limiting,
  deterministic flag
- Phase 3: subprocess timeouts
- Phase 4: atime deprecation

Organize design questions with sub-headers (Format Design, Signature
Design, Implementation Design) matching original TODO.md structure.
Author
Collaborator

Rework Summary

Fixed all review findings from the failed review. The README TODO section now integrates all items from TODO.md.

Changes

Design Questions — added 7 missing questions (now 14 total):

  • #3: Should atime be removed from the schema?
  • #4: What are the path normalization rules?
  • #7: What does the outer SHA-256 hash cover — compressed or uncompressed data?
  • #8: Should signatureString() sign raw bytes instead of hex?
  • #9: Should we support detached signature files (.mf.sig)?
  • #11: Should manifests be deterministic by default?
  • #12: Should we consolidate or keep both scanner/checker implementations?

Design questions are now organized with sub-headers matching the original TODO.md structure: Format Design, Signature Design, Implementation Design.

Implementation Tasks — added all 12+ missing items:

Format & Correctness:

  • Specify path invariants (UTF-8, forward-slash, relative, no ..)
  • Remove or deprecate atime from proto
  • Write format specification document

Library (6 new items):

  • Delete internal/scanner/ and internal/checker/ — consolidate on mfer/ package
  • Add deterministic file ordering in Builder.Build()
  • Add decompression size limit (io.LimitReader in deserializeInner())
  • Fix errors.Is dead code in checker
  • Fix AddFile to verify size matches totalRead
  • Add timeout to any remaining subprocess calls

CLI (4 new items):

  • Fix flag naming — kebab-case
  • Fix URL construction in fetch — url.JoinPath()
  • Add progress rate-limiting to Checker
  • Add --deterministic flag or make it default (updated from documentation-only to full implementation scope)

Build

docker build . passes — all tests, linting, and formatting pass.

Verification

All 14 design questions present (was 7). 34 implementation tasks total. Every item from the original TODO.md is now integrated into the README TODO section, satisfying issue #50's requirement to integrate all items.

## Rework Summary Fixed all review findings from the failed review. The README TODO section now integrates **all** items from `TODO.md`. ### Changes **Design Questions — added 7 missing questions (now 14 total):** - #3: Should `atime` be removed from the schema? - #4: What are the path normalization rules? - #7: What does the outer SHA-256 hash cover — compressed or uncompressed data? - #8: Should `signatureString()` sign raw bytes instead of hex? - #9: Should we support detached signature files (`.mf.sig`)? - #11: Should manifests be deterministic by default? - #12: Should we consolidate or keep both scanner/checker implementations? Design questions are now organized with sub-headers matching the original `TODO.md` structure: **Format Design**, **Signature Design**, **Implementation Design**. **Implementation Tasks — added all 12+ missing items:** *Format & Correctness:* - Specify path invariants (UTF-8, forward-slash, relative, no `..`) - Remove or deprecate `atime` from proto - Write format specification document *Library (6 new items):* - Delete `internal/scanner/` and `internal/checker/` — consolidate on `mfer/` package - Add deterministic file ordering in `Builder.Build()` - Add decompression size limit (`io.LimitReader` in `deserializeInner()`) - Fix `errors.Is` dead code in checker - Fix `AddFile` to verify size matches `totalRead` - Add timeout to any remaining subprocess calls *CLI (4 new items):* - Fix flag naming — kebab-case - Fix URL construction in fetch — `url.JoinPath()` - Add progress rate-limiting to Checker - Add `--deterministic` flag or make it default (updated from documentation-only to full implementation scope) ### Build ✅ `docker build .` passes — all tests, linting, and formatting pass. ### Verification All 14 design questions present (was 7). 34 implementation tasks total. Every item from the original `TODO.md` is now integrated into the README TODO section, satisfying [issue #50](https://git.eeqj.de/sneak/mfer/issues/50)'s requirement to integrate **all** items.
clawbot added needs-review and removed needs-rework labels 2026-03-17 09:59:20 +01:00
clawbot reviewed 2026-03-17 10:13:01 +01:00
clawbot left a comment
Author
Collaborator

Re-Review: PR #54 (post-rework)

Policy Divergences

No policy violations found. No changes to Makefile, Dockerfile, .golangci.yml, CI config, or test files. All new README content respects the 80-column prose wrap convention. Changed files are limited to README.md, AGENTS.md, and removal of TODO.md — all documentation-only.

Requirements Checklist

Issue #47: Add TODO section to README for 1.0 release

Requirement Status
Add TODO section to README Met
Missing features for 1.0 Met — 34 implementation tasks across 7 categories
Known bugs or incomplete implementations Met — errors.Is dead code, AddFile size verification, URL construction, HTTP timeout, etc.
Architectural issues Met — scanner/checker consolidation, manifest type export, module path inconsistency
Documentation gaps Met — FORMAT.md promotion, signature scheme docs, error message audit
Concrete and actionable Met — each item specifies the specific code location, function, or file to change

Issue #50: Integrate ALL items from TODO.md into README, remove TODO.md

Requirement Status
Integrate all items from TODO.md Met (see audit below)
Remove TODO.md Met

TODO.md Coverage Audit

Item-by-item comparison of git show main:TODO.md against the new README TODO section:

Design Questions (14/14 present):

# Question In README?
1 MFFileChecksum simplification
2 File permissions/mode storage (minor text compression: dropped "ExFAT, filesystem-independent checksums" parenthetical, but substance preserved)
3 atime removal from schema
4 Path normalization rules
5 Version byte after magic
6 Length-prefix after magic
7 Outer SHA-256 hash scope (rewording: "The review notes" → "The code currently" — substance identical)
8 signatureString raw bytes vs hex
9 Detached signature files (.mf.sig)
10 GPG vs pure-Go crypto
11 Deterministic by default
12 Scanner/checker consolidation
13 Manifest type export (constructor names updated to match actual code: NewManifestFromReader, NewManifestFromFile — more accurate than TODO.md's New, NewFromPaths)
14 Go module path for 1.0 (added specifics about proto go_package path)

Implementation Tasks (22/22 from TODO.md present):

TODO.md Phase Item README Location
Phase 1 Delete internal/scanner/ and internal/checker/ Library
Phase 1 Deterministic file ordering in Builder.Build() Library
Phase 1 Decompression size limit (io.LimitReader) Library
Phase 1 Fix errors.Is dead code in checker Library
Phase 1 Fix AddFile size verification Library
Phase 1 Specify path invariants Format & Correctness
Phase 2 Fix flag naming (kebab-case) CLI
Phase 2 Fix URL construction in fetch CLI
Phase 2 Progress rate-limiting for Checker CLI
Phase 2 --deterministic flag CLI
Phase 3 Replace GPG subprocess with pure-Go crypto Library
Phase 3 Add timeout to subprocess calls Library
Phase 3 Fuzzing tests for NewManifestFromReader Testing
Phase 3 Retry logic for fetch CLI
Phase 4 Remove/deprecate atime from proto Format & Correctness
Phase 4 Reserve mode field (uint32 mode = 305) Format & Correctness
Phase 4 Version byte after magic Format & Correctness
Phase 4 Write format specification document Format & Correctness
Phase 5 Finalize Go module path Release
Phase 5 Audit all error messages Documentation
Phase 5 --version output matching SemVer Release
Phase 5 Tag v1.0.0 Release

Missing items from TODO.md: NONE.

The README also adds 12 new items beyond TODO.md (repo infrastructure, proto path inconsistency, version flag wiring, HTTP timeout, integration tests, documentation, version constant). These are legitimate additions from the design review per issue #47's scope.

Additional Checks

  • AGENTS.md reference updated: TODO.mdREADME.md TODO section
  • Build status section: Stale Drone CI badge replaced with accurate description of Docker-based CI
  • No cheating: No modifications to Makefile, Dockerfile, .golangci.yml, CI workflows, or test files
  • No unaddressed human comments: No human comments exist on the PR or linked issues
  • Markdown formatting: All new lines within 80-column limit

Build Result

docker build . passes — all tests, linting, and formatting pass.

Verdict: PASS

The rework successfully addressed every finding from the first review. All 14 design questions and all 22 implementation tasks from TODO.md are now integrated into the README TODO section. The additional items from the design review are appropriate additions per issue #47's scope. TODO.md is removed. No policy violations. Build is green.

## Re-Review: [PR #54](https://git.eeqj.de/sneak/mfer/pulls/54) (post-rework) ### Policy Divergences No policy violations found. No changes to Makefile, Dockerfile, `.golangci.yml`, CI config, or test files. All new README content respects the 80-column prose wrap convention. Changed files are limited to `README.md`, `AGENTS.md`, and removal of `TODO.md` — all documentation-only. ### Requirements Checklist **[Issue #47](https://git.eeqj.de/sneak/mfer/issues/47): Add TODO section to README for 1.0 release** | Requirement | Status | |---|---| | Add TODO section to README | ✅ Met | | Missing features for 1.0 | ✅ Met — 34 implementation tasks across 7 categories | | Known bugs or incomplete implementations | ✅ Met — errors.Is dead code, AddFile size verification, URL construction, HTTP timeout, etc. | | Architectural issues | ✅ Met — scanner/checker consolidation, manifest type export, module path inconsistency | | Documentation gaps | ✅ Met — FORMAT.md promotion, signature scheme docs, error message audit | | Concrete and actionable | ✅ Met — each item specifies the specific code location, function, or file to change | **[Issue #50](https://git.eeqj.de/sneak/mfer/issues/50): Integrate ALL items from TODO.md into README, remove TODO.md** | Requirement | Status | |---|---| | Integrate all items from TODO.md | ✅ Met (see audit below) | | Remove TODO.md | ✅ Met | ### TODO.md Coverage Audit Item-by-item comparison of `git show main:TODO.md` against the new README TODO section: **Design Questions (14/14 present):** | # | Question | In README? | |---|---|---| | 1 | MFFileChecksum simplification | ✅ | | 2 | File permissions/mode storage | ✅ (minor text compression: dropped "ExFAT, filesystem-independent checksums" parenthetical, but substance preserved) | | 3 | atime removal from schema | ✅ | | 4 | Path normalization rules | ✅ | | 5 | Version byte after magic | ✅ | | 6 | Length-prefix after magic | ✅ | | 7 | Outer SHA-256 hash scope | ✅ (rewording: "The review notes" → "The code currently" — substance identical) | | 8 | signatureString raw bytes vs hex | ✅ | | 9 | Detached signature files (.mf.sig) | ✅ | | 10 | GPG vs pure-Go crypto | ✅ | | 11 | Deterministic by default | ✅ | | 12 | Scanner/checker consolidation | ✅ | | 13 | Manifest type export | ✅ (constructor names updated to match actual code: `NewManifestFromReader`, `NewManifestFromFile` — more accurate than TODO.md's `New`, `NewFromPaths`) | | 14 | Go module path for 1.0 | ✅ (added specifics about proto `go_package` path) | **Implementation Tasks (22/22 from TODO.md present):** | TODO.md Phase | Item | README Location | |---|---|---| | Phase 1 | Delete internal/scanner/ and internal/checker/ | Library ✅ | | Phase 1 | Deterministic file ordering in Builder.Build() | Library ✅ | | Phase 1 | Decompression size limit (io.LimitReader) | Library ✅ | | Phase 1 | Fix errors.Is dead code in checker | Library ✅ | | Phase 1 | Fix AddFile size verification | Library ✅ | | Phase 1 | Specify path invariants | Format & Correctness ✅ | | Phase 2 | Fix flag naming (kebab-case) | CLI ✅ | | Phase 2 | Fix URL construction in fetch | CLI ✅ | | Phase 2 | Progress rate-limiting for Checker | CLI ✅ | | Phase 2 | --deterministic flag | CLI ✅ | | Phase 3 | Replace GPG subprocess with pure-Go crypto | Library ✅ | | Phase 3 | Add timeout to subprocess calls | Library ✅ | | Phase 3 | Fuzzing tests for NewManifestFromReader | Testing ✅ | | Phase 3 | Retry logic for fetch | CLI ✅ | | Phase 4 | Remove/deprecate atime from proto | Format & Correctness ✅ | | Phase 4 | Reserve mode field (uint32 mode = 305) | Format & Correctness ✅ | | Phase 4 | Version byte after magic | Format & Correctness ✅ | | Phase 4 | Write format specification document | Format & Correctness ✅ | | Phase 5 | Finalize Go module path | Release ✅ | | Phase 5 | Audit all error messages | Documentation ✅ | | Phase 5 | --version output matching SemVer | Release ✅ | | Phase 5 | Tag v1.0.0 | Release ✅ | **Missing items from TODO.md: NONE.** The README also adds 12 new items beyond TODO.md (repo infrastructure, proto path inconsistency, version flag wiring, HTTP timeout, integration tests, documentation, version constant). These are legitimate additions from the design review per [issue #47](https://git.eeqj.de/sneak/mfer/issues/47)'s scope. ### Additional Checks - **AGENTS.md reference updated:** `TODO.md` → `README.md` TODO section ✅ - **Build status section:** Stale Drone CI badge replaced with accurate description of Docker-based CI ✅ - **No cheating:** No modifications to Makefile, Dockerfile, `.golangci.yml`, CI workflows, or test files ✅ - **No unaddressed human comments:** No human comments exist on the PR or linked issues ✅ - **Markdown formatting:** All new lines within 80-column limit ✅ ### Build Result ✅ `docker build .` passes — all tests, linting, and formatting pass. ### Verdict: **PASS** The rework successfully addressed every finding from the first review. All 14 design questions and all 22 implementation tasks from `TODO.md` are now integrated into the README TODO section. The additional items from the design review are appropriate additions per [issue #47](https://git.eeqj.de/sneak/mfer/issues/47)'s scope. `TODO.md` is removed. No policy violations. Build is green.
clawbot added merge-ready and removed needs-review labels 2026-03-17 10:13:34 +01:00
sneak was assigned by clawbot 2026-03-17 10:13:34 +01:00
Some required checks are missing.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin 47-add-todo-section-to-readme:47-add-todo-section-to-readme
git checkout 47-add-todo-section-to-readme
Sign in to join this conversation.
No Reviewers
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/mfer#54