Commit Graph

143 Commits

Author SHA1 Message Date
clawbot
ca93d80f1e fix: address PR #32 review findings
- Add --batch --no-tty to all GPG invocations (fixes TestManifestTamperedSignatureFails hang)
- Add 'reserved 304' to mf.proto for removed atime field
- Restore IncludeDotfiles alias on include-dotfiles flag
- Replace http.Get with http.Client{Timeout: 30s} in manifest_loader.go
2026-02-20 03:52:27 -08:00
clawbot
e27f8a6c3b revert version bump: 1.0.0 back to 0.1.0
Per review feedback — version bumps and releases are not
within scope for this PR.
2026-02-20 03:45:19 -08:00
clawbot
ca3e29e802 docs: add FORMAT.md, answer design questions, bump version to 1.0.0
- Write complete .mf format specification (FORMAT.md)
- Fill in all design question answers in TODO.md
- Mark completed implementation items in TODO.md
- Bump VERSION from 0.1.0 to 1.0.0 in Makefile
- Update README to reference FORMAT.md and reflect 1.0 status
2026-02-20 03:45:19 -08:00
clawbot
472221a7f6 feat: add export command, HTTP URL support, --version flag, error wrapping audit
- Add 'mfer export' command: dumps manifest as JSON to stdout for piping to jq etc
- Add HTTP/HTTPS URL support for manifest path arguments (check, list, export)
- Enable --version flag (was hidden, now shown)
- Audit all error messages: wrap with fmt.Errorf context throughout CLI and library
- Add tests for export command and URL-based manifest loading
- Add manifest_loader.go with shared resolveManifestArg and openManifestReader helpers
2026-02-20 03:45:19 -08:00
clawbot
dacc97d1f6 feat: deterministic manifests by default, remove atime, rate-limit checker progress
- Remove atime field from proto schema (field 304 reserved)
- Omit createdAt timestamp by default for deterministic output
- Add --include-timestamps flag to gen and freshen commands to opt in
- Add Builder.SetIncludeTimestamps() and ScannerOptions.IncludeTimestamps
- Rate-limit Checker progress updates to once per second (matching Scanner)
- Add tests for all changes

Closes design decisions: deterministic-by-default, atime removal.
2026-02-20 03:45:19 -08:00
user
2f0005bf64 Fix BaseURL.JoinPath encoding slashes in paths, add URL tests
JoinPath used url.PathEscape on the entire path which encoded slashes
as %2F. Now encodes each segment individually. Add tests for all URL
types.
2026-02-20 03:44:47 -08:00
user
e18ab550ae Add build instructions to README (closes #9)
Document prerequisites (Go, protoc, golangci-lint, gofumpt), build
commands, and go install instructions.
2026-02-20 03:44:47 -08:00
user
07e0fc166a Expand test coverage: path validation, round-trip, error cases
Add tests for ValidatePath, AddFile size mismatch, invalid paths,
progress reporting, manifest round-trip, invalid magic, truncated
input, empty input, and manifest String() method.
2026-02-20 03:44:47 -08:00
user
a6a72faafb Fix CLI flag naming to use kebab-case as primary names
Change --FollowSymLinks to --follow-symlinks (-L) and
--IncludeDotfiles to --include-dotfiles as primary flag names.
2026-02-20 03:44:47 -08:00
user
8bb70bc6a9 Fix FindExtraFiles reporting manifest file and dotfiles as extra (closes #16)
FindExtraFiles now skips hidden files/directories (dotfiles) and the
manifest file itself when walking the filesystem. The manifest's relative
path is computed at Checker construction time.
2026-02-20 03:43:56 -08:00
user
008f270d90 Fix IsHiddenPath treating current directory as hidden (closes #14)
IsHiddenPath(".") incorrectly returned true because path.Clean(".")
starts with a dot. Add explicit check for "." before the HasPrefix
check. Add test cases for ".", "./", and "./file.txt".
2026-02-20 03:43:56 -08:00
bbab6e73f4 Add deterministic file ordering in Builder.Build() (closes #23) (#28)
Reviewed-on: #28
2026-02-20 12:15:13 +01:00
615eecff79 Merge branch 'next' into fix/issue-23 2026-02-20 12:14:53 +01:00
9b67de016d chore: remove committed vendor/modcache archives (#35)
Removes `vendor.tzst` and `modcache.tzst` that should never have been committed. Adds both to `.gitignore`.

Reviewed-on: #35
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
Co-committed-by: clawbot <sneak+clawbot@sneak.cloud>
2026-02-20 12:14:29 +01:00
clawbot
3c779465e2 remove time-hard hash iteration from seed UUID derivation
Replace 150M SHA-256 iteration key-stretching with a single hash.
Remove all references to iteration counts, timing (~5-10s), and
key-stretching from code and documentation.

The seed flag is retained for deterministic UUID generation, but
now derives the UUID with a single SHA-256 hash instead of the
unnecessary iterative approach.
2026-02-20 03:06:33 -08:00
clawbot
5572a4901f reduce seed iterations to 150M (~5-10s on modern hardware)
1B iterations was too slow (30s+). Benchmarked on Apple Silicon:
- 150M iterations ≈ 6.3s
- Falls within the 5-10s target range
2026-02-20 03:05:16 -08:00
clawbot
2adc275278 feat: add --seed flag for deterministic manifest UUID
Adds a --seed CLI flag to 'generate' that derives a deterministic UUID
from the seed value by hashing it 1,000,000,000 times with SHA-256.
This makes manifest generation fully reproducible when the same seed
and input files are provided.

- Builder.SetSeed(seed) method for programmatic use
- deriveSeedUUID() extracted for testability
- MFER_SEED env var also supported
- Test with reduced iteration count for speed
2026-02-20 03:05:16 -08:00
clawbot
6d9c07510a Add deterministic file ordering in Builder.Build()
Sort file entries by path (lexicographic, byte-order) before
serialization to ensure deterministic output. Add fixedUUID support
for testing reproducibility, and a test asserting byte-identical
output from two runs with the same input.

Closes #23
2026-02-20 03:05:16 -08:00
6d1bdbb00f chore: remove stale .drone.yml (#37)
Remove obsolete Drone CI config. Added to .gitignore.

.golangci.yaml was already removed from next branch.

Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #37
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-20 12:00:49 +01:00
ae70cf6fb5 Fix FindExtraFiles reporting manifest and dotfiles as extra (closes #16) (#21)
Reviewed-on: #21
2026-02-20 11:38:52 +01:00
5099b6951b Fix IsHiddenPath treating current directory as hidden (closes #14) (#19)
Reviewed-on: #19
2026-02-20 11:37:23 +01:00
clawbot
7b61bdd62b fix: handle errcheck warnings in gpg.go and gpg_test.go 2026-02-19 23:41:14 -08:00
clawbot
8c7eef6240 fix: handle errcheck warnings in gpg.go and gpg_test.go, fix gofmt 2026-02-19 23:40:50 -08:00
clawbot
97dbe47c32 fix: FindExtraFiles skips dotfiles and manifest files to avoid false positives
FindExtraFiles now skips hidden files/directories and manifest files
(index.mf, .index.mf) when looking for extra files. Previously it would
report these as 'extra' even though they are intentionally excluded from
manifests by default, making --no-extra-files unusable.

Also includes IsHiddenPath fix for '.' (needed by the new filtering).
2026-02-19 23:36:48 -08:00
clawbot
f6478858d7 fix: IsHiddenPath returns false for "." and "/" (current dir/root are not hidden)
path.Clean(".") returns "." which starts with a dot, causing IsHiddenPath
to incorrectly treat the current directory as hidden. Add explicit checks
for "." and "/" before the dot-prefix check.

Fixed in both mfer/scanner.go and internal/scanner/scanner.go.
2026-02-19 23:36:42 -08:00
1f12d10cb7 Fix errors.Is with errors.New() never matching in checker (closes #12) (#17)
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #17
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 02:15:08 +01:00
7f25970dd3 Fix URL encoding for file paths in fetch command (closes #13) (#18)
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #18
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 02:14:20 +01:00
70af055d4e Fix newTimestampFromTime panic on extreme dates (closes #15) (#20)
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #20
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 02:10:21 +01:00
04b05e01e8 Consolidate scanner/checker — delete internal/scanner/ and internal/checker/ (closes #22) (#27)
Remove unused `internal/scanner/` and `internal/checker/` packages. The CLI already uses `mfer.Scanner` and `mfer.Checker` from the `mfer/` package directly, so these were dead code.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #27
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 02:09:01 +01:00
7144617d0e Add decompression size limit in deserializeInner() (closes #24) (#29)
Wrap zstd decompressor with `io.LimitReader` (256MB max) to prevent decompression bombs.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #29
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 01:45:55 +01:00
2efffd9da8 Specify and enforce path invariants (closes #26) (#31)
Add `ValidatePath()` enforcing UTF-8, forward-slash, relative, no `..`, no empty segments. Applied in `AddFile` and `AddFileWithHash`. Proto comments document the rules.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #31
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 01:45:29 +01:00
ebaf2a65ca Fix AddFile to verify actual bytes read matches declared size (closes #25) (#30)
After reading file content, verify `totalRead == size` and return an error on mismatch.

Co-authored-by: clawbot <clawbot@openclaw>
Reviewed-on: #30
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
2026-02-09 01:35:07 +01:00
4b80c0067b docs: replace TODO.md with design questions and implementation plan 2026-02-08 18:40:31 +01:00
5ab092098b progress 2026-02-08 09:25:58 -08:00
4a2060087d Add GPG signature verification on manifest load
- Implement gpgVerify function that creates a temporary keyring to verify
  detached signatures against embedded public keys
- Signature verification happens during deserialization after hash
  validation but before decompression
- Extract signatureString() as a method on manifest for generating the
  canonical signature string (MAGIC-UUID-MULTIHASH)
- Add --require-signature flag to check command to mandate signature from
  a specific GPG key ID
- Expose IsSigned() and Signer() methods on Checker for signature status
2025-12-18 05:56:16 -08:00
213364bab5 Add UUID to manifest and verify integrity before decompression
- Add UUID field to both inner and outer manifest messages
- Generate random v4 UUID when creating manifest
- Hash compressed data (not uncompressed) for integrity check
- Verify hash before decompression to prevent malicious payloads
- Validate UUIDs are proper format and match between inner/outer
- Sign string format: MAGIC-UUID-MULTIHASH
2025-12-18 02:20:51 -08:00
778999a285 Add GPG signing support for manifest generation
- Add --sign-key flag and MFER_SIGN_KEY env var to gen and freshen commands
- Sign inner message multihash with GPG detached signature
- Include signer fingerprint and public key in outer wrapper
- Add comprehensive tests with temporary GPG keyring
- Increase test timeout to 10s for GPG key generation
2025-12-18 02:12:54 -08:00
308c583d57 Remove codebase structure section from README
godoc provides this documentation automatically
2025-12-18 01:38:13 -08:00
019fe41c3d Update .gitignore for new bin/ build directory 2025-12-18 01:30:50 -08:00
fc0b38ea19 Add TODO.md with codebase audit findings
Document issues found during code audit including:
- Critical: broken error comparison, unchecked hash writes, URL path traversal
- Important: goroutine leak, timestamp precision, missing context cancellation
- Code quality: duplicate functions, inefficient calculations, missing validation
2025-12-18 01:30:01 -08:00
61c17ca585 Normalize markdown formatting in documentation
- Use consistent dash-style bullet points
- Remove trailing whitespace
- Add missing blank lines between sections
- Add trailing newline to README.md
2025-12-18 01:29:56 -08:00
dae6c64e24 Change build output path from mfer.cmd to bin/mfer
Use conventional bin/ directory for build output instead of
placing executable in project root.
2025-12-18 01:29:47 -08:00
a5b0343b28 Use Go 1.13+ octal literal syntax throughout codebase
Update file permission literals from legacy octal format (0755, 0644)
to explicit Go 1.13+ format (0o755, 0o644) for improved readability.
2025-12-18 01:29:40 -08:00
e25e309581 Move checker package into mfer package
Consolidate checker functionality into the mfer package alongside
scanner, removing the need for a separate internal/checker package.
2025-12-18 01:28:35 -08:00
dc115c5ba2 Add custom types for type safety throughout codebase
- Add FileCount, FileSize, RelFilePath, AbsFilePath, ModTime, Multihash types
- Add UnixSeconds and UnixNanos types for timestamp handling
- Add URL types (ManifestURL, FileURL, BaseURL) with safe path joining
- Consolidate scanner package into mfer package
- Update checker to use custom types in Result and CheckStatus
- Add ModTime.Timestamp() method for protobuf conversion
- Update all tests to use proper custom types
2025-12-18 01:01:18 -08:00
a9f0d2abe4 Update README to reflect current API (FileProgress was already a channel) 2025-12-17 17:19:08 -08:00
1588e1bb9f Remove unused legacy manifest APIs
Removed:
- New(), NewFromPaths(), NewFromFS() - unused constructors
- Scan(), addFile(), addInputPath(), addInputFS() - unused scanning code
- WriteToFile(), Write() - unused output methods (Builder.Build() is used)
- GetFileCount(), GetTotalFileSize() - unused accessors
- pathIsHidden() - duplicated in internal/scanner
- ManifestScanOptions - unused options struct
- HasError(), AddError(), WithContext() - unused error/context handling
- NewFromProto() - deprecated alias
- manifestFile struct - unused internal type

Kept:
- manifest struct (simplified to just pbInner, pbOuter, output)
- NewManifestFromReader(), NewManifestFromFile() - for loading manifests
- Files() - returns files from loaded manifest
- Builder and its methods - for creating manifests
2025-12-17 17:16:35 -08:00
09e8da0855 Update CLAUDE.md and clean up completed TODOs in README 2025-12-17 17:09:33 -08:00
efa4bb929a Update README: mark FIXMEs as resolved 2025-12-17 17:08:37 -08:00
16e3538ea6 Document WriteToFile overwrite behavior, remove misplaced FIXME 2025-12-17 17:08:11 -08:00