sneak/mfer
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Releases Wiki Activity

next #44

Open
sneak wants to merge 78 commits from next into main
pull from: next
merge into: sneak:main
Conversation 1 Commits 78 Files Changed 50 +7341 -572
sneak commented 2026-03-02 11:07:11 +01:00
Owner
Copy Link

getting rid of next branch

getting rid of next branch
sneak added 79 commits 2026-03-02 11:07:11 +01:00
latest
All checks were successful
continuous-integration/drone/push Build is passing
Details
01bffc8388
remove gofumpt from linting
Some checks failed
continuous-integration/drone/push Build is passing
Details
continuous-integration/drone/pr Build is failing
Details
13f39d598f 
Major changes:
- Refactor CLI to accept injected I/O streams and filesystem (afero.Fs)
  for testing without touching the real filesystem
- Add RunOptions struct and RunWithOptions() for configurable CLI execution
- Add internal/scanner package with two-phase manifest generation:
  - Phase 1 (Enumeration): walk directories, collect metadata
  - Phase 2 (Scan): read contents, compute hashes, write manifest
- Add internal/checker package for manifest verification with progress
  reporting and channel-based result streaming
- Add mfer/builder.go for incremental manifest construction
- Add --no-extra-files flag to check command to detect files not in manifest
- Add timing summaries showing file count, size, elapsed time, and throughput
- Add comprehensive tests using afero.MemMapFs (no real filesystem access)
- Add contrib/usage.sh integration test script
- Fix banner ASCII art alignment (consistent spacing)
- Fix verbosity levels so summaries display at default log level
- Update internal/log to support configurable output writers
Documents:
- cli: NO_COLOR, RunOptions fields, CLIApp, VersionString
- checker: Result fields, Status constants, CheckStatus fields
- scanner: EnumerateStatus, ScanStatus, Options, FileEntry fields
- log: Level alias, DisableStyling, Init, Info/Debug functions,
  verbosity helpers, GetLogger, GetLevel, WithError
- mfer: ManifestScanOptions, New, NewFromPaths, NewFromFS, MAGIC
- Remove generateInner() from serialize.go
- Update generate() to error if pbInner not set
- Remove legacy tests that depended on old code path
- Update TODO item to reflect removal
Replace callback-based progress reporting in Builder.AddFile with
channel-based FileHashProgress for consistency with EnumerateStatus
and ScanStatus patterns. Update scanner.go to use the new channel API.
- Add explicit error ignoring with _ = for Close/Remove calls
- Rename WriteTo to Write to avoid io.WriterTo interface conflict
- Fix errcheck warnings in fetch, freshen, gen, mfer, checker,
  deserialize, serialize, and output files
Use zstd with SpeedBestCompression level for better compression
ratios. Remove gzip support entirely. Include generated protobuf
file to allow building without protoc.
Changed the default behavior to exclude dotfiles (files/dirs starting with .)
which is the more common use case. Added --include-dotfiles flag for when
hidden files need to be included in the manifest.
Implemented full log level hierarchy: Fatal, Error, Warn, Info, Verbose, Debug.
- Verbose level (-v) shows detailed operations like file changes (M/A/D)
- Debug level (-vv) shows low-level tracing with caller info
- Quiet mode (-q) sets level to Error, suppressing Info messages
- Banner and summary output now use log levels for filtering
- pathIsHidden(".") was returning true, causing freshen to skip entire
  directory tree when dotfiles excluded
- Banner now prints directly to stdout to avoid log prefix artifacts
Added mfer/constants.go with Version and ReleaseDate constants for
deterministic builds. Banner now shows "mfer by @sneak: v0.1.0 released 2025-12-17"
- Version command now uses mfer.Version constant instead of empty build flags
- Running just 'mfer' shows banner + help
- Unknown commands still return error with exit code 1
- mfer list: shows file paths one per line
- mfer list -l/--long: shows size, mtime, and path
- mfer list --print0: NUL-separated output for xargs -0
- Atomic writes for mfer gen: writes to temp file, renames on success,
  cleans up temp on error/interrupt. Prevents empty manifests on Ctrl-C.
- Humanized byte sizes using dustin/go-humanize (e.g., "10 MiB" not "10485760")
- Progress lines clear when done (using ANSI escape \r\033[K])
- Debug logging when files are added to manifest (mfer gen -vv)
- Move -v/-q flags from global to per-command for better UX
- Add tests for atomic write behavior with failing filesystem mock
Generates a ~1MB manifest (20000 files with random names), then:
- Verifies truncated manifest causes check to fail
- Runs 500 iterations of random single-byte corruption
- Each iteration verifies check detects the corruption
- Humanize file sizes in verbose file listing (e.g., "76.8 MiB" not "76836984 bytes")
- Add signal handler to clean up temp file on Ctrl-C/SIGTERM during gen
- Add comprehensive unit tests for internal/checker (88.5% coverage)
  - Status string representations
  - NewChecker validation
  - Check operation (OK, missing, size/hash mismatch)
  - Progress reporting and context cancellation
  - FindExtraFiles functionality

- Add comprehensive unit tests for internal/scanner (80.1% coverage)
  - Constructors and options
  - File/path enumeration
  - Dotfile exclusion/inclusion
  - ToManifest with progress and cancellation
  - Non-blocking status channel sends

- Validate input paths before scanning in generate command
  - Fail fast with clear error if paths don't exist
  - Prevents confusing errors deep in enumeration
Use the FileInfo already provided by Walk instead of calling Stat again.
Only stat if fi is nil (defensive, shouldn't happen in normal Walk usage).
Also fixes potential nil pointer dereference if fi was nil.
Removed:
- New(), NewFromPaths(), NewFromFS() - unused constructors
- Scan(), addFile(), addInputPath(), addInputFS() - unused scanning code
- WriteToFile(), Write() - unused output methods (Builder.Build() is used)
- GetFileCount(), GetTotalFileSize() - unused accessors
- pathIsHidden() - duplicated in internal/scanner
- ManifestScanOptions - unused options struct
- HasError(), AddError(), WithContext() - unused error/context handling
- NewFromProto() - deprecated alias
- manifestFile struct - unused internal type

Kept:
- manifest struct (simplified to just pbInner, pbOuter, output)
- NewManifestFromReader(), NewManifestFromFile() - for loading manifests
- Files() - returns files from loaded manifest
- Builder and its methods - for creating manifests
- Add FileCount, FileSize, RelFilePath, AbsFilePath, ModTime, Multihash types
- Add UnixSeconds and UnixNanos types for timestamp handling
- Add URL types (ManifestURL, FileURL, BaseURL) with safe path joining
- Consolidate scanner package into mfer package
- Update checker to use custom types in Result and CheckStatus
- Add ModTime.Timestamp() method for protobuf conversion
- Update all tests to use proper custom types
Consolidate checker functionality into the mfer package alongside
scanner, removing the need for a separate internal/checker package.
Update file permission literals from legacy octal format (0755, 0644)
to explicit Go 1.13+ format (0o755, 0o644) for improved readability.
Use conventional bin/ directory for build output instead of
placing executable in project root.
- Use consistent dash-style bullet points
- Remove trailing whitespace
- Add missing blank lines between sections
- Add trailing newline to README.md
Document issues found during code audit including:
- Critical: broken error comparison, unchecked hash writes, URL path traversal
- Important: goroutine leak, timestamp precision, missing context cancellation
- Code quality: duplicate functions, inefficient calculations, missing validation
godoc provides this documentation automatically
- Add --sign-key flag and MFER_SIGN_KEY env var to gen and freshen commands
- Sign inner message multihash with GPG detached signature
- Include signer fingerprint and public key in outer wrapper
- Add comprehensive tests with temporary GPG keyring
- Increase test timeout to 10s for GPG key generation
- Add UUID field to both inner and outer manifest messages
- Generate random v4 UUID when creating manifest
- Hash compressed data (not uncompressed) for integrity check
- Verify hash before decompression to prevent malicious payloads
- Validate UUIDs are proper format and match between inner/outer
- Sign string format: MAGIC-UUID-MULTIHASH
- Implement gpgVerify function that creates a temporary keyring to verify
  detached signatures against embedded public keys
- Signature verification happens during deserialization after hash
  validation but before decompression
- Extract signatureString() as a method on manifest for generating the
  canonical signature string (MAGIC-UUID-MULTIHASH)
- Add --require-signature flag to check command to mandate signature from
  a specific GPG key ID
- Expose IsSigned() and Signer() methods on Checker for signature status
After reading file content, verify `totalRead == size` and return an error on mismatch.

Co-authored-by: clawbot <clawbot@openclaw>
Reviewed-on: #30
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Add `ValidatePath()` enforcing UTF-8, forward-slash, relative, no `..`, no empty segments. Applied in `AddFile` and `AddFileWithHash`. Proto comments document the rules.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #31
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Wrap zstd decompressor with `io.LimitReader` (256MB max) to prevent decompression bombs.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #29
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Remove unused `internal/scanner/` and `internal/checker/` packages. The CLI already uses `mfer.Scanner` and `mfer.Checker` from the `mfer/` package directly, so these were dead code.

Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #27
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #20
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #18
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Co-authored-by: clawbot <clawbot@openclaw>
Co-authored-by: Jeffrey Paul <sneak@noreply.example.org>
Reviewed-on: #17
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
path.Clean(".") returns "." which starts with a dot, causing IsHiddenPath
to incorrectly treat the current directory as hidden. Add explicit checks
for "." and "/" before the dot-prefix check.

Fixed in both mfer/scanner.go and internal/scanner/scanner.go.
FindExtraFiles now skips hidden files/directories and manifest files
(index.mf, .index.mf) when looking for extra files. Previously it would
report these as 'extra' even though they are intentionally excluded from
manifests by default, making --no-extra-files unusable.

Also includes IsHiddenPath fix for '.' (needed by the new filtering).
Reviewed-on: #19
Reviewed-on: #21
Remove obsolete Drone CI config. Added to .gitignore.

.golangci.yaml was already removed from next branch.

Co-authored-by: user <user@Mac.lan guest wan>
Reviewed-on: #37
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Sort file entries by path (lexicographic, byte-order) before
serialization to ensure deterministic output. Add fixedUUID support
for testing reproducibility, and a test asserting byte-identical
output from two runs with the same input.

Closes #23
Adds a --seed CLI flag to 'generate' that derives a deterministic UUID
from the seed value by hashing it 1,000,000,000 times with SHA-256.
This makes manifest generation fully reproducible when the same seed
and input files are provided.

- Builder.SetSeed(seed) method for programmatic use
- deriveSeedUUID() extracted for testability
- MFER_SEED env var also supported
- Test with reduced iteration count for speed
1B iterations was too slow (30s+). Benchmarked on Apple Silicon:
- 150M iterations ≈ 6.3s
- Falls within the 5-10s target range
Replace 150M SHA-256 iteration key-stretching with a single hash.
Remove all references to iteration counts, timing (~5-10s), and
key-stretching from code and documentation.

The seed flag is retained for deterministic UUID generation, but
now derives the UUID with a single SHA-256 hash instead of the
unnecessary iterative approach.
Removes `vendor.tzst` and `modcache.tzst` that should never have been committed. Adds both to `.gitignore`.

Reviewed-on: #35
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
Co-committed-by: clawbot <sneak+clawbot@sneak.cloud>
Reviewed-on: #28
Comprehensive quality pass targeting 1.0 release:

- Code review and refactoring
- Fix open bugs (#14, #16, #23)
- Expand test coverage
- Lint clean
- README update with build instructions (#9)
- Documentation improvements

Branched from `next` (active dev branch).

Reviewed-on: #32
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
Add make check target and CI workflow (#36)
Some checks failed
check / check (pull_request) Failing after 6s
Details
acbdf3a376 
Adds a `make check` target that verifies formatting (gofmt), linting (golangci-lint), and tests (go test -race) without modifying files.

Also adds `.gitea/workflows/check.yml` CI workflow that runs on pushes and PRs to main and next.

`make check` passes cleanly on current next branch.

Co-authored-by: clawbot <clawbot@noreply.git.eeqj.de>
Reviewed-on: #36
Co-authored-by: clawbot <clawbot@noreply.example.org>
Co-committed-by: clawbot <clawbot@noreply.example.org>
sneak commented 2026-03-02 11:07:40 +01:00
Author
Owner
Copy Link

@clawbot pls fix merge issues.

@clawbot pls fix merge issues.
clawbot force-pushed next from acbdf3a376 to 43916c7746 2026-03-02 11:12:17 +01:00 Compare
This pull request has changes conflicting with the target branch.
  • README.md
  • TODO.md
  • go.mod
View command line instructions.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin next:next
git checkout next
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bot

Agent's turn to work

  
merge-ready

All checks pass, reviewed, rebased, ready for merge

  
needs-checks

make check does not pass cleanly

  
needs-rebase

PR has merge conflicts or is behind main

  
needs-review

Code review not yet done

  
needs-rework

Code review found issues to fix

No Label
bot
merge-ready
needs-checks
needs-rebase
needs-review
needs-rework
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: sneak/mfer#44
No description provided.
Delete Branch "next"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?