Loading accepted a file entry whose hashes list was empty. check and
fetch then failed the file as a hash mismatch, and freshen left it out
of the rewritten manifest without a word.
NewManifestFromReader now refuses such an entry, naming its path, after
checking the path itself. freshen's addExistingToBuilder returns an
error for an entry with no hash instead of skipping it. The file entry
table in docs/FORMAT.md says readers reject an entry with none.
Model: opus-5-5
mfer ran the gpg binary to sign, export keys and verify, so it failed
wherever gpg is missing. It now uses github.com/ProtonMail/go-crypto/openpgp.
--sign-key and MFER_SIGN_KEY name a file holding one version 4 OpenPGP
secret key; a protected key's passphrase comes from
MFER_SIGN_KEY_PASSPHRASE or a terminal prompt. gen and freshen check that
the key can sign before they read any file. Verification keeps the rules
of the --require-signature fix: one primary key in the embedded block,
counted from its packets, exactly one signature, made by that key or a
subkey, and signer equal to its fingerprint. The embedded block may hold
no DSA key and no secret key, and an armored field must be one
well-formed block.
Model: opus-5-5
Error messages in mfer/ and internal/cli/ are lowercase except names and
acronyms, carry no "failed to" or command-name prefix, and each wrap names
only the operation and thing the wrapped error does not already name, so a
stacked message names what failed once. Wraps around errors that already
name their operation and path (os and afero path errors, url.Error, the
builder's path errors, the gpg helpers' own errors) are dropped. gpg's
stderr is appended to a gpg failure, and to the error for a signing key gpg
did not report, only when gpg wrote some. errHTTPStatus reads "unexpected
HTTP status"; both inner-not-set sentinels read "inner message not set".
No sentinel, errors.Is result or exit status changes.
Model: opus-5-5
gen given arguments whose files share a path, such as gen a b with a.txt
in both, or gen . ., now fails while listing the files, before hashing
any, naming the path and both files. Builder.AddFile and
Builder.AddFileWithHash refuse a path already added. Loading refuses a
manifest that lists a path twice, compared byte for byte; fetch keeps
its own letter-case check. The Path Rules in docs/FORMAT.md say each
path appears at most once. The decode-size test listed one path 1000
times; each entry now has its own path of the same length.
Model: opus-5-5
Loading a manifest checked only the outer message's version, so an
inner message of version 0 or a later version loaded as if it were
version one, although docs/FORMAT.md requires VERSION_ONE in both.
deserializeInner now refuses any other inner version with the same
error as an unknown outer version. Two existing tests built inner
messages with no version and expected them to load; they now write
version one.
Model: opus-5-5
NewManifestFromReader reads at most one byte past MaxManifestSize, a new
constant of 258 MiB: the 256 MiB decompressed limit grown by zstd's worst
case of 1/256, plus 1 MiB for the signature, the signing key and the
other outer fields. It refuses a larger manifest. fetch, and check given
a URL, stop downloading a manifest one byte past the same size and report
it as too large; tests lower that size to keep their memory small. fetch
stops reading a file one byte past its listed size, so a longer body ends
in the size mismatch at once instead of filling the disk. docs/FORMAT.md
states the limit and gives the decompressed limit as 256 MiB, the size
the code uses.
Model: opus-5-5
check and fetch --require-signature compared the required fingerprint
with the first key in the manifest's embedded public key block, while
gpg accepted a good signature by any key in that block.
Loading a signed manifest now refuses one whose embedded block holds
more than one primary key, counted as gpg reads the block, or whose
signer field is not the primary key fingerprint gpg reports for the
signature. --require-signature compares with the signer field, which
loading has checked. Signing names and embeds the key gpg reports it
signed with, so a key ID matching several keys still writes a manifest
that loads. docs/FORMAT.md states what a verifier checks.
Model: opus-5-5
Go 1.27.1 in go.mod and in the Dockerfile's test and build images.
Every module go.mod requires is at its current release; protoc-gen-go
follows protobuf to v1.36.12 and mf.pb.go is regenerated. The standard
library uuid package replaces github.com/google/uuid; FromBytes could
only fail on a length validateUUID already checks, so that call and its
unreachable error are gone. make vulncheck runs govulncheck v1.8.0,
installed with go install at its release commit, in a vulncheck stage
of the Dockerfile; script/check does not run it. The newer go directive
switches on lint checks for strings.SplitSeq and t.Chdir, now used. A
new test pins the bytes of a seeded manifest written by an mfer built
before this change.
Model: opus-5-5
NewChecker now takes *CheckerOptions (ManifestPath, BasePath, Fs), named
like ScannerOptions. A nil Fs still means the OS filesystem, as before and
as in ScannerOptions; nil options or an empty path return an error naming
the missing path.
Audit of the other exported constructors in mfer: NewManifestFromFile took
a filesystem and a path positionally; it now takes
*ManifestFromFileOptions (Path, Fs) with the same nil and empty rules.
NewBuilder and NewScanner take no arguments, NewScannerWithOptions already
takes options, and NewManifestFromReader takes one reader, which the style
guide exempts; these are unchanged.
Model: opus-5-5
Before decoding the manifest, the parser adds up what decoding sets
aside for each file entry, hash, timestamp and MIME type, however short
its encoding, and refuses the manifest once that passes 8 times the
decompressed size; manifests mfer writes come to at most about 7.15
times. Empty entries decoded to about 50 times their size: under 1 KB of
manifest allocated about 500 MB. Fields the decoder does not know are
dropped; kept, they took up to 5 times more. A test refuses entries
counted just over 8 times and loads them just under. The fuzz ceiling
rises from 16 to 20 times the input and decompressed data; seeds of
empty entries and of empty hashes fail it without the fix.
Model: opus-5-5
FuzzNewManifestFromReader fails when the parser returns both or neither
of a manifest and an error, or allocates more than a fixed multiple of
its input and the decompressed data it may read, plus room for the
decoder's window buffers. make test runs the seed corpus; make fuzz
fuzzes for one minute.
Parser bug: MaxDecompressedSize did not bound decompression; the zstd
decoder decoded a payload under 128 KiB in full before the LimitReader
read any of it. It now decodes only what the LimitReader reads and
refuses windows over the 8 MiB mfer writes with. Seeds: a frame claiming
8 GiB, two frames together over the limit, empty frames with growing
windows.
Model: opus-5-5
Every gpg run now has a one-minute deadline (gpgTimeout) on top of its
caller's context and is killed when either ends. A timeout is reported
as "gpg timed out" under the failing operation instead of "signal:
killed". Only gpg itself is killed; WaitDelay (one second) stops the run
from waiting on a process gpg left behind that still holds its output,
such as a wrapper script that does not exec the real gpg.
Builder.Build and Checker.ExtractEmbeddedSigningKeyFP take a context, so
ToManifest's context now reaches signing and the contextcheck
suppression calling signing non-cancellable is gone. Manifest loading
takes no context, so its signature check is bounded by the timeout
alone.
Model: opus-5-5
Untrusted .mf files were parsed with no path validation, so an entry like ../../etc/passwd reached filepath.Join against the checker's base path and mfer check could stat and read outside it. ValidatePath ran only when building a manifest. It now runs on every entry as the manifest loads, so every consumer is covered. The whole manifest is rejected on the first bad entry instead of dropping it, which could hide files from a check; the error wraps errInvalidManifestPath and names the path.
Disclosure: a path that is not valid UTF-8 is refused earlier, by the protobuf string decoder, so that error does not name the path.
Model: opus-4-8 (implementation); fable-5-1 (summary)
Adopts golangci-lint v2.12.2 and the canonical .golangci.yml (default: all), and fixes all resulting findings across the tree.
Two intended behavior changes: absent MFFilePath.Mtime is handled explicitly in freshen, list and export rather than dereferenced (main panicked); gpg positional key IDs now follow an explicit -- end-of-options marker.
All twelve reworded user-visible error messages restored to byte-identical parity with main and pinned by tests.
- Implement gpgVerify function that creates a temporary keyring to verify
detached signatures against embedded public keys
- Signature verification happens during deserialization after hash
validation but before decompression
- Extract signatureString() as a method on manifest for generating the
canonical signature string (MAGIC-UUID-MULTIHASH)
- Add --require-signature flag to check command to mandate signature from
a specific GPG key ID
- Expose IsSigned() and Signer() methods on Checker for signature status
- Add UUID field to both inner and outer manifest messages
- Generate random v4 UUID when creating manifest
- Hash compressed data (not uncompressed) for integrity check
- Verify hash before decompression to prevent malicious payloads
- Validate UUIDs are proper format and match between inner/outer
- Sign string format: MAGIC-UUID-MULTIHASH
Use zstd with SpeedBestCompression level for better compression
ratios. Remove gzip support entirely. Include generated protobuf
file to allow building without protoc.
Major changes:
- Refactor CLI to accept injected I/O streams and filesystem (afero.Fs)
for testing without touching the real filesystem
- Add RunOptions struct and RunWithOptions() for configurable CLI execution
- Add internal/scanner package with two-phase manifest generation:
- Phase 1 (Enumeration): walk directories, collect metadata
- Phase 2 (Scan): read contents, compute hashes, write manifest
- Add internal/checker package for manifest verification with progress
reporting and channel-based result streaming
- Add mfer/builder.go for incremental manifest construction
- Add --no-extra-files flag to check command to detect files not in manifest
- Add timing summaries showing file count, size, elapsed time, and throughput
- Add comprehensive tests using afero.MemMapFs (no real filesystem access)
- Add contrib/usage.sh integration test script
- Fix banner ASCII art alignment (consistent spacing)
- Fix verbosity levels so summaries display at default log level
- Update internal/log to support configurable output writers
clawbot
