Remove TODO.md; track open work in the issues (closes #76)

TODO.md is deleted and the README TODO section now only points to the
issue tracker, where open work and open design questions live from now
on. AGENTS.md says so too, and says this overrides the shared policy's
README todo list for this repo. The README Open Questions section
becomes Original Design Questions, each noting where it now stands.

Every open item in both lists was already done or already owned by an
issue, apart from one, now #121,
and the chunk checksum question, now on
#81.

Model: opus-5-5

.dockerignore

@@ -1,4 +1,64 @@
+# .dockerignore does NOT use .gitignore semantics. Docker matches with
+# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
+# `/` and an unprefixed pattern is anchored at the context root. Every
+# depth-independent pattern therefore needs `**/`, or `config/.env` and
+# `certs/server.key` still ship while this file reads as solved. Only
+# genuinely root-anchored entries go unprefixed. Never transplant these
+# into .gitignore, where `**/` is wrong.
+#
+# Matching is case-sensitive, so secrets use character ranges rather
+# than an ALL-CAPS twin, which would still miss `Server.Key`.
+#
+# Extend with this repo's own host-built artifacts, written anchored:
+# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
+# deletes the package directory from the context.
+
+# .git is sent without its config. Without a VERSION build argument the
+# stage that compiles runs `git describe --tags --always` on .git, which
+# does not need .git/config; that file can hold a credential, such as a
+# password in a remote URL or the token the CI checkout step stores there.
+.git/config
+
+# Agent scratch: one full checkout of the repo per in-flight agent.
+# Anchored because it occurs once where agents run at the repo root.
+# KNOWN GAP: a repo running agents in subdirectories still ships
+# `services/api/.claude/` and must add its own anchored entry.
+.claude
+
+# Environment files. `*.env` covers bare `.env` and the `prod.env`
+# convention. Re-include a committed template with a negation if the
+# build needs one: `!docs/example.env`.
+**/*.[eE][nN][vV]
+**/.[eE][nN][vV].*
+**/.[eE][nN][vV][rR][cC]
+
+# Private keys and the bundles carrying them. Public certificates
+# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
+**/*.[pP][eE][mM]
+**/*.[kK][eE][yY]
+**/*.[pP]12
+**/*.[pP][fF][xX]
+**/[iI][dD]_[rR][sS][aA]
+**/[iI][dD]_[dD][sS][aA]
+**/[iI][dD]_[eE][cC][dD][sS][aA]
+**/[iI][dD]_[eE][dD]25519
+
+# Dependencies: restored inside the image, never copied in.
+**/node_modules
+
+# OS metadata.
+**/.DS_Store
+**/Thumbs.db
+
+# Editor state: never a build input, and it churns COPY.
+**/*.swp
+**/*.swo
+**/*~
+**/*.bak
+**/.idea
+**/.vscode
+**/*.sublime-*
+
+# This repo's own host-built archives (Makefile).
 *.tmp
 *.dockerimage
-.git
-node_modules

AGENTS.md

@@ -27,5 +27,8 @@ source for coding standards, formatting, linting, and workflow rules.
 - The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are
   committed (required for `go get` compatibility).
 - The format specification is in `FORMAT.md`.
-- See the TODO section in `README.md` for the 1.0 implementation plan and open
-  design questions.
+- Open work, open design questions included, is tracked only in the repo's
+  issues: https://git.eeqj.de/sneak/mfer/issues. There is no `TODO.md` and no
+  TODO list in `README.md`; do not add either. For this repo this overrides the
+  `REPO_POLICIES.md` rule to put the todo list in the README, per sneak's
+  ruling: https://git.eeqj.de/sneak/mfer/issues/76#issuecomment-118130.

Dockerfile

@@ -48,7 +48,26 @@ COPY . .
 RUN touch mfer/mf.pb.go
 
 RUN make test
-RUN cd cmd/mfer && go build -tags urfave_cli_no_docs -o /mfer .
+
+# A build context sent as a tar archive, as upaas sends it, keeps its files'
+# owners, and git refuses to read a checkout owned by another user.
+RUN git config --system --add safe.directory /src
+
+# The revision `mfer version` prints, stamped into main.Gitrev: the VERSION
+# build argument when one is given (script/docker passes one), otherwise
+# `git describe --tags --always` of the .git the build context carries: the
+# tag on a tagged commit, tag-N-gHASH on a commit after one, the short commit
+# when no tag is reachable. git ships in this base image. A context that
+# carries .git and still yields no version fails the build.
+ARG VERSION
+RUN version="${VERSION:-$(git describe --tags --always)}"; \
+    if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
+        [ "$version" = unknown ]; }; then \
+        echo "no version could be derived although the build context carries .git" >&2; \
+        exit 1; \
+    fi; \
+    cd cmd/mfer && \
+    go build -tags urfave_cli_no_docs -ldflags "-X main.Gitrev=$version" -o /mfer .
 
 FROM scratch
 COPY --from=builder /mfer /mfer

README.md

@@ -1,12 +1,12 @@
 # mfer
 
-[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
-(public domain) [Go](https://golang.org) library and command-line tool by
-[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
-over a directory tree to encapsulate metadata about the files — such as
-cryptographic checksums and signatures over same — to aid in archiving,
-downloading, streaming, and mirroring. It was first published in 2022. The
-manifest files' data is serialized with Google's
+[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
+thin wrapper command-line utility written in [Go](https://golang.org) and first
+published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
+It specifies and generates `.mf` manifest files over a directory tree of files
+to encapsulate metadata about them (such as cryptographic checksums or
+signatures over same) to aid in archiving, downloading, and streaming, or
+mirroring. The manifest files' data is serialized with Google's
 [protobuf serialization format](https://developers.google.com/protocol-buffers).
 The structure of these files can be found
 [in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
@@ -21,40 +21,11 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
 as a de-facto standard and be incorporated into other software. A compatible
 javascript library is planned.
 
-# Getting Started
-
-`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
-is committed, so no `protoc` toolchain is required:
-
-```sh
-git clone https://git.eeqj.de/sneak/mfer.git
-cd mfer
-go build -o bin/mfer ./cmd/mfer
-```
-
-Generate a manifest for a directory tree, verify it later, and fetch a published
-tree by URL:
-
-```sh
-# Write .index.mf describing every file under the current directory.
-bin/mfer gen .
-
-# Verify the files on disk against the manifest. Exits nonzero if any file
-# is missing or corrupted.
-bin/mfer check .index.mf
-
-# Download and cryptographically verify a tree published over HTTP: mfer
-# fetches <url>/index.mf, then downloads every file it lists.
-bin/mfer fetch https://example.com/tree/
-```
-
-Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
-for a single command's options.
-
 # Build Status
 
-CI runs via `script/cibuild` (`docker build .`), which executes `make check`
-(formatting, linting, tests). The `main` branch must always be green.
+CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
+the formatting, lint and test steps in the `Dockerfile` run on every build. The
+`main` branch must always be green.
 
 # Entrypoints
 
@@ -86,8 +57,8 @@ provide:
   Docker lint stage, whose image has no node
 - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
 - `script/docker` — build the Docker image tagged with the project name
-- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile runs the
-  checks)
+- `script/cibuild` — CI entrypoint: builds the image with the same command as
+  `script/docker`, uncached, so the checks in the Dockerfile run every time
 - `script/precommit` — pre-commit checks: `go mod tidy` verification, then
   `script/check`
 - `script/install-precommit` — install the git pre-commit hook that runs
@@ -111,9 +82,7 @@ Any changes submitted to this project must also be
 See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
 tooling requirements, and workflow conventions.
 
-# Rationale
-
-## The problem
+# Problem Statement
 
 Given a plain URL, there is no standard way to safely and programmatically
 download everything "under" that URL path. `wget -r` can traverse directory
@@ -137,7 +106,7 @@ Real issues I face:
 - when I download a large file via HTTP, I have no way of knowing if the file
   content is what it's supposed to be
 
-## The solution
+# Proposed Solution
 
 A standard, a manifest file format, and a tool for generating same.
 
@@ -169,27 +138,6 @@ The manifest file would do several important things:
     - maybe a bittorrent chunklist for torrent client compatibility? perhaps a
       top-level infohash for the whole manifest?
 
-# Design
-
-The repository is split into a reusable library and a thin command-line wrapper
-around it.
-
-- `mfer/` is the reusable library and the heart of the project: it defines the
-  manifest format and implements building, scanning, checking, serialization,
-  and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
-  produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
-  with `go get` and needs no `protoc` toolchain.
-- `internal/cli/` holds the command implementations — `gen`, `check`, `freshen`,
-  `export`, `list`, and `fetch` — that wire the library to the command-line
-  interface.
-- `internal/log/` provides the logging used across the commands.
-- `internal/bork/` provides error-handling support.
-- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
-  into the `mfer` binary.
-
-Everything under `internal/` is private to this repository; only the `mfer/`
-package is intended for import by other software.
-
 # Design Goals
 
 - Replace SHASUMS/SHASUMS.asc files
@@ -209,18 +157,23 @@ package is intended for import by other software.
     - metadata size should not be used as an excuse to sacrifice utility (such
       as providing checksums over each chunk of a large file)
 
-# Open Questions
+# Original Design Questions
+
+These were the open questions when the project started; open design questions
+are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
 
 - Should the manifest file include checksums of individual file chunks, or just
-  for the whole assembled file?
-
-    - If so, should the chunksize be fixed or dynamic?
+  for the whole assembled file? If so, should the chunk size be fixed or
+  dynamic? Still open, on
+  [issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698).
 
 - Should the manifest signature format be GnuPG signatures, or those from
   OpenBSD's signify (of which there is a good
-  [golang implementation](https://github.com/frankbraun/gosignify)?
+  [golang implementation](https://github.com/frankbraun/gosignify))? Still open,
+  as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).
 
-- Should the on-disk serialization format be proto3 or json?
+- Should the on-disk serialization format be proto3 or json? Settled: it is
+  proto3, see `FORMAT.md` and `mfer/mf.proto`.
 
 # Tool Examples
 
@@ -298,207 +251,10 @@ regardless of filesystem format.
 Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired
 username for an account on this Gitea instance.
 
-# TODO: Remaining Work for 1.0
+# TODO
 
-## Design Questions (Owner Decision Required)
-
-These require @sneak's input before implementation. Answers should be added
-inline below each question.
-
-### Format Design
-
-**1. Should `MFFileChecksum` be simplified?** Currently it's a separate message
-wrapping a single `bytes multiHash` field. Since multihash already
-self-describes the algorithm, `repeated bytes hashes` directly on `MFFilePath`
-would be simpler and reduce per-file protobuf overhead. Is the extra message
-layer intentional (e.g. planning to add per-hash metadata like `verified_at`)?
-
-> _answer:_
-
-**2. Should file permissions/mode be stored?** The format stores mtime/ctime but
-not Unix file permissions. For archival use this may not matter, but for
-software distribution or filesystem restoration it's a gap. Should we reserve a
-field now (e.g. `optional uint32 mode = 305`) even if we don't populate it yet?
-
-> _answer:_
-
-**3. Should `atime` be removed from the schema?** Access time is volatile,
-non-deterministic, and often disabled (`noatime`). Including it means two
-manifests of the same directory at different times will differ, which conflicts
-with the determinism goal. Remove it, or document it as "never set by default"?
-
-> _answer:_
-
-**4. What are the path normalization rules?** The proto has `string path` with
-no specification about: always forward-slash? Must be relative? No `..`
-components allowed? UTF-8 NFC vs NFD normalization (macOS vs Linux)? Max path
-length? This is a security issue (path traversal) and a cross-platform
-compatibility issue. What rules should the spec mandate?
-
-> _answer:_
-
-**5. Should we add a version byte after the magic?** Currently `ZNAVSRFG` is
-followed immediately by protobuf. Adding a version byte (`ZNAVSRFG\x01`) would
-allow future framing changes without requiring protobuf parsing to detect the
-version. `MFFileOuter.Version` serves this purpose but requires successful
-deserialization to read. Worth the extra byte?
-
-> _answer:_
-
-**6. Should we add a length-prefix after the magic?** Protobuf is not
-self-delimiting. If we ever want to concatenate manifests or append data after
-the protobuf, the current framing is insufficient. Add a varint or fixed-width
-length-prefix?
-
-> _answer:_
-
-### Signature Design
-
-**7. What does the outer SHA-256 hash cover — compressed or uncompressed data?**
-The code currently hashes compressed data (good for verifying before
-decompression), but this should be explicitly documented. Which is the intended
-behavior?
-
-> _answer:_
-
-**8. Should `signatureString()` sign raw bytes instead of a hex-encoded
-string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with hex
-encoding, which adds a transformation layer. Signing the raw `sha256` bytes (or
-compressed `innerMessage` directly) would be simpler. Keep the string format or
-switch to raw bytes?
-
-> _answer:_
-
-**9. Should we support detached signature files (`.mf.sig`)?** Embedded
-signatures are better for single-file distribution. Detached `.mf.sig` files
-follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are simpler for HTTP
-serving. Support both modes?
-
-> _answer:_
-
-**10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is fragile
-(may not be installed, version-dependent output).
-`github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could use
-Ed25519/signify (simpler, no key management). Which direction?
-
-> _answer:_
-
-### Implementation Design
-
-**11. Should manifests be deterministic by default?** This means: sort file
-entries by path, omit `createdAt` timestamp (or make it opt-in), no `atime`.
-Should determinism be the default, with a `--include-timestamps` flag to opt in?
-
-> _answer:_
-
-**12. Should we consolidate or keep both scanner/checker implementations?**
-There are two parallel implementations: `mfer/scanner.go` + `mfer/checker.go`
-(typed with `FileSize`, `RelFilePath`) and `internal/scanner/` +
-`internal/checker/` (raw `int64`, `string`). The `mfer/` versions are superior.
-Delete the `internal/` versions?
-
-> _answer:_
-
-**13. Should the `manifest` type be exported?** Currently unexported with
-exported constructors (`NewManifestFromReader`, `NewManifestFromFile`).
-Consumers can't declare `var m *mfer.manifest`. Export the type, or define an
-interface?
-
-> _answer:_
-
-**14. What should the Go module path be for 1.0?** Currently
-`sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in the
-proto `go_package` option. Which is canonical?
-
-> _answer:_
-
-## Implementation Tasks
-
-### Repo Infrastructure
-
-- [ ] Add `.golangci.yml` (fetch from
-      `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`)
-- [ ] Add `.editorconfig`
-- [ ] Add `.gitea/workflows/check.yml` that runs `docker build .`
-
-### Format & Correctness
-
-- [ ] Resolve proto `go_package` path inconsistency
-      (`git.eeqj.de/sneak/mfer/mfer` vs `sneak.berlin/go/mfer`)
-- [ ] Specify path invariants — add proto comments requiring UTF-8,
-      forward-slash, relative paths, no `..`, no leading `/`; validate in
-      `Builder.AddFile` and `Builder.AddFileWithHash` (pending design question
-      answer)
-- [ ] Remove or deprecate `atime` from proto (pending design question answer)
-- [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future file
-      permissions (pending design question answer)
-- [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version 1
-      (pending design question answer)
-- [ ] Write format specification document — separate from README: magic, outer
-      structure, compression, inner structure, path invariants, signature
-      scheme, canonical serialization
-
-### Library
-
-- [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate on
-      `mfer/` package versions; update CLI code (pending design question answer)
-- [ ] Add deterministic file ordering — sort entries by path (lexicographic,
-      byte-order) in `Builder.Build()`; add test asserting byte-identical output
-      from two runs
-- [ ] Add decompression size limit — `io.LimitReader` in `deserializeInner()`
-      with `m.pbOuter.Size` as bound
-- [ ] Fix `errors.Is` dead code in checker — replace with `os.IsNotExist(err)`
-      or `errors.Is(err, fs.ErrNotExist)`
-- [ ] Fix `AddFile` to verify size — check `totalRead == size` after reading,
-      return error on mismatch
-- [ ] Export the `manifest` type or define a public interface (pending design
-      question answer) — currently consumers cannot hold a reference to a loaded
-      manifest in their own type declarations
-- [ ] Replace GPG subprocess calls with pure-Go crypto (pending design question
-      answer) — current implementation shells out to `gpg` which may not be
-      installed
-- [ ] Add timeout to any remaining subprocess calls
-
-### CLI
-
-- [ ] Fix flag naming — all CLI flags should use kebab-case as primary
-      (`--include-dotfiles`, `--follow-symlinks`)
-- [ ] Fix URL construction in fetch — use `BaseURL.JoinPath()` or
-      `url.JoinPath()` instead of string concatenation
-- [ ] Add progress rate-limiting to Checker — throttle to once per second,
-      matching Scanner
-- [ ] Add `--deterministic` flag or make it default — omit `createdAt`, sort
-      files (pending design question answer)
-- [ ] Wire `--version` flag properly (currently only a `version` subcommand
-      exists; top-level `--version` shows urfave/cli generic output)
-- [ ] Add retry logic to `fetch` — currently no retries on transient HTTP
-      errors; needs exponential backoff
-- [ ] `fetch` command uses bare `http.Get` with no timeout — needs `http.Client`
-      with configurable timeout
-
-### Testing & Robustness
-
-- [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf deserialization
-      of untrusted input needs fuzz coverage
-- [ ] Add integration test for `freshen` CLI command — current tests only verify
-      setup, not the actual freshen operation end-to-end
-- [ ] Add test for `fetch` CLI command end-to-end (currently only `downloadFile`
-      is tested)
-
-### Documentation
-
-- [ ] Promote `FORMAT.md` as primary spec reference; README should link to it
-      more prominently
-- [ ] Audit and update all error messages for consistency and helpfulness
-- [ ] Document the signature scheme more thoroughly (canonical string format,
-      verification steps)
-
-### Release
-
-- [ ] Finalize Go module path
-- [ ] Update version constant in `mfer/constants.go`
-- [ ] Add `--version` output matching SemVer
-- [ ] Tag `v1.0.0`
+Open work, open design questions included, is tracked in this repo's issues:
+[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues).
 
 # See Also
 
@@ -515,9 +271,9 @@ proto `go_package` option. Which is canonical?
 - Issues:
   [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
 
-# Author
+# Authors
 
-- [@sneak](https://sneak.berlin)
+- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
 
 # License
 

TODO.md

@@ -1,117 +0,0 @@
-# Workflow
-
-- branch (from `main`)
-- do the work in Next Step
-- move Next Step to the top of Completed Steps
-- move the top item of Future Steps into Next Step
-- commit (`TODO.md` changes in the same commit as the work)
-- merge to `main` if the branch is not protected, otherwise open a PR
-- push
-
-# Status
-
-pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists open
-design questions and implementation tasks; policy compliance work is in flight
-and unmerged.
-
-# Next Step
-
-Work through the remaining compliance items folded from the 2026-07-02 audit
-(the first group under Future Steps): `.editorconfig`, `.gitignore` coverage,
-gofumpt-based `fmt-check`, README "Getting Started", and the rest.
-`.golangci.yml` and `TODO.md` are tracked and committed as of 2026-08-07, so the
-only thing left of the `chore/align-repo-policies` branch is the list below.
-
-# Completed Steps
-
-- 2026-09-21: added the required README sections — named author, license, and
-  category in the Description first line; added Getting Started (verified
-  install/usage block), Rationale (folding in Problem Statement and Proposed
-  Solution), and a Design section for the package layout; renamed Authors to
-  Author (#75)
-- 2026-09-21: added the canonical `.editorconfig`, made `.gitignore` cover
-  secrets, OS, editor, and Go artifacts, and removed the dead Drone CI
-  references from `.gitignore` and `bin/gitrev.sh` (#72)
-- 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and
-  `script/fmt-check` one shared prettier file set via `script/prettier`, dropped
-  the `|| true` that hid prettier failures, and added a node-based Dockerfile
-  stage so a markdown formatting violation fails `docker build .` (#69)
-- 2026-08-07: updated golangci-lint to v2.12.2 everywhere it is pinned
-  (`Makefile`, `Dockerfile`), added the canonical `.golangci.yml`
-  (`default: all`), and fixed all resulting lint findings across the codebase
-- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
-  shims, README Entrypoints section
-- 2026-07-03: aligned repo tooling, docs, and config with standardized policies
-  (7d9a138, on chore/align-repo-policies, unmerged)
-- 2026-06-28: moved to standardized repo policies (#56, on main)
-- 2026-04-07: added 1.0 roadmap as README TODO section, removed old TODO.md
-  (#54)
-- 2026-03-20: added Gitea Actions CI workflow (#53)
-- 2026-03-17: added REPO_POLICIES.md, renamed CLAUDE.md to AGENTS.md (#51);
-  removed committed .index.mf (#52)
-- 2026-03-15: split Dockerfile with pre-built golangci-lint stage for faster CI
-  (#45)
-- 2026-03-01: 1.0 quality polish: code review, tests, bug fixes, docs (#32)
-- 2026-02-20: deterministic file ordering in Builder.Build() (#28); removed
-  committed vendor/modcache archives (#35)
-- 2026-02-08: added --seed flag for deterministic manifest UUID
-
-# Future Steps
-
-- Compliance (fold of TODO.md audit 2026-07-02; verify which items the in-flight
-  branch already closes, then check off):
-    - Add .editorconfig (canonical copy from sneak/prompts)
-    - Make .gitignore cover secrets (.env, _.key, _.pem), OS files (.DS_Store),
-      and editor files (_.swp, _~)
-    - Make fmt-check/lint verify with gofumpt, not gofmt -l, so `make check`
-      matches what `make fmt` writes
-    - Add README "Getting Started" section with copy-pasteable install/usage
-      block
-    - Move FORMAT.md from repo root to docs/ and update the AGENTS.md reference
-    - Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`,
-      `golangci-lint@v2.12.2`) by module hash, not mutable tag
-    - Set `make test` timeout to 30s (currently 10s)
-    - Add explicit README "Rationale" heading (content exists under other
-      names); name the author in the README Description first line
-    - Reconcile root-level AGENTS.md with directory-hygiene policy (keep or
-      relocate)
-    - Add a `make build` target
-    - Rewrite `make hooks` to use printf or a heredoc instead of non-portable
-      `echo '...\n...'`
-- Answer the 14 owner design questions in the README 1.0 roadmap:
-    - Format: simplify MFFileChecksum; store file mode; drop atime; specify path
-      normalization rules; version byte after magic; length-prefix after magic
-    - Signatures: hash covers compressed or uncompressed data; sign raw bytes vs
-      hex canonical string; detached .mf.sig support; GPG subprocess vs pure-Go
-      crypto
-    - Implementation: deterministic manifests by default; consolidate duplicate
-      scanner/checker implementations; export the manifest type; canonical Go
-      module path for 1.0
-- Format and correctness:
-    - Resolve proto go_package vs go.mod module path inconsistency
-    - Specify and validate path invariants (UTF-8, forward-slash, relative, no
-      .., no leading /)
-    - Remove or deprecate atime; reserve mode field; add version byte (all
-      pending design answers)
-    - Write a standalone format specification document
-- Library:
-    - Delete internal/scanner and internal/checker; consolidate on the mfer/
-      package versions (pending design answer)
-    - Add decompression size limit via io.LimitReader in deserializeInner()
-    - Fix errors.Is dead code in checker; make AddFile verify totalRead == size
-    - Export manifest type or define a public interface (pending)
-    - Replace GPG subprocess with pure-Go crypto (pending); add timeouts to
-      remaining subprocess calls
-- CLI:
-    - Kebab-case primary flag names; fix fetch URL construction with
-      url.JoinPath; add http.Client timeout and retry with backoff to fetch;
-      rate-limit Checker progress output; add --deterministic flag or default;
-      wire top-level --version properly
-- Testing:
-    - Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch
-- Documentation:
-    - Promote docs/FORMAT.md as primary spec reference; audit error messages;
-      document the signature scheme fully
-- Release:
-    - Finalize module path, bump version constant, SemVer --version output, tag
-      v1.0.0

bin/gitrev.sh

@@ -3,5 +3,5 @@
 if [[ ! -z "$GITREV" ]]; then
     echo $GITREV
 else
-    git describe --always --dirty=-dirty
+    git describe --tags --always --dirty=-dirty
 fi

internal/cli/entry_test.go

@@ -126,7 +126,7 @@ func TestHelpCommand(t *testing.T) {
 	stdout := testStdout(t, opts)
 	assert.Contains(t, stdout, cmdGenerate)
 	assert.Contains(t, stdout, cmdCheck)
-	assert.Contains(t, stdout, "fetch")
+	assert.Contains(t, stdout, cmdFetch)
 }
 
 func TestGenerateCommand(t *testing.T) {
@@ -679,11 +679,13 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
 	fs := afero.NewMemMapFs()
 	rng := rand.New(rand.NewSource(42)) //nolint:gosec // deterministic test data
 
-	// Create many small files with random names to generate a ~1MB manifest
-	// Each manifest entry is roughly 50-60 bytes, so we need ~20000 files
+	// Create many small files with random names so the manifest has many
+	// entries and random single-byte flips land at varied offsets. Each
+	// manifest entry is roughly 50-60 bytes. Kept modest so the suite stays
+	// within its wall-clock budget under -race.
 	require.NoError(t, fs.MkdirAll(testDir, 0o755))
 
-	numFiles := 20000
+	numFiles := 1500
 	for range numFiles {
 		// Generate random filename
 		filename := fmt.Sprintf("/testdir/%08x%08x%08x.dat",
@@ -699,11 +701,11 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
 	exitCode := runCLI(opts)
 	require.Equal(t, 0, exitCode, "generate should succeed")
 
-	// Read the valid manifest and verify it's approximately 1MB
+	// Read the valid manifest and verify it has real size.
 	validManifest, err := afero.ReadFile(fs, testManifest)
 	require.NoError(t, err)
-	require.GreaterOrEqual(t, len(validManifest), 1024*1024,
-		"manifest should be at least 1MB, got %d bytes", len(validManifest))
+	require.GreaterOrEqual(t, len(validManifest), 64*1024,
+		"manifest should be at least 64KB, got %d bytes", len(validManifest))
 	t.Logf("manifest size: %d bytes (%d files)", len(validManifest), numFiles)
 
 	// First corruption: truncate the manifest
@@ -726,8 +728,8 @@ func TestCheckDetectsManifestCorruption(t *testing.T) {
 	exitCode = runCLI(opts)
 	require.Equal(t, 0, exitCode, "check should pass with valid manifest")
 
-	// Now do 500 random corruption iterations
-	for i := range 500 {
+	// Now do 100 random corruption iterations
+	for i := range 100 {
 		// Corrupt: write a random byte at a random offset
 		corrupted := make([]byte, len(validManifest))
 		copy(corrupted, validManifest)

internal/cli/errmsg_test.go

@@ -2,167 +2,352 @@
 package cli
 
 import (
-	"fmt"
+	"bytes"
+	"context"
+	"flag"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"os/exec"
+	"path/filepath"
 	"testing"
 
+	"github.com/spf13/afero"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	urfcli "github.com/urfave/cli/v2"
+	"sneak.berlin/go/mfer/mfer"
 )
 
-// errMsgCase is one pinned user-visible error message.
-type errMsgCase struct {
-	name string
-	err  error
-	want string
-}
+// These tests pin the exact rendered text of the CLI's user-visible error
+// messages. The messages are grepped for in CI pipelines and quoted in bug
+// reports, so a reword is a deliberate change, never a refactoring side
+// effect.
+//
+// Every case drives the real function that emits the message and asserts on
+// what it returns. No production format string is restated here: a test that
+// only re-rendered a copied format string would keep passing after the real
+// message changed, which is exactly the regression these tests exist to
+// catch.
 
+// Full 40-hex fingerprints used where a message embeds one.
 const (
 	msgFpA = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
 	msgFpB = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"
 )
 
-func checkErrMsgCases(t *testing.T, cases []errMsgCase) {
+// runLocked runs fn while holding runMu, so operations that write to the
+// process-global logger do not race the other CLI runs.
+func runLocked(fn func() error) error {
+	runMu.Lock()
+	defer runMu.Unlock()
+
+	return fn()
+}
+
+// unsignedChecker builds a Checker over a freshly scanned, unsigned manifest.
+func unsignedChecker(t *testing.T) *mfer.Checker {
 	t.Helper()
 
-	for _, tc := range cases {
-		t.Run(tc.name, func(t *testing.T) {
-			t.Parallel()
-			assert.Equal(t, tc.want, tc.err.Error())
-		})
-	}
+	fs := afero.NewMemMapFs()
+	require.NoError(t, fs.MkdirAll("/d", 0o755))
+	require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644))
+
+	s := mfer.NewScannerWithOptions(&mfer.ScannerOptions{Fs: fs})
+	require.NoError(t, s.EnumeratePath("/d", nil))
+
+	var buf bytes.Buffer
+
+	require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
+	require.NoError(t, afero.WriteFile(fs, "/d/index.mf", buf.Bytes(), 0o644))
+
+	chk, err := mfer.NewChecker("/d/index.mf", "/d", fs)
+	require.NoError(t, err)
+	require.False(t, chk.IsSigned())
+
+	return chk
 }
 
-// TestErrorMessagesVerbatim pins the exact rendered text of the CLI's
-// user-visible error messages.
+func TestNoManifestFoundMessage(t *testing.T) {
+	t.Parallel()
+
+	_, err := findManifest(afero.NewMemMapFs(), "/tmp/x")
+	require.ErrorIs(t, err, errNoManifestFound)
+	assert.EqualError(t, err,
+		"no manifest found in /tmp/x (looked for index.mf and .index.mf)")
+}
+
+func TestVerifyRequiredSignerMessages(t *testing.T) {
+	t.Parallel()
+
+	t.Run("invalid fingerprint length", func(t *testing.T) {
+		t.Parallel()
+
+		err := verifyRequiredSigner(unsignedChecker(t), "12345678")
+		require.ErrorIs(t, err, errInvalidFingerprint)
+		assert.EqualError(t, err,
+			"invalid fingerprint: must be exactly 40 hex characters, got 8")
+	})
+
+	t.Run("manifest not signed", func(t *testing.T) {
+		t.Parallel()
+
+		err := verifyRequiredSigner(unsignedChecker(t), msgFpA)
+		require.ErrorIs(t, err, errManifestNotSigned)
+		assert.EqualError(t, err,
+			"manifest is not signed, but signature from "+msgFpA+" is required")
+	})
+}
+
+// TestSignerMismatchMessage drives verifyRequiredSigner against a real signed
+// manifest. The embedded fingerprint is whatever the generated key produced,
+// so it is read back from the checker and substituted into the expected
+// string; the required signer is a fixed value that cannot match it. Requires
+// gpg and is skipped where it is absent, as the other signing tests are.
 //
-// These strings are an interface: they are grepped for in CI pipelines
-// and quoted in bug reports. The messages are assembled by wrapping
-// static sentinels, and it is easy to change what a user sees while
-// only meaning to make an error matchable with errors.Is - which is
-// precisely what happened once already. Any change to a string below is
-// therefore a deliberate, separately stated change, never a side effect
-// of a refactor.
-func TestErrorMessagesVerbatim(t *testing.T) {
+//nolint:paralleltest // signedChecker calls t.Setenv, which bars t.Parallel
+func TestSignerMismatchMessage(t *testing.T) {
+	chk := signedChecker(t)
+
+	embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP()
+	require.NoError(t, err)
+
+	err = verifyRequiredSigner(chk, msgFpB)
+	require.ErrorIs(t, err, errSignerMismatch)
+	assert.EqualError(t, err,
+		"embedded signing key fingerprint "+embeddedFP+
+			" does not match required "+msgFpB)
+}
+
+// signedChecker builds a Checker over a manifest signed by a throwaway GPG
+// key generated in a temporary GNUPGHOME.
+func signedChecker(t *testing.T) *mfer.Checker {
+	t.Helper()
+
+	_, err := exec.LookPath("gpg")
+	if err != nil {
+		t.Skip("gpg not installed, skipping signing test")
+	}
+
+	gpgHome := t.TempDir()
+	params := "%no-protection\n" +
+		"Key-Type: RSA\nKey-Length: 2048\n" +
+		"Name-Real: MFER Test Key\nName-Email: test@mfer.test\n" +
+		"Expire-Date: 0\n%commit\n"
+	paramsFile := filepath.Join(gpgHome, "key-params")
+	require.NoError(t, os.WriteFile(paramsFile, []byte(params), 0o600))
+
+	//nolint:gosec // paramsFile is a test-controlled path inside t.TempDir()
+	cmd := exec.CommandContext(context.Background(), "gpg",
+		"--batch", "--gen-key", paramsFile)
+
+	cmd.Env = append(os.Environ(), "GNUPGHOME="+gpgHome)
+
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Skipf("failed to generate test GPG key: %v: %s", err, out)
+	}
+
+	t.Setenv("GNUPGHOME", gpgHome)
+
+	b := mfer.NewBuilder()
+	b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
+
+	content := []byte("signed file")
+	_, err = b.AddFile("f.txt", mfer.FileSize(len(content)), mfer.ModTime{},
+		bytes.NewReader(content), nil)
+	require.NoError(t, err)
+
+	var buf bytes.Buffer
+
+	require.NoError(t, b.Build(&buf))
+
+	fs := afero.NewMemMapFs()
+	require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
+
+	chk, err := mfer.NewChecker("/index.mf", "/", fs)
+	require.NoError(t, err)
+	require.True(t, chk.IsSigned())
+
+	return chk
+}
+
+func TestPathDoesNotExistMessage(t *testing.T) {
 	t.Parallel()
 
-	checkErrMsgCases(t, []errMsgCase{
-		{
-			name: "check: no manifest found",
-			err: fmt.Errorf("%w in %s (looked for index.mf and .index.mf)",
-				errNoManifestFound, "/tmp/x"),
-			want: "no manifest found in /tmp/x " +
-				"(looked for index.mf and .index.mf)",
-		},
-		{
-			name: "check: invalid fingerprint length",
-			err:  fmt.Errorf("%w, got %d", errInvalidFingerprint, 8),
-			want: "invalid fingerprint: must be exactly 40 hex characters, got 8",
-		},
-		{
-			name: "check: manifest not signed",
-			err: fmt.Errorf("%w, but signature from %s is required",
-				errManifestNotSigned, msgFpA),
-			want: "manifest is not signed, but signature from " + msgFpA +
-				" is required",
-		},
-		{
-			name: "check: signer mismatch",
-			err: fmt.Errorf("embedded signing key fingerprint %s %w %s",
-				msgFpA, errSignerMismatch, msgFpB),
-			want: "embedded signing key fingerprint " + msgFpA +
-				" does not match required " + msgFpB,
-		},
-		{
-			name: "gen: path does not exist",
-			err:  fmt.Errorf("%w: %s", errPathNotExist, "nope"),
-			want: "path does not exist: nope",
-		},
-		{
-			name: "gen: output file exists",
-			err:  fmt.Errorf("output file %s %w", "index.mf", errOutputExists),
-			want: "output file index.mf already exists " +
-				"(use --force to overwrite)",
-		},
-		{
-			name: "mfer: unknown command",
-			err:  fmt.Errorf("%w %q", errUnknownCommand, "bogus"),
-			want: `unknown command "bogus"`,
-		},
+	set := flag.NewFlagSet("gen", flag.ContinueOnError)
+	require.NoError(t, set.Parse([]string{"nope"}))
+
+	mfa := &CLIApp{Fs: afero.NewMemMapFs()}
+	ctx := urfcli.NewContext(nil, set, nil)
+
+	_, err := mfa.collectInputPaths(ctx.Args())
+	require.ErrorIs(t, err, errPathNotExist)
+	assert.EqualError(t, err, "path does not exist: nope")
+}
+
+func TestOutputFileExistsMessage(t *testing.T) {
+	t.Parallel()
+
+	fs := afero.NewMemMapFs()
+	require.NoError(t, fs.MkdirAll("/d", 0o755))
+	require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644))
+	require.NoError(t, afero.WriteFile(fs, "/out.mf", []byte("old"), 0o644))
+
+	set := flag.NewFlagSet("gen", flag.ContinueOnError)
+	set.String("output", "", "")
+	set.Bool("force", false, "")
+	require.NoError(t, set.Parse([]string{"/d"}))
+	require.NoError(t, set.Set("output", "/out.mf"))
+
+	mfa := &CLIApp{Fs: fs}
+	ctx := urfcli.NewContext(nil, set, nil)
+
+	// generateManifestOperation writes to the process-global logger during
+	// enumeration, so serialize with the other CLI runs.
+	err := runLocked(func() error { return mfa.generateManifestOperation(ctx) })
+	require.ErrorIs(t, err, errOutputExists)
+	assert.EqualError(t, err,
+		"output file /out.mf already exists (use --force to overwrite)")
+}
+
+// TestUnknownCommandMessage drives the root command's action. run only logs
+// the error that action returns, so the test lets run build the app with no
+// command given and then runs that same app on an unknown command to get the
+// error itself.
+func TestUnknownCommandMessage(t *testing.T) {
+	t.Parallel()
+
+	mfa := &CLIApp{
+		appname: testApp,
+		Stdout:  &bytes.Buffer{},
+		Stderr:  &bytes.Buffer{},
+		Fs:      afero.NewMemMapFs(),
+	}
+
+	// run points the process-global logger at this app's output, so
+	// serialize with the other CLI runs.
+	err := runLocked(func() error {
+		mfa.run([]string{testApp})
+
+		return mfa.app.Run([]string{testApp, "bogus"})
+	})
+	require.ErrorIs(t, err, errUnknownCommand)
+	assert.EqualError(t, err, `unknown command "bogus"`)
+}
+
+func TestManifestLoaderHTTPStatusMessage(t *testing.T) {
+	t.Parallel()
+
+	server := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusNotFound)
+		}))
+	defer server.Close()
+
+	mfa := &CLIApp{Fs: afero.NewMemMapFs()}
+
+	_, err := mfa.openManifestReader(server.URL + "/foo.mf")
+	require.ErrorIs(t, err, errHTTPStatus)
+	assert.EqualError(t, err,
+		"failed to fetch "+server.URL+"/foo.mf: HTTP 404")
+}
+
+func TestFetchManifestHTTPStatusMessage(t *testing.T) {
+	t.Parallel()
+
+	server := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusNotFound)
+		}))
+	defer server.Close()
+
+	set := flag.NewFlagSet("fetch", flag.ContinueOnError)
+	require.NoError(t, set.Parse([]string{server.URL}))
+
+	mfa := &CLIApp{Fs: afero.NewMemMapFs()}
+	ctx := urfcli.NewContext(nil, set, nil)
+
+	// fetchManifestOperation logs to the process-global logger.
+	err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) })
+	require.ErrorIs(t, err, errHTTPStatus)
+	assert.EqualError(t, err, "failed to fetch manifest: HTTP 404")
+}
+
+func TestFetchFileHTTPStatusMessage(t *testing.T) {
+	t.Parallel()
+
+	server := httptest.NewServer(
+		http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+			w.WriteHeader(http.StatusInternalServerError)
+		}))
+	defer server.Close()
+
+	err := downloadFile(context.Background(), server.URL+"/x", "x",
+		&mfer.MFFilePath{}, nil)
+	require.ErrorIs(t, err, errHTTPStatus)
+	assert.EqualError(t, err, "HTTP 500")
+}
+
+func TestURLRequiredMessage(t *testing.T) {
+	t.Parallel()
+
+	set := flag.NewFlagSet("fetch", flag.ContinueOnError)
+	require.NoError(t, set.Parse([]string{}))
+
+	mfa := &CLIApp{Fs: afero.NewMemMapFs()}
+	ctx := urfcli.NewContext(nil, set, nil)
+
+	// fetchManifestOperation logs to the process-global logger.
+	err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) })
+	require.ErrorIs(t, err, errURLRequired)
+	assert.EqualError(t, err, "URL argument required")
+}
+
+func TestSanitizePathMessages(t *testing.T) {
+	t.Parallel()
+
+	t.Run("empty", func(t *testing.T) {
+		t.Parallel()
+
+		_, err := sanitizePath("")
+		require.ErrorIs(t, err, errEmptyPath)
+		assert.EqualError(t, err, "empty path")
+	})
+
+	t.Run("absolute", func(t *testing.T) {
+		t.Parallel()
+
+		_, err := sanitizePath("/etc/passwd")
+		require.ErrorIs(t, err, errAbsolutePath)
+		assert.EqualError(t, err, "absolute path not allowed: /etc/passwd")
+	})
+
+	t.Run("traversal", func(t *testing.T) {
+		t.Parallel()
+
+		_, err := sanitizePath("../x")
+		require.ErrorIs(t, err, errPathTraversal)
+		assert.EqualError(t, err, "path traversal not allowed: ../x")
 	})
 }
 
-// TestFetchErrorMessagesVerbatim pins the fetch and manifest-loader
-// messages; see TestErrorMessagesVerbatim for why.
-func TestFetchErrorMessagesVerbatim(t *testing.T) {
+func TestSizeMismatchMessage(t *testing.T) {
 	t.Parallel()
 
-	checkErrMsgCases(t, []errMsgCase{
-		{
-			name: "manifest_loader: http status",
-			err: fmt.Errorf("failed to fetch %s: %w %d",
-				"https://example.com/index.mf", errHTTPStatus, 404),
-			want: "failed to fetch https://example.com/index.mf: HTTP 404",
-		},
-		{
-			name: "fetch: manifest http status",
-			err: fmt.Errorf("failed to fetch manifest: %w %d",
-				errHTTPStatus, 404),
-			want: "failed to fetch manifest: HTTP 404",
-		},
-		{
-			name: "fetch: file http status",
-			err:  fmt.Errorf("%w %d", errHTTPStatus, 500),
-			want: "HTTP 500",
-		},
-		{
-			name: "fetch: empty path",
-			err:  errEmptyPath,
-			want: "empty path",
-		},
-		{
-			name: "fetch: absolute path",
-			err:  fmt.Errorf("%w: %s", errAbsolutePath, "/etc/passwd"),
-			want: "absolute path not allowed: /etc/passwd",
-		},
-		{
-			name: "fetch: path traversal",
-			err:  fmt.Errorf("%w: %s", errPathTraversal, "../x"),
-			want: "path traversal not allowed: ../x",
-		},
-		{
-			name: "fetch: size mismatch",
-			err: fmt.Errorf("%w: expected %d bytes, got %d",
-				errSizeMismatch, 10, 9),
-			want: "size mismatch: expected 10 bytes, got 9",
-		},
-		{
-			name: "fetch: url required",
-			err:  errURLRequired,
-			want: "URL argument required",
-		},
-		{
-			name: "fetch: hash mismatch",
-			err:  errHashMismatch,
-			want: "hash mismatch",
-		},
-	})
+	// finishDownload returns the size-mismatch error before it touches the
+	// paths, digest, or entry, so those can be zero here.
+	err := finishDownload("", "", 9, 10, nil, nil, nil, nil)
+	require.ErrorIs(t, err, errSizeMismatch)
+	assert.EqualError(t, err, "size mismatch: expected 10 bytes, got 9")
 }
 
-// TestSentinelsAreMatchable checks that the wrapped forms of the
-// messages above remain matchable with errors.Is, which is the reason
-// the sentinels exist at all.
-func TestSentinelsAreMatchable(t *testing.T) {
+func TestHashMismatchMessage(t *testing.T) {
 	t.Parallel()
 
-	wrapped := fmt.Errorf("embedded signing key fingerprint %s %w %s",
-		"a", errSignerMismatch, "b")
-	require.ErrorIs(t, wrapped, errSignerMismatch)
-
-	wrapped = fmt.Errorf("output file %s %w", "index.mf", errOutputExists)
-	require.ErrorIs(t, wrapped, errOutputExists)
-
-	wrapped = fmt.Errorf("failed to fetch manifest: %w %d", errHTTPStatus, 404)
-	require.ErrorIs(t, wrapped, errHTTPStatus)
-
-	assert.NotErrorIs(t, errHashMismatch, errSizeMismatch)
+	// A 32-byte digest that matches none of the (empty) manifest hashes.
+	err := verifyDownloadedHash(make([]byte, 32), &mfer.MFFilePath{})
+	require.ErrorIs(t, err, errHashMismatch)
+	require.NotErrorIs(t, err, errSizeMismatch)
+	assert.EqualError(t, err, "hash mismatch")
 }

internal/cli/fetch.go

@@ -36,6 +36,11 @@ const (
 	// traversal bit for group and other must stay set.
 	dirPerms os.FileMode = 0o755
 
+	// filePerms is the permission mode, before the umask, for downloaded
+	// files. It is the mode os.Create uses; like dirPerms, it keeps group
+	// and other read access.
+	filePerms os.FileMode = 0o666
+
 	// Bitrate unit thresholds in bits per second.
 	bpsPerGbps = 1e9
 	bpsPerMbps = 1e6
@@ -53,6 +58,9 @@ var (
 	// errPathTraversal indicates a manifest path escaping the target
 	// directory.
 	errPathTraversal = errors.New("path traversal not allowed")
+	// errSymlinkInPath indicates a manifest path running through a
+	// symlink that already exists in the target directory.
+	errSymlinkInPath = errors.New("symlink in path not allowed")
 	// errSizeMismatch indicates a downloaded file with an unexpected
 	// size.
 	errSizeMismatch = errors.New("size mismatch")
@@ -274,6 +282,35 @@ func sanitizePath(p string) (string, error) {
 	return cleaned, nil
 }
 
+// checkNoSymlinks returns an error if any part of the relative path p
+// already exists as a symlink. sanitizePath checks p only as text, so
+// without this a symlink inside the target directory could send a write
+// to p outside of it. Parts that do not exist yet are fine: fetch creates
+// them as plain directories and files. Call it immediately before each
+// write: a symlink created after it returns is not caught.
+func checkNoSymlinks(p string) error {
+	current := ""
+
+	for _, part := range strings.Split(p, string(filepath.Separator)) {
+		current = filepath.Join(current, part)
+
+		info, err := os.Lstat(current)
+		if errors.Is(err, os.ErrNotExist) {
+			return nil
+		}
+
+		if err != nil {
+			return fmt.Errorf("failed to check %s for a symlink: %w", current, err)
+		}
+
+		if info.Mode()&os.ModeSymlink != 0 {
+			return fmt.Errorf("%w: %s", errSymlinkInPath, current)
+		}
+	}
+
+	return nil
+}
+
 // resolveManifestURL takes a URL and returns the manifest URL.
 // If the URL already ends with .mf, it's returned as-is.
 // Otherwise, index.mf is appended.
@@ -419,7 +456,12 @@ func downloadFile(
 	// Create parent directories if needed
 	dir := filepath.Dir(localPath)
 	if dir != "" && dir != "." {
-		err := os.MkdirAll(dir, dirPerms)
+		err = checkNoSymlinks(dir)
+		if err != nil {
+			return err
+		}
+
+		err = os.MkdirAll(dir, dirPerms)
 		if err != nil {
 			return fmt.Errorf("failed to create directory %s: %w", dir, err)
 		}
@@ -447,15 +489,25 @@ func downloadFile(
 		totalBytes = expectedSize
 	}
 
-	// Create temp file.
+	err = checkNoSymlinks(tmpPath)
+	if err != nil {
+		return err
+	}
+
+	// Remove whatever is at tmpPath, such as a leftover from an
+	// interrupted run, rather than write into it: it may be a hard link
+	// to a file outside the target directory, and removing a hard link
+	// removes only this name. If the removal fails, O_EXCL below makes
+	// the create fail.
+	_ = os.Remove(tmpPath)
+
+	// Create the temp file only if nothing is at tmpPath (O_EXCL).
 	//
-	// G304: tmpPath is derived from localPath, which sanitizePath above
-	// constrains lexically to a relative path that does not escape the
-	// destination directory. That is a purely lexical guarantee: it does
-	// not resolve symlinks, so a pre-existing symlink inside the
-	// destination tree can still redirect this write outside of it
-	// (tracked in issue #86).
-	out, err := os.Create(tmpPath) //nolint:gosec // G304: see comment above
+	// G304: tmpPath is a relative path that sanitizePath keeps inside the
+	// target directory as text, and checkNoSymlinks just found no symlink
+	// in it.
+	out, err := os.OpenFile( //nolint:gosec // G304: see comment above
+		tmpPath, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
 	if err != nil {
 		return fmt.Errorf("failed to create temp file: %w", err)
 	}
@@ -519,6 +571,11 @@ func finishDownload(
 		return err
 	}
 
+	err = checkNoSymlinks(localPath)
+	if err != nil {
+		return err
+	}
+
 	// Rename temp file to final path
 	err = os.Rename(tmpPath, localPath)
 	if err != nil {

internal/cli/fetch_test.go

@@ -440,3 +440,86 @@ func TestFetchProgress(t *testing.T) {
 	require.NoError(t, err)
 	assert.Equal(t, content, downloaded)
 }
+
+// TestFetchRefusesSymlinks runs fetch into a destination directory that
+// holds a symlink pointing outside it, in each of the three places fetch
+// writes: a parent directory, the temp file, and the file itself, which
+// the temp file is renamed onto; and once as a directory inside a plain
+// directory. The fetch must fail and nothing outside may change.
+//
+//nolint:paralleltest // changes the process-global working directory
+func TestFetchRefusesSymlinks(t *testing.T) {
+	tests := []struct {
+		name   string
+		entry  string // the manifest's only file
+		link   string // symlink placed in the destination directory
+		target string // what link points to, relative to the outside directory
+	}{
+		{"parent directory", "sub/deeper/file.txt", "sub", "."},
+		{"directory inside a plain directory", "docs/data/passwd", "docs/data", "."},
+		{"temp file", testFileTxt, ".file.txt.tmp", "new.txt"},
+		{"file", testFileTxt, testFileTxt, "new.txt"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			content := []byte("fetched")
+			sourceFs := afero.NewMemMapFs()
+			require.NoError(t, sourceFs.MkdirAll(filepath.Dir("/"+tt.entry), 0o755))
+			require.NoError(t, afero.WriteFile(sourceFs, "/"+tt.entry, content, 0o644))
+
+			server := httptest.NewServer(fetchTestHandler(
+				scanToManifest(t, sourceFs), map[string][]byte{tt.entry: content}))
+			defer server.Close()
+
+			outside := t.TempDir()
+
+			chdirTemp(t)
+			require.NoError(t, os.MkdirAll(filepath.Dir(tt.link), 0o750))
+			require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), tt.link))
+
+			opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
+			assert.Equal(t, 1, runCLI(opts))
+			assert.Contains(t, testStderr(t, opts), "failed to download "+tt.entry+
+				": symlink in path not allowed: "+tt.link)
+
+			written, err := os.ReadDir(outside)
+			require.NoError(t, err)
+			assert.Empty(t, written, "fetch wrote outside the destination")
+		})
+	}
+}
+
+// TestFetchReplacesHardLinkAtTempName runs fetch into a destination
+// directory that holds, at the temp file's name, a hard link to a file
+// outside it. To fetch that is an ordinary leftover from an interrupted
+// earlier run: it must replace it and succeed, and the outside file must
+// not change.
+//
+//nolint:paralleltest // changes the process-global working directory
+func TestFetchReplacesHardLinkAtTempName(t *testing.T) {
+	content := []byte("fetched")
+	sourceFs := afero.NewMemMapFs()
+	require.NoError(t, afero.WriteFile(sourceFs, "/"+testFileTxt, content, 0o644))
+
+	server := httptest.NewServer(fetchTestHandler(
+		scanToManifest(t, sourceFs), map[string][]byte{testFileTxt: content}))
+	defer server.Close()
+
+	outsideFile := filepath.Join(t.TempDir(), "secret.txt")
+	require.NoError(t, os.WriteFile(outsideFile, []byte("outside"), 0o600))
+
+	chdirTemp(t)
+	require.NoError(t, os.Link(outsideFile, ".file.txt.tmp"))
+
+	opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
+	require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
+
+	fetched, err := os.ReadFile(testFileTxt)
+	require.NoError(t, err)
+	assert.Equal(t, content, fetched)
+
+	outside, err := os.ReadFile(outsideFile) //nolint:gosec // test-controlled path
+	require.NoError(t, err)
+	assert.Equal(t, "outside", string(outside), "fetch wrote outside the destination")
+}

internal/cli/freshen_test.go

@@ -110,7 +110,10 @@ func TestFreshenRecordEntryMtimePresence(t *testing.T) {
 
 	const relPath = "file1.txt"
 
-	mtime := time.Unix(1_700_000_000, 0)
+	// The scanned file's mtime is the Unix epoch. If recordEntry ever misreads
+	// an absent manifest mtime as the epoch, the "absent" case below would
+	// compare equal to this and be classified unchanged, so the test fails.
+	mtime := time.Unix(0, 0)
 	info := stubFileInfo{size: 8, mtime: mtime}
 
 	for _, tc := range []struct {

internal/cli/mfer.go

@@ -18,6 +18,7 @@ const (
 	cmdGenerate = "generate"
 	cmdCheck    = "check"
 	cmdExport   = "export"
+	cmdFetch    = "fetch"
 
 	flagProgress = "progress"
 
@@ -300,7 +301,7 @@ func (mfa *CLIApp) listCommand() *cli.Command {
 
 func (mfa *CLIApp) fetchCommand() *cli.Command {
 	return &cli.Command{
-		Name:  "fetch",
+		Name:  cmdFetch,
 		Usage: "fetch manifest and referenced files",
 		Action: func(c *cli.Context) error {
 			mfa.setVerbosity(c)
@@ -357,6 +358,6 @@ func (mfa *CLIApp) run(args []string) {
 	if err != nil {
 		mfa.exitCode = 1
 
-		log.WithError(err).Debugf("exiting")
+		log.Errorf("%s", err)
 	}
 }

internal/log/log.go

@@ -112,13 +112,16 @@ func DisableStyling() {
 }
 
 // Init initializes the logger with the CLI handler and default log level.
+//
+// It reconfigures the process-global apex/log logger under the write lock so
+// the global is never mutated while another goroutine holds the read lock to
+// read it in emit. Without this, parallel callers (e.g. the test suite) race
+// Init's SetLevel/SetHandler against concurrent log calls.
 func Init() {
-	mu.RLock()
+	mu.Lock()
+	defer mu.Unlock()
 
-	w := stderr
-
-	mu.RUnlock()
-	log.SetHandler(acli.New(w))
+	log.SetHandler(acli.New(stderr))
 	log.SetLevel(log.DebugLevel) // Let apex/log pass everything; we filter ourselves
 }
 
@@ -130,74 +133,66 @@ func isEnabled(l Level) bool {
 	return l >= currentLevel
 }
 
+// emit calls fn while holding the read lock if messages at level l are
+// enabled. Holding the read lock across the apex/log call keeps the global
+// logger from being read while Init reconfigures it under the write lock.
+func emit(l Level, fn func()) {
+	mu.RLock()
+	defer mu.RUnlock()
+
+	if l >= currentLevel {
+		fn()
+	}
+}
+
 // Fatalf logs a formatted message at fatal level.
 func Fatalf(format string, args ...any) {
-	if isEnabled(FatalLevel) {
-		log.Fatalf(format, args...)
-	}
+	emit(FatalLevel, func() { log.Fatalf(format, args...) })
 }
 
 // Fatal logs a message at fatal level.
 func Fatal(arg string) {
-	if isEnabled(FatalLevel) {
-		log.Fatal(arg)
-	}
+	emit(FatalLevel, func() { log.Fatal(arg) })
 }
 
 // Errorf logs a formatted message at error level.
 func Errorf(format string, args ...any) {
-	if isEnabled(ErrorLevel) {
-		log.Errorf(format, args...)
-	}
+	emit(ErrorLevel, func() { log.Errorf(format, args...) })
 }
 
 // Error logs a message at error level.
 func Error(arg string) {
-	if isEnabled(ErrorLevel) {
-		log.Error(arg)
-	}
+	emit(ErrorLevel, func() { log.Error(arg) })
 }
 
 // Warnf logs a formatted message at warn level.
 func Warnf(format string, args ...any) {
-	if isEnabled(WarnLevel) {
-		log.Warnf(format, args...)
-	}
+	emit(WarnLevel, func() { log.Warnf(format, args...) })
 }
 
 // Warn logs a message at warn level.
 func Warn(arg string) {
-	if isEnabled(WarnLevel) {
-		log.Warn(arg)
-	}
+	emit(WarnLevel, func() { log.Warn(arg) })
 }
 
 // Infof logs a formatted message at info level.
 func Infof(format string, args ...any) {
-	if isEnabled(InfoLevel) {
-		log.Infof(format, args...)
-	}
+	emit(InfoLevel, func() { log.Infof(format, args...) })
 }
 
 // Info logs a message at info level.
 func Info(arg string) {
-	if isEnabled(InfoLevel) {
-		log.Info(arg)
-	}
+	emit(InfoLevel, func() { log.Info(arg) })
 }
 
 // Verbosef logs a formatted message at verbose level.
 func Verbosef(format string, args ...any) {
-	if isEnabled(VerboseLevel) {
-		log.Infof(format, args...)
-	}
+	emit(VerboseLevel, func() { log.Infof(format, args...) })
 }
 
 // Verbose logs a message at verbose level.
 func Verbose(arg string) {
-	if isEnabled(VerboseLevel) {
-		log.Info(arg)
-	}
+	emit(VerboseLevel, func() { log.Info(arg) })
 }
 
 // Debugf logs a formatted message at debug level with caller location.
@@ -216,7 +211,10 @@ func Debug(arg string) {
 
 // DebugReal logs at debug level with caller info from the specified stack depth.
 func DebugReal(arg string, cs int) {
-	if !isEnabled(DebugLevel) {
+	mu.RLock()
+	defer mu.RUnlock()
+
+	if DebugLevel < currentLevel {
 		return
 	}
 
@@ -275,11 +273,6 @@ func GetLevel() Level {
 	return currentLevel
 }
 
-// WithError returns a log entry with the error attached.
-func WithError(e error) *log.Entry {
-	return log.Log.WithError(e)
-}
-
 // Progressf prints a progress message that overwrites the current line.
 // Use ProgressDone() when progress is complete to move to the next line.
 func Progressf(format string, args ...any) {

mfer/checker.go

@@ -312,6 +312,10 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
 }
 
 func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
+	// entry.GetPath() is safe to join here: a manifest's entry paths are
+	// validated against the path invariants when it is loaded (see
+	// deserializeInner) or built (see Builder.AddFile), so a traversal or
+	// absolute path can never reach this point.
 	absPath := filepath.Join(string(c.basePath), entry.GetPath())
 	relPath := RelFilePath(entry.GetPath())
 

mfer/deserialize.go

@@ -25,6 +25,7 @@ var (
 	errDecompressedTooLarge = errors.New("decompressed data exceeds maximum allowed size")
 	errUUIDMismatch         = errors.New("outer and inner UUID mismatch")
 	errInvalidFileFormat    = errors.New("invalid file format")
+	errInvalidManifestPath  = errors.New("manifest contains invalid path")
 )
 
 // validateUUID checks that the byte slice is a valid UUID (16 bytes, parseable).
@@ -181,6 +182,19 @@ func (m *manifest) deserializeInner() error {
 		return errUUIDMismatch
 	}
 
+	// Enforce the manifest path invariants on every entry as it is loaded,
+	// so that no consumer of a manifest — Checker today, any restore or
+	// extract path tomorrow — acts on a traversal or absolute path from an
+	// untrusted .mf. Reject loudly on the first offender rather than
+	// dropping entries, which would let a hostile manifest hide files from a
+	// check.
+	for _, f := range m.pbInner.GetFiles() {
+		err = ValidatePath(f.GetPath())
+		if err != nil {
+			return fmt.Errorf("%w: %w", errInvalidManifestPath, err)
+		}
+	}
+
 	log.Infof("loaded manifest with %d files", len(m.pbInner.GetFiles()))
 
 	return nil

mfer/deserialize_path_test.go

@@ -0,0 +1,145 @@
+//nolint:testpackage // white-box tests exercise unexported internals
+package mfer
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"fmt"
+	"testing"
+
+	"github.com/google/uuid"
+	"github.com/klauspost/compress/zstd"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/protobuf/encoding/protowire"
+	"google.golang.org/protobuf/proto"
+)
+
+// craftInnerBytes builds the wire bytes of an inner MFFile holding a single
+// file entry whose path is exactly pathBytes. It writes the wire form by hand
+// so a hostile path — including one that is not valid UTF-8 — can be embedded
+// without proto.Marshal's own UTF-8 enforcement rejecting it first.
+func craftInnerBytes(id uuid.UUID, pathBytes string) []byte {
+	entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
+	entry = protowire.AppendString(entry, pathBytes)
+
+	inner := protowire.AppendTag(nil, 100, protowire.VarintType) // MFFile.version
+	inner = protowire.AppendVarint(inner, uint64(MFFile_VERSION_ONE))
+	inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
+	inner = protowire.AppendBytes(inner, entry)
+	inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
+	inner = protowire.AppendBytes(inner, id[:])
+
+	return inner
+}
+
+// wrapInner wraps inner MFFile wire bytes in a complete, well-formed .mf
+// envelope (magic prefix, zstd-compressed payload, matching hash and UUID) so
+// that deserialization reaches path validation rather than failing earlier on
+// an integrity check.
+func wrapInner(t *testing.T, id uuid.UUID, innerData []byte) []byte {
+	t.Helper()
+
+	var cbuf bytes.Buffer
+
+	zw, err := zstd.NewWriter(&cbuf, zstd.WithEncoderLevel(zstd.SpeedBestCompression))
+	require.NoError(t, err)
+
+	_, err = zw.Write(innerData)
+	require.NoError(t, err)
+	require.NoError(t, zw.Close())
+
+	compressed := cbuf.Bytes()
+	sum := sha256.Sum256(compressed)
+
+	outer := &MFFileOuter{
+		InnerMessage:    compressed,
+		Size:            int64(len(innerData)),
+		Sha256:          sum[:],
+		Uuid:            id[:],
+		Version:         MFFileOuter_VERSION_ONE,
+		CompressionType: MFFileOuter_COMPRESSION_ZSTD,
+	}
+
+	ob, err := proto.Marshal(outer)
+	require.NoError(t, err)
+
+	return append([]byte(MAGIC), ob...)
+}
+
+func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name string
+		path string
+	}{
+		{"parent traversal", "../escape"},
+		{"interior traversal", "a/../../escape"},
+		{"absolute path", "/etc/passwd"},
+		{"backslash path", `a\b`},
+		{"double slash", "a//b"},
+		{"empty path", ""},
+		{"invalid utf-8", "abc\xff"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			id := uuid.New()
+			data := wrapInner(t, id, craftInnerBytes(id, tt.path))
+
+			_, err := NewManifestFromReader(bytes.NewReader(data))
+			require.Error(t, err)
+
+			if tt.path == "abc\xff" {
+				// A path that is not valid UTF-8 cannot survive the proto3
+				// string decoder, which rejects it before path validation
+				// runs; the manifest is still refused at load time.
+				return
+			}
+
+			require.ErrorIs(t, err, errInvalidManifestPath)
+
+			if tt.path != "" {
+				// ValidatePath quotes the path with %q; assert against the
+				// same rendering so escaped characters (e.g. a backslash)
+				// still match.
+				assert.Contains(t, err.Error(), fmt.Sprintf("%q", tt.path),
+					"error must name the offending path")
+			}
+		})
+	}
+}
+
+func TestDeserializeValidManifestRoundTrips(t *testing.T) {
+	t.Parallel()
+
+	hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
+
+	b := NewBuilder()
+	require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
+
+	var buf bytes.Buffer
+	require.NoError(t, b.Build(&buf))
+
+	m, err := NewManifestFromReader(bytes.NewReader(buf.Bytes()))
+	require.NoError(t, err)
+
+	files := m.Files()
+	require.Len(t, files, 1)
+	assert.Equal(t, "dir/file.txt", files[0].GetPath())
+	assert.Equal(t, int64(123), files[0].GetSize())
+}
+
+// TestValidatePathRejectsInvalidUTF8 pins the ValidatePath rule that a manifest
+// path must be valid UTF-8, independent of the proto decoder that also enforces
+// it on the wire.
+func TestValidatePathRejectsInvalidUTF8(t *testing.T) {
+	t.Parallel()
+
+	err := ValidatePath("abc\xff")
+	require.ErrorIs(t, err, errPathNotUTF8)
+	assert.Contains(t, err.Error(), "UTF-8")
+}

script/cibuild

@@ -1,14 +1,24 @@
 #!/bin/sh
-# script/cibuild: run the CI build. The Dockerfile runs script/check
-# (via make check), so a successful build implies all checks pass.
-# Generic: needs no adaptation. The Gitea workflow runs this on push.
+# script/cibuild: run the CI build; the Gitea workflow runs this on push.
+# It builds the image with the same command as script/docker. --no-cache
+# because the checks the final stage depends on are RUN steps, and a
+# cached one is a check that did not run.
 set -eu
 
-ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    docker build .
+    # Own line: a failing command substitution inside an argument does
+    # not trip `set -e`, so the inline form degrades silently to an
+    # empty constant. The VERSION build argument takes precedence over
+    # the version a build stage derives from the .git in the context.
+    version="$(git describe --tags --always --dirty 2>/dev/null || true)"
+    [ -n "$version" ] || version="unknown"
+    docker build --no-cache \
+        --build-arg VERSION="$version" \
+        -t "$("$SCRIPT_DIR/projectname")" .
 }
 
 main "$@"

script/docker

@@ -1,7 +1,8 @@
 #!/bin/sh
 # script/docker: build the Docker image tagged with the project name.
 # Identical in all repos; the tag comes from script/projectname.
-# Generic: needs no adaptation.
+# --no-cache because the gate phases the final stage depends on are RUN
+# steps, and a cached one is a check that did not run.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -9,7 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    docker build -t "$("$SCRIPT_DIR/projectname")" .
+    # Own line: a failing command substitution inside an argument does
+    # not trip `set -e`, so the inline form degrades silently to an
+    # empty constant. The VERSION build argument takes precedence over
+    # the version a build stage derives from the .git in the context.
+    version="$(git describe --tags --always --dirty 2>/dev/null || true)"
+    [ -n "$version" ] || version="unknown"
+    docker build --no-cache \
+        --build-arg VERSION="$version" \
+        -t "$("$SCRIPT_DIR/projectname")" .
 }
 
 main "$@"

script/test

@@ -17,7 +17,12 @@ ensure_pb() {
 main() {
     cd "$ROOT"
     ensure_pb
-    go test -v --timeout 10s ./...
+    go test -timeout 30s -race -cover ./... ||
+        {
+            echo "--- Rerunning with -v for details ---"
+            go test -timeout 30s -race -v ./...
+            exit 1
+        }
 }
 
 main "$@"