Add the required README sections (closes #75)
check / check (push) Successful in 1m44s

The Description first line now names the project, purpose, category,
WTFPL license, and author. A new Getting Started section gives a
copy-pasteable build-from-source block and gen/check/fetch usage. Problem
Statement and Proposed Solution move under a new Rationale heading, the
prose kept. A new Design section documents the package layout: the mfer/
library with its committed protobuf code, the internal/cli commands,
internal/log and internal/bork, and the cmd/mfer entrypoint. Authors is
renamed Author with the canonical link.

Getting Started uses go build because the make target that builds the
binary runs protoc, which script/bootstrap does not install.

Model: opus-4-8 (implementation); opus-5-5 (rebase)
This commit is contained in:
2026-10-04 01:52:26 +00:00
parent 1adad7d3bc
commit 19a57e30b3
+65 -11
View File
@@ -1,12 +1,12 @@
# mfer
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
thin wrapper command-line utility written in [Go](https://golang.org) and first
published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
It specifies and generates `.mf` manifest files over a directory tree of files
to encapsulate metadata about them (such as cryptographic checksums or
signatures over same) to aid in archiving, downloading, and streaming, or
mirroring. The manifest files' data is serialized with Google's
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
(public domain) [Go](https://golang.org) library and command-line tool by
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
over a directory tree to encapsulate metadata about the files — such as
cryptographic checksums and signatures over same — to aid in archiving,
downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
@@ -21,6 +21,36 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
as a de-facto standard and be incorporated into other software. A compatible
javascript library is planned.
# Getting Started
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
is committed, so no `protoc` toolchain is required:
```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
go build -o bin/mfer ./cmd/mfer
```
Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:
```sh
# Write .index.mf, a manifest of the files under the current directory.
bin/mfer gen .
# Verify the files on disk against the manifest. Exits nonzero if any file
# is missing or corrupted.
bin/mfer check .index.mf
# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/
```
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
for a single command's options.
# Build Status
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
@@ -82,7 +112,9 @@ Any changes submitted to this project must also be
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions.
# Problem Statement
# Rationale
## The problem
Given a plain URL, there is no standard way to safely and programmatically
download everything "under" that URL path. `wget -r` can traverse directory
@@ -106,7 +138,7 @@ Real issues I face:
- when I download a large file via HTTP, I have no way of knowing if the file
content is what it's supposed to be
# Proposed Solution
## The solution
A standard, a manifest file format, and a tool for generating same.
@@ -138,6 +170,28 @@ The manifest file would do several important things:
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
top-level infohash for the whole manifest?
# Design
The repository is split into a reusable library and a thin command-line wrapper
around it.
- `mfer/` is the reusable library and the heart of the project: it defines the
manifest format and implements building, scanning, checking, serialization,
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
with `go get` and needs no `protoc` toolchain.
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
`check`, `freshen`, `export`, `list`, `fetch`, and `version` — that wire the
library to the command-line interface.
- `internal/log/` provides the logging used by the commands and the library.
- `internal/bork/` defines the errors the library returns when reading a
manifest that is truncated or lacks the magic bytes.
- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
into the `mfer` binary.
Everything under `internal/` is private to this repository; only the `mfer/`
package is intended for import by other software.
# Design Goals
- Replace SHASUMS/SHASUMS.asc files
@@ -271,9 +325,9 @@ Open work, open design questions included, is tracked in this repo's issues:
- Issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
# Authors
# Author
- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
- [@sneak](https://sneak.berlin)
# License