# mfer [mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed (public domain) [Go](https://golang.org) library and command-line tool by [@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files over a directory tree to encapsulate metadata about the files — such as cryptographic checksums and signatures over same — to aid in archiving, downloading, streaming, and mirroring. It was first published in 2022. The manifest files' data is serialized with Google's [protobuf serialization format](https://developers.google.com/protocol-buffers). The structure of these files can be found [in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto) which is included in the [project repository](https://git.eeqj.de/sneak/mfer). The current version is pre-1.0 and while the repo was published in 2022, there has not yet been any versioned release. [SemVer](https://semver.org) will be used for releases. This project was started by [@sneak](https://sneak.berlin) to scratch an itch in 2022 and is currently a one-person effort, though the goal is for this to emerge as a de-facto standard and be incorporated into other software. A compatible javascript library is planned. # Getting Started `mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code is committed, so no `protoc` toolchain is required: ```sh git clone https://git.eeqj.de/sneak/mfer.git cd mfer go build -o bin/mfer ./cmd/mfer ``` Generate a manifest for a directory tree, verify it later, and fetch a published tree by URL: ```sh # Write index.mf, a manifest of the files under the current directory. bin/mfer gen . # Verify the files on disk against the manifest. Exits nonzero if any file # it lists is missing or corrupted; warns about files it does not list. bin/mfer check index.mf # Download and cryptographically verify a tree published over HTTP: mfer # fetches /index.mf, then downloads every file it lists. bin/mfer fetch https://example.com/tree/ ``` Run `bin/mfer help` for the full command list, or `bin/mfer --help` for a single command's options. # Build Status CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so the formatting, lint and test steps in the `Dockerfile` run on every build. The `main` branch must always be green. # Entrypoints This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) standard: normalized scripts in `script/` are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. We provide: - `script/bootstrap` — install all dependencies (Go, Go module download, and node/yarn plus the prettier version pinned in `package.json`/`yarn.lock`), idempotently; golangci-lint is not installed, it runs only in Docker - `script/setup` — make a fresh clone ready for development: runs `script/bootstrap`, then `script/install-precommit` - `script/projectname` — output the project name (`mfer`); used by other scripts such as `script/docker` - `script/test` — run the test suite (`go test`); one test fails when `mfer/mf.proto` no longer matches the hash `script/generate` recorded - `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from `mfer/mf.proto` and record the hash of that `mfer/mf.proto` in `mfer/mf.proto.sha256`; the only thing that regenerates the committed `mfer/mf.pb.go`. It needs the exact versions that wrote the committed file, and refuses to run with any other: `protoc` 33.4 (unpack `protoc-33.4-.zip` from [its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4) and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11 (`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which installs it in `$(go env GOPATH)/bin`; `make generate` adds that directory to `PATH`) - `script/fuzz` — fuzz the manifest parser for one minute; run by hand (`make fuzz`), never by CI, while `script/test` runs its committed seed corpus as ordinary tests - `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of the `Dockerfile` (the Go format check, then the linter), uncached so it runs every time, then removes the image - `script/fmt` — format all code and docs (writes): `gofumpt` and `script/prettier --write` - `script/prettier` — run prettier over the repository's canonical file set (Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or `--check`; the single definition of that file set, so `script/fmt` and `script/fmt-check` cannot disagree about it - `script/fmt-check` — check formatting without writing: `script/fmt-check-go` plus `script/prettier --check` - `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the Docker lint stage, whose image has no node - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check` - `script/docker` — build the Docker image tagged with the project name - `script/cibuild` — CI entrypoint: builds the image with the same command as `script/docker`, uncached, so the checks in the Dockerfile run every time - `script/precommit` — pre-commit checks: `go mod tidy` verification, then `script/check` - `script/install-precommit` — install the git pre-commit hook that runs `script/precommit` # Participation The community is as yet nonexistent so there are no defined policies or norms yet. Primary development happens on a privately-run Gitea instance at [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues are [tracked there](https://git.eeqj.de/sneak/mfer/issues). Changes must always be formatted with a standard `go fmt`, syntactically valid, and must pass the linting defined in the repository's `.golangci.yml`, which `make lint` runs in Docker. The `main` branch is protected and all changes must be made via [pull requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged. Any changes submitted to this project must also be [WTFPL-licensed](https://wtfpl.net) to be considered. See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards, tooling requirements, and workflow conventions. # Rationale ## The problem Given a plain URL, there is no standard way to safely and programmatically download everything "under" that URL path. `wget -r` can traverse directory listings if they're enabled, but every server has a different format, and this does not verify cryptographic integrity of the files, or enable them to be fetched using a different protocol other than HTTP/s. Currently, the solution that people are using are sidecar files in the format of `SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached signature. This is not checksum-algorithm-agnostic and the sidecar file is not always consistently named. Real issues I face: - when I plug in an ExFAT hard drive, I don't know if any files on the filesystem are corrupted or missing - current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files - when I want to mirror an HTTP archive, I have to use special tools like debmirror that understand the archive format - the debian repository metadata structure is hot garbage - when I download a large file via HTTP, I have no way of knowing if the file content is what it's supposed to be ## The solution A standard, a manifest file format, and a tool for generating same. The manifest file would be called `index.mf`, and the tool for generating such would be called `mfer`. The manifest file would do several important things: - have a standard filename, so if given `https://example.com/downloadpackage/` one could fetch `https://example.com/downloadpackage/index.mf` to enumerate the full directory listing. - contain a version field for extensibility - contain structured data (protobuf, json, or cbor) - provide an inner signed container, so that the manifest file itself can embed a signature and a public key alongside in a single file - contain a list of files, each with a relative path to the manifest - contain manifest timestamp - contain ctime/mtime information for files so that file metadata can be preserved - contain cryptographic checksums in several different algorithms for each file - probably encoded with multihash to indicate algo + hash - sha256 at the minimum - would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which likely involves doing an ipfs file object chunking - maybe even including the complete IPFS/IPLD directory tree objects and chunklists? - this is because generating an `index.mf` does not imply publishing on ipfs at that time - maybe a bittorrent chunklist for torrent client compatibility? perhaps a top-level infohash for the whole manifest? # Design The repository is split into a reusable library and a thin command-line wrapper around it. - `mfer/` is the reusable library and the heart of the project: it defines the manifest format and implements building, scanning, checking, serialization, and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it produces (`mfer/mf.pb.go`) is committed alongside it so the library builds with `go get` and needs no `protoc` toolchain. - `internal/cli/` holds the command implementations — `generate` (alias `gen`), `check`, `freshen`, `export`, `list` (alias `ls`), `fetch`, and `version` — that wire the library to the command-line interface. - `internal/log/` provides the logging used by the commands and the library. - `internal/bork/` provides the error the library returns when a manifest's decompressed contents are not the size the manifest records. - `cmd/mfer/` is the entrypoint: its `main` package runs `internal/cli` and exits with the status it returns. Everything under `internal/` is private to this repository; only the `mfer/` package is intended for import by other software. # Design Goals - Replace SHASUMS/SHASUMS.asc files - be easy to download/resume a whole directory tree published via HTTP - be easy to use across protocols (given an HTTPS url, fetch manifest, then download file contents via bittorrent or ipfs) - not strongly coupled to HTTP use case, should not require special hosting, content types, or HTTP headers being sent # Non-Goals - Manifest generation speed - likely involves IPFS chunking, bittorrent chunking, and several different cryptographic hash functions over the entirety of each and every file - Small manifest file size (within reason) - 30MiB files are "small" these days, given modern storage/bandwidth - metadata size should not be used as an excuse to sacrifice utility (such as providing checksums over each chunk of a large file) # Original Design Questions These were the open questions when the project started; open design questions are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues). - Should the manifest file include checksums of individual file chunks, or just for the whole assembled file? If so, should the chunk size be fixed or dynamic? Still open, on [issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698). - Should the manifest signature format be GnuPG signatures, or those from OpenBSD's signify (of which there is a good [golang implementation](https://github.com/frankbraun/gosignify))? Still open, as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82). - Should the on-disk serialization format be proto3 or json? Settled: it is proto3, see `FORMAT.md` and `mfer/mf.proto`. # Tool Examples - `mfer gen` / `mfer gen .` - recurses under current directory and writes out an `index.mf` - `mfer check` / `mfer check .` - verifies checksums of all files in manifest, displaying error and exiting nonzero if any files are missing or corrupted - warns about each file under the base directory that the manifest does not list, hidden files included; with `--no-extra-files` each one is a failure instead - `mfer fetch https://example.com/stuff/` - fetches `/stuff/index.mf` and downloads all files listed in manifest, optionally resuming any that already exist locally, and assures cryptographic integrity of downloaded files. # Implementation Plan ## Phase One: - golang module for reusability/embedding - golang module client providing `mfer` CLI ## Phase Two: - ES6 or TypeScript module for reusability/embedding - ES6/TypeScript module client providing `mfer.js` CLI # Hopes And Dreams - `aria2c https://example.com/manifestdirectory/` - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and checksums all files, resumes any that exist locally already) - `mfer fetch https://example.com/manifestdirectory/` - a command line option to zero/omit mtime/ctime, as well as manifest timestamp, and sort all directory listings so that manifest file generation is deterministic/reproducible - URL format `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2` to assert in the URL which PGP signing key should be used in the manifest, so that shared URLs have a cryptographic trust root - a "well-known" key in the manifest that maps well known keys (could reuse the http spec) to specific file paths in the manifest. - example: a `berlin.sneak.app.slideshow` key that maps to a json slideshow config listing what image paths to show, and for how long, and in what order # Use Cases ## Web Images I'd like to be able to put a bunch of images into a directory, generate a manifest, and then point a slideshow client (such as an ambient display, or a react app with the target directory in a query string arg) at that statically hosted directory, and have it discover the full list of images available at that URL. ## Software Distribution I'd like to be able to download a whole tree of files available via HTTP resumably by either HTTP or IPFS/BitTorrent without a .torrent file. ## Filesystem Archive Integrity I use filesystems that don't include data checksums, and I would like a cryptographically signed checksum file so that I can later verify that a set of archive files have not been modified, none are missing, and that the checksums have not been altered in storage by a second party. ## Filesystem-Independent Checksums I would like to be able to plug in a hard drive or flash drive and, if there is an `index.mf` in the root, automatically detect missing/corrupted files, regardless of filesystem format. # Collaboration Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired username for an account on this Gitea instance. # TODO Open work, open design questions included, is tracked in this repo's issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues). # See Also ## Prior Art: Metalink - [Metalink - Mozilla Wiki](https://wiki.mozilla.org/Metalink) - [Metalink - Wikipedia](https://en.wikipedia.org/wiki/Metalink) - [RFC 5854 - The Metalink Download Description Format](https://datatracker.ietf.org/doc/html/rfc5854) - [RFC 6249 - Metalink/HTTP: Mirrors and Hashes](https://www.rfc-editor.org/rfc/rfc6249.html) ## Links - Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) - Issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues) # Author - [@sneak](https://sneak.berlin) # License - [WTFPL](https://wtfpl.net)