mfer/README.md

mfer

mfer is a WTFPL-licensed (public domain) Go library and command-line tool by @sneak that specifies and generates .mf manifest files over a directory tree to encapsulate metadata about the files — such as cryptographic checksums and signatures over same — to aid in archiving, downloading, streaming, and mirroring. It was first published in 2022. The manifest files' data is serialized with Google's protobuf serialization format. The structure of these files can be found in the format specification which is included in the project repository.

The current version is pre-1.0 and while the repo was published in 2022, there has not yet been any versioned release. SemVer will be used for releases.

This project was started by @sneak to scratch an itch in 2022 and is currently a one-person effort, though the goal is for this to emerge as a de-facto standard and be incorporated into other software. A compatible javascript library is planned.

Getting Started

mfer builds from source with a Go 1.23+ toolchain. The generated protobuf code is committed, so no protoc toolchain is required:

git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
go build -o bin/mfer ./cmd/mfer

Generate a manifest for a directory tree, verify it later, and fetch a published tree by URL:

# Write index.mf, a manifest of the files under the current directory.
bin/mfer gen .

# Verify the files on disk against the manifest. Exits nonzero if any file
# is missing or corrupted.
bin/mfer check index.mf

# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/

Run bin/mfer help for the full command list, or bin/mfer <command> --help for a single command's options.

Build Status

CI runs script/cibuild, which builds the Docker image with --no-cache, so the formatting, lint and test steps in the Dockerfile run on every build. The main branch must always be green.

Entrypoints

This repository adheres to the Scripts to Rule Them All standard: normalized scripts in script/ are the entrypoints for the development workflow, and the Makefile targets are thin shims that call them. We provide:

• script/bootstrap — install all dependencies (Go, Go module download, and node/yarn plus the prettier version pinned in package.json/yarn.lock), idempotently; golangci-lint is not installed, it runs only in Docker
• script/setup — make a fresh clone ready for development: runs script/bootstrap, then script/install-precommit
• script/projectname — output the project name (mfer); used by other scripts such as script/docker
• script/test — run the test suite (go test); one test fails when mfer/mf.proto no longer matches the hash script/generate recorded
• script/generate (make generate) — regenerate mfer/mf.pb.go from mfer/mf.proto and record the hash of that mfer/mf.proto in mfer/mf.proto.sha256; the only thing that regenerates the committed mfer/mf.pb.go. It needs the exact versions that wrote the committed file, and refuses to run with any other: protoc 33.4 (unpack protoc-33.4-<platform>.zip from its release and put its bin/protoc on PATH) and protoc-gen-go v1.36.11 (go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11, which installs it in $(go env GOPATH)/bin; make generate adds that directory to PATH)
• script/fuzz — fuzz the manifest parser for one minute; run by hand (make fuzz), never by CI, while script/test runs its committed seed corpus as ordinary tests
• script/lint — run golangci-lint in Docker: builds only the lint stage of the Dockerfile (the Go format check, then the linter), uncached so it runs every time, then removes the image
• script/fmt — format all code and docs (writes): gofumpt and script/prettier --write
• script/prettier — run prettier over the repository's canonical file set (Markdown and JSON, minus .prettierignore) in the given mode, --write or --check; the single definition of that file set, so script/fmt and script/fmt-check cannot disagree about it
• script/fmt-check — check formatting without writing: script/fmt-check-go plus script/prettier --check
• script/fmt-check-go — the Go half of script/fmt-check, on its own, for the Docker lint stage, whose image has no node
• script/check — run script/test, script/lint, and script/fmt-check
• script/docker — build the Docker image tagged with the project name
• script/cibuild — CI entrypoint: builds the image with the same command as script/docker, uncached, so the checks in the Dockerfile run every time
• script/precommit — pre-commit checks: go mod tidy verification, then script/check
• script/install-precommit — install the git pre-commit hook that runs script/precommit

Participation

The community is as yet nonexistent so there are no defined policies or norms yet. Primary development happens on a privately-run Gitea instance at https://git.eeqj.de/sneak/mfer and issues are tracked there.

Changes must always be formatted with a standard go fmt, syntactically valid, and must pass the linting defined in the repository's .golangci.yml, which make lint runs in Docker. The main branch is protected and all changes must be made via pull requests and pass CI to be merged. Any changes submitted to this project must also be WTFPL-licensed to be considered.

See REPO_POLICIES.md for detailed coding standards, tooling requirements, and workflow conventions.

Rationale

The problem

Given a plain URL, there is no standard way to safely and programmatically download everything "under" that URL path. wget -r can traverse directory listings if they're enabled, but every server has a different format, and this does not verify cryptographic integrity of the files, or enable them to be fetched using a different protocol other than HTTP/s.

Currently, the solution that people are using are sidecar files in the format of SHASUMS checksum files, as well as a SHASUMS.asc PGP detached signature. This is not checksum-algorithm-agnostic and the sidecar file is not always consistently named.

Real issues I face:

• when I plug in an ExFAT hard drive, I don't know if any files on the filesystem are corrupted or missing
  • current ad-hoc solution are SHASUMS/SHASUMS.asc files
• when I want to mirror an HTTP archive, I have to use special tools like debmirror that understand the archive format
  • the debian repository metadata structure is hot garbage
• when I download a large file via HTTP, I have no way of knowing if the file content is what it's supposed to be

The solution

A standard, a manifest file format, and a tool for generating same.

The manifest file would be called index.mf, and the tool for generating such would be called mfer.

The manifest file would do several important things:

• have a standard filename, so if given https://example.com/downloadpackage/ one could fetch https://example.com/downloadpackage/index.mf to enumerate the full directory listing.
• contain a version field for extensibility
• contain structured data (protobuf, json, or cbor)
• provide an inner signed container, so that the manifest file itself can embed a signature and a public key alongside in a single file
• contain a list of files, each with a relative path to the manifest
• contain manifest timestamp
• contain ctime/mtime information for files so that file metadata can be preserved
• contain cryptographic checksums in several different algorithms for each file
  • probably encoded with multihash to indicate algo + hash
  • sha256 at the minimum
  • would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which likely involves doing an ipfs file object chunking
  • maybe even including the complete IPFS/IPLD directory tree objects and chunklists?
    • this is because generating an index.mf does not imply publishing on ipfs at that time
  • maybe a bittorrent chunklist for torrent client compatibility? perhaps a top-level infohash for the whole manifest?

Design

The repository is split into a reusable library and a thin command-line wrapper around it.

• mfer/ is the reusable library and the heart of the project: it defines the manifest format and implements building, scanning, checking, serialization, and signing. The protobuf schema is mfer/mf.proto, and the generated code it produces (mfer/mf.pb.go) is committed alongside it so the library builds with go get and needs no protoc toolchain.
• internal/cli/ holds the command implementations — generate (alias gen), check, freshen, export, list (alias ls), fetch, and version — that wire the library to the command-line interface.
• internal/log/ provides the logging used by the commands and the library.
• internal/bork/ provides the error the library returns when a manifest's decompressed contents are not the size the manifest records.
• cmd/mfer/ is the entrypoint: its main package runs internal/cli and exits with the status it returns.

Everything under internal/ is private to this repository; only the mfer/ package is intended for import by other software.

Design Goals

• Replace SHASUMS/SHASUMS.asc files
• be easy to download/resume a whole directory tree published via HTTP
• be easy to use across protocols (given an HTTPS url, fetch manifest, then download file contents via bittorrent or ipfs)
• not strongly coupled to HTTP use case, should not require special hosting, content types, or HTTP headers being sent

Non-Goals

• Manifest generation speed
  • likely involves IPFS chunking, bittorrent chunking, and several different cryptographic hash functions over the entirety of each and every file
• Small manifest file size (within reason)
  • 30MiB files are "small" these days, given modern storage/bandwidth
  • metadata size should not be used as an excuse to sacrifice utility (such as providing checksums over each chunk of a large file)

Original Design Questions

These were the open questions when the project started; open design questions are now tracked only in the issues.

• Should the manifest file include checksums of individual file chunks, or just for the whole assembled file? If so, should the chunk size be fixed or dynamic? Still open, on issue 81.

• Should the manifest signature format be GnuPG signatures, or those from OpenBSD's signify (of which there is a good golang implementation)? Still open, as question 10 on issue 82.

• Should the on-disk serialization format be proto3 or json? Settled: it is proto3, see docs/FORMAT.md and mfer/mf.proto.

Tool Examples

• mfer gen / mfer gen .
  • recurses under current directory and writes out an index.mf
• mfer check / mfer check .
  • verifies checksums of all files in manifest, displaying error and exiting nonzero if any files are missing or corrupted
• mfer fetch https://example.com/stuff/
  • fetches /stuff/index.mf and downloads all files listed in manifest, optionally resuming any that already exist locally, and assures cryptographic integrity of downloaded files.

Implementation Plan

Phase One:

• golang module for reusability/embedding
• golang module client providing mfer CLI

Phase Two:

• ES6 or TypeScript module for reusability/embedding
• ES6/TypeScript module client providing mfer.js CLI

Hopes And Dreams

• aria2c https://example.com/manifestdirectory/
  • (fetches https://example.com/manifestdirectory/index.mf, downloads and checksums all files, resumes any that exist locally already)
• mfer fetch https://example.com/manifestdirectory/
• a command line option to zero/omit mtime/ctime, as well as manifest timestamp, and sort all directory listings so that manifest file generation is deterministic/reproducible
• URL format mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2 to assert in the URL which PGP signing key should be used in the manifest, so that shared URLs have a cryptographic trust root
• a "well-known" key in the manifest that maps well known keys (could reuse the http spec) to specific file paths in the manifest.
  • example: a berlin.sneak.app.slideshow key that maps to a json slideshow config listing what image paths to show, and for how long, and in what order

Use Cases

Web Images

I'd like to be able to put a bunch of images into a directory, generate a manifest, and then point a slideshow client (such as an ambient display, or a react app with the target directory in a query string arg) at that statically hosted directory, and have it discover the full list of images available at that URL.

Software Distribution

I'd like to be able to download a whole tree of files available via HTTP resumably by either HTTP or IPFS/BitTorrent without a .torrent file.

Filesystem Archive Integrity

I use filesystems that don't include data checksums, and I would like a cryptographically signed checksum file so that I can later verify that a set of archive files have not been modified, none are missing, and that the checksums have not been altered in storage by a second party.

Filesystem-Independent Checksums

I would like to be able to plug in a hard drive or flash drive and, if there is an index.mf in the root, automatically detect missing/corrupted files, regardless of filesystem format.

Collaboration

Please email sneak@sneak.berlin with your desired username for an account on this Gitea instance.

TODO

Open work, open design questions included, is tracked in this repo's issues: https://git.eeqj.de/sneak/mfer/issues.

See Also

Prior Art: Metalink

• Metalink - Mozilla Wiki
• Metalink - Wikipedia
• RFC 5854 - The Metalink Download Description Format
• RFC 6249 - Metalink/HTTP: Mirrors and Hashes

Links

• Repo: https://git.eeqj.de/sneak/mfer
• Issues: https://git.eeqj.de/sneak/mfer/issues

Author

• @sneak

License

• WTFPL