mfer/README.md

# mfer

[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
(public domain) [Go](https://golang.org) library and command-line tool by
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
over a directory tree to encapsulate metadata about the files — such as
cryptographic checksums and signatures over same — to aid in archiving,
downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](docs/FORMAT.md), which refers to the protobuf
schema `mfer/mf.proto` for exact field numbers and types. Both are included in
the [project repository](https://git.eeqj.de/sneak/mfer).

The current version is pre-1.0 and while the repo was published in 2022, there
has not yet been any versioned release. [SemVer](https://semver.org) will be
used for releases.

This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
2022 and is currently a one-person effort, though the goal is for this to emerge
as a de-facto standard and be incorporated into other software. A compatible
javascript library is planned.

# Getting Started

`mfer` builds from source with Go 1.27.1 or later. The generated protobuf code
is committed, so no `protoc` toolchain is required:

```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
make build
```

Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:

```sh
# Write index.mf, a manifest of the files under the current directory.
bin/mfer gen .

# Verify the files on disk against the manifest. Exits nonzero if any file
# it lists is missing or corrupted; warns about files it does not list.
bin/mfer check index.mf

# Download and cryptographically verify a tree published over HTTP into
# ./mirror: mfer fetches <url>/index.mf, downloads every file it lists,
# skipping any already there with the right hash, then saves the manifest as
# mirror/index.mf.
bin/mfer fetch --dest mirror https://example.com/tree/
```

Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
for a single command's options.

# Build Status

CI runs `script/cibuild`, which runs `script/bootstrap` and `script/check`, then
builds the Docker image with `--no-cache`, so the lint and test phases in the
`Dockerfile` run on every build. The `main` branch must always be green.

# Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. We
provide:

- `script/bootstrap` — install all dependencies, idempotently: Go and the
  modules of `go.mod`; node (the version `.nvmrc` names, through nvm when there
  is no node on `PATH`) and yarn, plus the prettier version pinned in
  `package.json`/`yarn.lock`; `gofumpt` v0.12.0 and `protoc-gen-go` v1.36.12,
  installed into `bin/` with `go install`, each pinned to a commit; and `protoc`
  33.4, unpacked into `bin/protoc` from its release archive once the archive
  matches the sha256 the script holds for this platform. Each of those three is
  installed again whenever the one in `bin/` reports another version.
  golangci-lint is not installed, it runs only in Docker
- `script/setup` — make a fresh clone ready for development: runs
  `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (`mfer`); used by other scripts
  such as `script/docker`
- `script/test` — run the test suite in Docker: builds only the `test` stage of
  the `Dockerfile`, whose build runs `go test -race`, uncached so it runs every
  time; one test fails when `mfer/mf.proto` no longer matches the hash
  `script/generate` recorded
- `script/build` (`make build`) — build the `mfer` binary into `bin/mfer`,
  stamped with the revision `mfer version` prints: the output of
  `git describe --tags --always --dirty`, as `script/docker` passes it
- `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from
  `mfer/mf.proto` and record the hash of that `mfer/mf.proto` in
  `mfer/mf.proto.sha256`; the only thing that regenerates the committed
  `mfer/mf.pb.go`. It runs the `protoc` that `script/bootstrap` unpacks into
  `bin/protoc` and the `protoc-gen-go` it installs into `bin/`, refusing any
  version of either but the pinned one
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand
  (`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
  as ordinary tests
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
  the `Dockerfile`, whose build runs the linter, uncached so it runs every time
- `script/vulncheck` (`make vulncheck`) — run `govulncheck` in Docker: builds
  only the `vulncheck` stage of the `Dockerfile`, uncached, which reports known
  vulnerabilities in the code `mfer` calls, from the Go vulnerability database.
  `script/check` does not run it, so an advisory published later never turns the
  gate red
- `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and
  `script/prettier --write`
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
  given mode, `--write` or `--check`, with the `bin/gofumpt` that
  `script/bootstrap` installs, refusing any version but the pinned one;
  `script/fmt` and `script/fmt-check` both go through it, so they cannot
  disagree about Go formatting
- `script/prettier` — run prettier over the repository's canonical file set
  (Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
  `--check`, with the prettier version `yarn.lock` pins, run by the node on
  `PATH` or else the one `script/bootstrap` installed through nvm; the single
  definition of that file set, so `script/fmt` and `script/fmt-check` cannot
  disagree about it
- `script/fmt-check` — check formatting without writing:
  `script/gofumpt --check` plus `script/prettier --check`
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/docker` — build the Docker image tagged with the project name
- `script/cibuild` — CI entrypoint: runs `script/bootstrap` and `script/check`,
  then builds the image with the same command as `script/docker`, uncached, so
  the lint and test phases in the `Dockerfile` run every time
- `script/precommit` — pre-commit checks: `go mod tidy` verification, then
  `script/check`
- `script/install-precommit` — install the git pre-commit hook that runs
  `script/precommit`

# Participation

The community is as yet nonexistent so there are no defined policies or norms
yet. Primary development happens on a privately-run Gitea instance at
[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues are
[tracked there](https://git.eeqj.de/sneak/mfer/issues).

Changes must always be formatted with a standard `go fmt`, syntactically valid,
and must pass the linting defined in the repository's `.golangci.yml`, which
`make lint` runs in Docker. The `main` branch is protected and all changes must
be made via [pull requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to
be merged. Any changes submitted to this project must also be
[WTFPL-licensed](https://wtfpl.net) to be considered.

See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions.

# Rationale

## The problem

Given a plain URL, there is no standard way to safely and programmatically
download everything "under" that URL path. `wget -r` can traverse directory
listings if they're enabled, but every server has a different format, and this
does not verify cryptographic integrity of the files, or enable them to be
fetched using a different protocol other than HTTP/s.

Currently, the solution that people are using are sidecar files in the format of
`SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached signature.
This is not checksum-algorithm-agnostic and the sidecar file is not always
consistently named.

Real issues I face:

- when I plug in an ExFAT hard drive, I don't know if any files on the
  filesystem are corrupted or missing
    - current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files
- when I want to mirror an HTTP archive, I have to use special tools like
  debmirror that understand the archive format
    - the debian repository metadata structure is hot garbage
- when I download a large file via HTTP, I have no way of knowing if the file
  content is what it's supposed to be

## The solution

A standard, a manifest file format, and a tool for generating same.

The manifest file would be called `index.mf`, and the tool for generating such
would be called `mfer`.

The manifest file would do several important things:

- have a standard filename, so if given `https://example.com/downloadpackage/`
  one could fetch `https://example.com/downloadpackage/index.mf` to enumerate
  the full directory listing.
- contain a version field for extensibility
- contain structured data (protobuf, json, or cbor)
- provide an inner signed container, so that the manifest file itself can embed
  a signature and a public key alongside in a single file
- contain a list of files, each with a relative path to the manifest
- contain manifest timestamp
- contain ctime/mtime information for files so that file metadata can be
  preserved
- contain cryptographic checksums in several different algorithms for each file
    - probably encoded with multihash to indicate algo + hash
    - sha256 at the minimum
    - would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which
      likely involves doing an ipfs file object chunking
    - maybe even including the complete IPFS/IPLD directory tree objects and
      chunklists?
        - this is because generating an `index.mf` does not imply publishing on
          ipfs at that time
    - maybe a bittorrent chunklist for torrent client compatibility? perhaps a
      top-level infohash for the whole manifest?

# Design

The repository is split into a reusable library and a thin command-line wrapper
around it.

- `mfer/` is the reusable library and the heart of the project: it defines the
  manifest format and implements building, scanning, checking, serialization,
  and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
  produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
  with `go get` and needs no `protoc` toolchain.
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
  `check`, `freshen`, `export`, `list` (alias `ls`), `fetch`, and `version` —
  that wire the library to the command-line interface.
- `internal/log/` provides the logging used by the commands and the library.
- `internal/bork/` provides the error the library returns when a manifest's
  decompressed contents are not the size the manifest records.
- `cmd/mfer/` is the entrypoint: its `main` package runs `internal/cli` and
  exits with the status it returns.

Everything under `internal/` is private to this repository; only the `mfer/`
package is intended for import by other software.

# Design Goals

- Replace SHASUMS/SHASUMS.asc files
- be easy to download/resume a whole directory tree published via HTTP
- be easy to use across protocols (given an HTTPS url, fetch manifest, then
  download file contents via bittorrent or ipfs)
- not strongly coupled to HTTP use case, should not require special hosting,
  content types, or HTTP headers being sent

# Non-Goals

- Manifest generation speed
    - likely involves IPFS chunking, bittorrent chunking, and several different
      cryptographic hash functions over the entirety of each and every file
- Small manifest file size (within reason)
    - 30MiB files are "small" these days, given modern storage/bandwidth
    - metadata size should not be used as an excuse to sacrifice utility (such
      as providing checksums over each chunk of a large file)

# Original Design Questions

These were the open questions when the project started; open design questions
are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).

- Should the manifest file include checksums of individual file chunks, or just
  for the whole assembled file? If so, should the chunk size be fixed or
  dynamic? Still open, on
  [issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698).

- Should the manifest signature format be GnuPG signatures, or those from
  OpenBSD's signify (of which there is a good
  [golang implementation](https://github.com/frankbraun/gosignify))? Still open,
  as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).

- Should the on-disk serialization format be proto3 or json? Settled: it is
  proto3, see `docs/FORMAT.md` and `mfer/mf.proto`.

# Tool Examples

- `mfer gen` / `mfer gen .`
    - recurses under current directory and writes out an `index.mf`
    - records every file's mode as `0000` unless given `--include-permissions`,
      which records each file's permission bits (`0777` at most)
- `mfer gen /media/drive`
    - writes `/media/drive/index.mf`, listing each file by its path under
      `/media/drive`, so `mfer check /media/drive` verifies it. Given a file,
      gen writes `index.mf` beside it and lists the file by its name; given
      several paths, it writes `index.mf` in the current directory
    - `--output` names another file to write instead. What gen lists depends
      only on the paths it is given and the files under them, so with the same
      `--seed` and an unchanged tree it writes the same bytes wherever the
      manifest goes. The file it writes to is never listed
- `mfer check` / `mfer check .`
    - verifies checksums of all files in manifest, displaying error and exiting
      nonzero if any files are missing or corrupted, or have permission bits
      other than the mode the manifest records, unless that is `0000`
    - looks for those files under the base directory: the one `--base` names, or
      else the directory holding the manifest, or the current directory for a
      manifest given by URL. So `mfer check /media/drive` checks a drive against
      the `index.mf` at its root, from any directory
    - warns about each file under the base directory that the manifest does not
      list, hidden files included; with `--no-extra-files` each one is a failure
      instead
- `mfer freshen` / `mfer freshen .`
    - rewrites `index.mf` to list the files now under the directory holding it,
      or under the one `--base` names, hashing only the files that are new or
      changed
    - leaves out hidden files unless given `--include-dotfiles`, and symlinks
      unless given `--follow-symlinks`, which lists each symlink to a file under
      its own name with the contents of the file it points to
- `mfer fetch https://example.com/stuff/`
    - fetches `/stuff/index.mf` and downloads all files listed in manifest into
      the current directory, or the one given with `--dest`, and assures
      cryptographic integrity of downloaded files. A file already there with the
      size, hash and recorded mode the manifest lists is skipped. Once every
      file is in place, the manifest is saved there as `index.mf`, so
      `mfer check` can verify the tree later. Each file is downloaded to a temp
      file beside it, such as `.a.txt.tmp` for `a.txt`, given the mode the
      manifest records unless that is `0000`, then moved into place. A manifest
      is refused before any file is downloaded if it records a mode above
      `0777`, or lists a file where fetch writes another: at another listed file
      or a directory one is in, at the temp file of a listed file, or at
      `index.mf` or `.index.mf.tmp` at the top of the tree. Names are compared
      in any letter case, on every filesystem, since on a case-insensitive one
      `A.txt` and `a.txt` are one file; a directory two listed files are in must
      be spelled alike in both.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
    - as above, but first refuses a manifest not signed by the key with that
      fingerprint, as `mfer check --require-signature` does, before downloading
      any file.

# Implementation Plan

## Phase One:

- golang module for reusability/embedding
- golang module client providing `mfer` CLI

## Phase Two:

- ES6 or TypeScript module for reusability/embedding
- ES6/TypeScript module client providing `mfer.js` CLI

# Hopes And Dreams

- `aria2c https://example.com/manifestdirectory/`
    - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
      checksums all files, resumes any that exist locally already)
- `mfer fetch https://example.com/manifestdirectory/`
- a command line option to zero/omit mtime/ctime, as well as manifest timestamp,
  and sort all directory listings so that manifest file generation is
  deterministic/reproducible
- URL format
  `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`
  to assert in the URL which PGP signing key should be used in the manifest, so
  that shared URLs have a cryptographic trust root
- a "well-known" key in the manifest that maps well known keys (could reuse the
  http spec) to specific file paths in the manifest.
    - example: a `berlin.sneak.app.slideshow` key that maps to a json slideshow
      config listing what image paths to show, and for how long, and in what
      order

# Use Cases

## Web Images

I'd like to be able to put a bunch of images into a directory, generate a
manifest, and then point a slideshow client (such as an ambient display, or a
react app with the target directory in a query string arg) at that statically
hosted directory, and have it discover the full list of images available at that
URL.

## Software Distribution

I'd like to be able to download a whole tree of files available via HTTP
resumably by either HTTP or IPFS/BitTorrent without a .torrent file.

## Filesystem Archive Integrity

I use filesystems that don't include data checksums, and I would like a
cryptographically signed checksum file so that I can later verify that a set of
archive files have not been modified, none are missing, and that the checksums
have not been altered in storage by a second party.

## Filesystem-Independent Checksums

I would like to be able to plug in a hard drive or flash drive and, if there is
an `index.mf` in the root, automatically detect missing/corrupted files,
regardless of filesystem format.

# Collaboration

Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired
username for an account on this Gitea instance.

# TODO

Open work, open design questions included, is tracked in this repo's issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues).

# See Also

## Prior Art: Metalink

- [Metalink - Mozilla Wiki](https://wiki.mozilla.org/Metalink)
- [Metalink - Wikipedia](https://en.wikipedia.org/wiki/Metalink)
- [RFC 5854 - The Metalink Download Description Format](https://datatracker.ietf.org/doc/html/rfc5854)
- [RFC 6249 - Metalink/HTTP: Mirrors and Hashes](https://www.rfc-editor.org/rfc/rfc6249.html)

## Links

- Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
- Issues:
  [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)

# Author

- [@sneak](https://sneak.berlin)

# License

- [WTFPL](https://wtfpl.net)