sneak
74185ae5d6
check / check (push) Failing after 2s
The test and format scripts regenerated mfer/mf.pb.go whenever mfer/mf.proto looked newer by mtime, which a fresh checkout often causes, so make check could rewrite a committed file and needed protoc. Nothing regenerates it any more except make generate (script/generate), which also records the hash of mf.proto in mfer/mf.proto.sha256. A Go test compares that hash with mf.proto and fails, naming make generate, when they differ; it needs no protoc. The Makefile's mtime rule for mf.pb.go, make clean's deletion of it and the Dockerfile's touch workarounds are removed. Model: opus-5-5
343 lines
15 KiB
Markdown
343 lines
15 KiB
Markdown
# mfer
|
|
|
|
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
|
|
(public domain) [Go](https://golang.org) library and command-line tool by
|
|
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
|
|
over a directory tree to encapsulate metadata about the files — such as
|
|
cryptographic checksums and signatures over same — to aid in archiving,
|
|
downloading, streaming, and mirroring. It was first published in 2022. The
|
|
manifest files' data is serialized with Google's
|
|
[protobuf serialization format](https://developers.google.com/protocol-buffers).
|
|
The structure of these files can be found
|
|
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
|
|
which is included in the [project repository](https://git.eeqj.de/sneak/mfer).
|
|
|
|
The current version is pre-1.0 and while the repo was published in 2022, there
|
|
has not yet been any versioned release. [SemVer](https://semver.org) will be
|
|
used for releases.
|
|
|
|
This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
|
|
2022 and is currently a one-person effort, though the goal is for this to emerge
|
|
as a de-facto standard and be incorporated into other software. A compatible
|
|
javascript library is planned.
|
|
|
|
# Getting Started
|
|
|
|
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
|
|
is committed, so no `protoc` toolchain is required:
|
|
|
|
```sh
|
|
git clone https://git.eeqj.de/sneak/mfer.git
|
|
cd mfer
|
|
go build -o bin/mfer ./cmd/mfer
|
|
```
|
|
|
|
Generate a manifest for a directory tree, verify it later, and fetch a published
|
|
tree by URL:
|
|
|
|
```sh
|
|
# Write .index.mf, a manifest of the files under the current directory.
|
|
bin/mfer gen .
|
|
|
|
# Verify the files on disk against the manifest. Exits nonzero if any file
|
|
# is missing or corrupted.
|
|
bin/mfer check .index.mf
|
|
|
|
# Download and cryptographically verify a tree published over HTTP: mfer
|
|
# fetches <url>/index.mf, then downloads every file it lists.
|
|
bin/mfer fetch https://example.com/tree/
|
|
```
|
|
|
|
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
|
|
for a single command's options.
|
|
|
|
# Build Status
|
|
|
|
CI runs `script/cibuild`, which builds the Docker image with `--no-cache`, so
|
|
the formatting, lint and test steps in the `Dockerfile` run on every build. The
|
|
`main` branch must always be green.
|
|
|
|
# Entrypoints
|
|
|
|
This repository adheres to the
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
standard: normalized scripts in `script/` are the entrypoints for the
|
|
development workflow, and the Makefile targets are thin shims that call them. We
|
|
provide:
|
|
|
|
- `script/bootstrap` — install all dependencies (Go, Go module download, and
|
|
node/yarn plus the prettier version pinned in `package.json`/`yarn.lock`),
|
|
idempotently; golangci-lint is not installed, it runs only in Docker
|
|
- `script/setup` — make a fresh clone ready for development: runs
|
|
`script/bootstrap`, then `script/install-precommit`
|
|
- `script/projectname` — output the project name (`mfer`); used by other scripts
|
|
such as `script/docker`
|
|
- `script/test` — run the test suite (`go test`); one test fails when
|
|
`mfer/mf.proto` no longer matches the hash `script/generate` recorded
|
|
- `script/generate` — regenerate `mfer/mf.pb.go` from `mfer/mf.proto` and record
|
|
the hash of that `mfer/mf.proto` in `mfer/mf.proto.sha256`; needs `protoc` and
|
|
`protoc-gen-go`, and is the only thing that regenerates the committed
|
|
`mfer/mf.pb.go`
|
|
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand
|
|
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
|
|
as ordinary tests
|
|
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
|
|
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
|
|
every time, then removes the image
|
|
- `script/fmt` — format all code and docs (writes): `gofumpt` and
|
|
`script/prettier --write`
|
|
- `script/prettier` — run prettier over the repository's canonical file set
|
|
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
|
|
`--check`; the single definition of that file set, so `script/fmt` and
|
|
`script/fmt-check` cannot disagree about it
|
|
- `script/fmt-check` — check formatting without writing: `script/fmt-check-go`
|
|
plus `script/prettier --check`
|
|
- `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the
|
|
Docker lint stage, whose image has no node
|
|
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
|
|
- `script/docker` — build the Docker image tagged with the project name
|
|
- `script/cibuild` — CI entrypoint: builds the image with the same command as
|
|
`script/docker`, uncached, so the checks in the Dockerfile run every time
|
|
- `script/precommit` — pre-commit checks: `go mod tidy` verification, then
|
|
`script/check`
|
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
|
`script/precommit`
|
|
|
|
# Participation
|
|
|
|
The community is as yet nonexistent so there are no defined policies or norms
|
|
yet. Primary development happens on a privately-run Gitea instance at
|
|
[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues are
|
|
[tracked there](https://git.eeqj.de/sneak/mfer/issues).
|
|
|
|
Changes must always be formatted with a standard `go fmt`, syntactically valid,
|
|
and must pass the linting defined in the repository's `.golangci.yml`, which
|
|
`make lint` runs in Docker. The `main` branch is protected and all changes must
|
|
be made via [pull requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to
|
|
be merged. Any changes submitted to this project must also be
|
|
[WTFPL-licensed](https://wtfpl.net) to be considered.
|
|
|
|
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
|
|
tooling requirements, and workflow conventions.
|
|
|
|
# Rationale
|
|
|
|
## The problem
|
|
|
|
Given a plain URL, there is no standard way to safely and programmatically
|
|
download everything "under" that URL path. `wget -r` can traverse directory
|
|
listings if they're enabled, but every server has a different format, and this
|
|
does not verify cryptographic integrity of the files, or enable them to be
|
|
fetched using a different protocol other than HTTP/s.
|
|
|
|
Currently, the solution that people are using are sidecar files in the format of
|
|
`SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached signature.
|
|
This is not checksum-algorithm-agnostic and the sidecar file is not always
|
|
consistently named.
|
|
|
|
Real issues I face:
|
|
|
|
- when I plug in an ExFAT hard drive, I don't know if any files on the
|
|
filesystem are corrupted or missing
|
|
- current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files
|
|
- when I want to mirror an HTTP archive, I have to use special tools like
|
|
debmirror that understand the archive format
|
|
- the debian repository metadata structure is hot garbage
|
|
- when I download a large file via HTTP, I have no way of knowing if the file
|
|
content is what it's supposed to be
|
|
|
|
## The solution
|
|
|
|
A standard, a manifest file format, and a tool for generating same.
|
|
|
|
The manifest file would be called `index.mf`, and the tool for generating such
|
|
would be called `mfer`.
|
|
|
|
The manifest file would do several important things:
|
|
|
|
- have a standard filename, so if given `https://example.com/downloadpackage/`
|
|
one could fetch `https://example.com/downloadpackage/index.mf` to enumerate
|
|
the full directory listing.
|
|
- contain a version field for extensibility
|
|
- contain structured data (protobuf, json, or cbor)
|
|
- provide an inner signed container, so that the manifest file itself can embed
|
|
a signature and a public key alongside in a single file
|
|
- contain a list of files, each with a relative path to the manifest
|
|
- contain manifest timestamp
|
|
- contain ctime/mtime information for files so that file metadata can be
|
|
preserved
|
|
- contain cryptographic checksums in several different algorithms for each file
|
|
- probably encoded with multihash to indicate algo + hash
|
|
- sha256 at the minimum
|
|
- would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which
|
|
likely involves doing an ipfs file object chunking
|
|
- maybe even including the complete IPFS/IPLD directory tree objects and
|
|
chunklists?
|
|
- this is because generating an `index.mf` does not imply publishing on
|
|
ipfs at that time
|
|
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
|
|
top-level infohash for the whole manifest?
|
|
|
|
# Design
|
|
|
|
The repository is split into a reusable library and a thin command-line wrapper
|
|
around it.
|
|
|
|
- `mfer/` is the reusable library and the heart of the project: it defines the
|
|
manifest format and implements building, scanning, checking, serialization,
|
|
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
|
|
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
|
|
with `go get` and needs no `protoc` toolchain.
|
|
- `internal/cli/` holds the command implementations — `generate` (alias `gen`),
|
|
`check`, `freshen`, `export`, `list` (alias `ls`), `fetch`, and `version` —
|
|
that wire the library to the command-line interface.
|
|
- `internal/log/` provides the logging used by the commands and the library.
|
|
- `internal/bork/` provides the error the library returns when a manifest's
|
|
decompressed contents are not the size the manifest records.
|
|
- `cmd/mfer/` is the entrypoint: its `main` package runs `internal/cli` and
|
|
exits with the status it returns.
|
|
|
|
Everything under `internal/` is private to this repository; only the `mfer/`
|
|
package is intended for import by other software.
|
|
|
|
# Design Goals
|
|
|
|
- Replace SHASUMS/SHASUMS.asc files
|
|
- be easy to download/resume a whole directory tree published via HTTP
|
|
- be easy to use across protocols (given an HTTPS url, fetch manifest, then
|
|
download file contents via bittorrent or ipfs)
|
|
- not strongly coupled to HTTP use case, should not require special hosting,
|
|
content types, or HTTP headers being sent
|
|
|
|
# Non-Goals
|
|
|
|
- Manifest generation speed
|
|
- likely involves IPFS chunking, bittorrent chunking, and several different
|
|
cryptographic hash functions over the entirety of each and every file
|
|
- Small manifest file size (within reason)
|
|
- 30MiB files are "small" these days, given modern storage/bandwidth
|
|
- metadata size should not be used as an excuse to sacrifice utility (such
|
|
as providing checksums over each chunk of a large file)
|
|
|
|
# Original Design Questions
|
|
|
|
These were the open questions when the project started; open design questions
|
|
are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
|
|
|
|
- Should the manifest file include checksums of individual file chunks, or just
|
|
for the whole assembled file? If so, should the chunk size be fixed or
|
|
dynamic? Still open, on
|
|
[issue 81](https://git.eeqj.de/sneak/mfer/issues/81#issuecomment-118698).
|
|
|
|
- Should the manifest signature format be GnuPG signatures, or those from
|
|
OpenBSD's signify (of which there is a good
|
|
[golang implementation](https://github.com/frankbraun/gosignify))? Still open,
|
|
as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).
|
|
|
|
- Should the on-disk serialization format be proto3 or json? Settled: it is
|
|
proto3, see `FORMAT.md` and `mfer/mf.proto`.
|
|
|
|
# Tool Examples
|
|
|
|
- `mfer gen` / `mfer gen .`
|
|
- recurses under current directory and writes out an `index.mf`
|
|
- `mfer check` / `mfer check .`
|
|
- verifies checksums of all files in manifest, displaying error and exiting
|
|
nonzero if any files are missing or corrupted
|
|
- `mfer fetch https://example.com/stuff/`
|
|
- fetches `/stuff/index.mf` and downloads all files listed in manifest,
|
|
optionally resuming any that already exist locally, and assures
|
|
cryptographic integrity of downloaded files.
|
|
|
|
# Implementation Plan
|
|
|
|
## Phase One:
|
|
|
|
- golang module for reusability/embedding
|
|
- golang module client providing `mfer` CLI
|
|
|
|
## Phase Two:
|
|
|
|
- ES6 or TypeScript module for reusability/embedding
|
|
- ES6/TypeScript module client providing `mfer.js` CLI
|
|
|
|
# Hopes And Dreams
|
|
|
|
- `aria2c https://example.com/manifestdirectory/`
|
|
- (fetches `https://example.com/manifestdirectory/index.mf`, downloads and
|
|
checksums all files, resumes any that exist locally already)
|
|
- `mfer fetch https://example.com/manifestdirectory/`
|
|
- a command line option to zero/omit mtime/ctime, as well as manifest timestamp,
|
|
and sort all directory listings so that manifest file generation is
|
|
deterministic/reproducible
|
|
- URL format
|
|
`mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2`
|
|
to assert in the URL which PGP signing key should be used in the manifest, so
|
|
that shared URLs have a cryptographic trust root
|
|
- a "well-known" key in the manifest that maps well known keys (could reuse the
|
|
http spec) to specific file paths in the manifest.
|
|
- example: a `berlin.sneak.app.slideshow` key that maps to a json slideshow
|
|
config listing what image paths to show, and for how long, and in what
|
|
order
|
|
|
|
# Use Cases
|
|
|
|
## Web Images
|
|
|
|
I'd like to be able to put a bunch of images into a directory, generate a
|
|
manifest, and then point a slideshow client (such as an ambient display, or a
|
|
react app with the target directory in a query string arg) at that statically
|
|
hosted directory, and have it discover the full list of images available at that
|
|
URL.
|
|
|
|
## Software Distribution
|
|
|
|
I'd like to be able to download a whole tree of files available via HTTP
|
|
resumably by either HTTP or IPFS/BitTorrent without a .torrent file.
|
|
|
|
## Filesystem Archive Integrity
|
|
|
|
I use filesystems that don't include data checksums, and I would like a
|
|
cryptographically signed checksum file so that I can later verify that a set of
|
|
archive files have not been modified, none are missing, and that the checksums
|
|
have not been altered in storage by a second party.
|
|
|
|
## Filesystem-Independent Checksums
|
|
|
|
I would like to be able to plug in a hard drive or flash drive and, if there is
|
|
an `index.mf` in the root, automatically detect missing/corrupted files,
|
|
regardless of filesystem format.
|
|
|
|
# Collaboration
|
|
|
|
Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired
|
|
username for an account on this Gitea instance.
|
|
|
|
# TODO
|
|
|
|
Open work, open design questions included, is tracked in this repo's issues:
|
|
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues).
|
|
|
|
# See Also
|
|
|
|
## Prior Art: Metalink
|
|
|
|
- [Metalink - Mozilla Wiki](https://wiki.mozilla.org/Metalink)
|
|
- [Metalink - Wikipedia](https://en.wikipedia.org/wiki/Metalink)
|
|
- [RFC 5854 - The Metalink Download Description Format](https://datatracker.ietf.org/doc/html/rfc5854)
|
|
- [RFC 6249 - Metalink/HTTP: Mirrors and Hashes](https://www.rfc-editor.org/rfc/rfc6249.html)
|
|
|
|
## Links
|
|
|
|
- Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
|
|
- Issues:
|
|
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
|
|
|
|
# Author
|
|
|
|
- [@sneak](https://sneak.berlin)
|
|
|
|
# License
|
|
|
|
- [WTFPL](https://wtfpl.net)
|