Author SHA1 Message Date
sneak d90bef8580 Add the required README sections (closes #75)
check / check (push) Failing after 1s
Policy names five README sections that were missing or misnamed.

The Description first line now names the project, purpose, category,
WTFPL license, and author. A new Getting Started section gives a
copy-pasteable build-from-source block and gen/check/fetch usage; every
command was run against a built binary. Problem Statement and Proposed
Solution are consolidated under a new Rationale heading, keeping the
prose rather than duplicating it. A new Design section documents the
package layout: mfer/ library, internal/cli commands, internal/log and
internal/bork support, cmd/mfer entrypoint, and the committed protobuf
code. Authors is renamed Author with the canonical link.

Getting Started uses go build because the make build target invokes
protoc, which script/bootstrap does not install.

Model: opus-4-8
2026-09-21 08:03:30 +00:00
clawbot 6de3f1d714 Add .editorconfig, broaden .gitignore, drop dead Drone refs (closes #72)
check / check (push) Failing after 1s
Add the canonical .editorconfig from sneak/prompts verbatim. Broaden .gitignore to cover secrets (.env, .env.*, *.key, *.pem), OS files, editor files, and Go artifacts while keeping every repo-specific entry; no tracked file becomes ignored, and bin/gitrev.sh stays tracked despite the /bin/ rule. Remove the stale .drone.yml ignore entry and the DRONE_COMMIT_SHA branch from bin/gitrev.sh, left from the Gitea Actions migration; the GITREV and git-describe paths still work.

A reader may trip over .editorconfig saying four spaces for every file while Go files are tab-indented: gofmt governs Go, editorconfig does not, and the file is taken verbatim by policy.

Disclosure: the worker's cold docker build hit the 10s test timeout in the gpg signature test; this change has no Go code, and that timeout is tracked by issues 67 and 62.

Model: opus-4-8 (implementation); fable-5-1 (merge)
2026-09-21 09:56:15 +02:00
7 changed files with 109 additions and 41 deletions
+12
View File
@@ -0,0 +1,12 @@
root = true
[*]
indent_style = space
indent_size = 4
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true
[Makefile]
indent_style = tab
+21 -2
View File
@@ -10,5 +10,24 @@ modcache.tzst
# Generated manifest files
.index.mf
# Stale files
.drone.yml
# Secrets
.env
.env.*
*.key
*.pem
# OS files
.DS_Store
Thumbs.db
# Editor files
*.swp
*.swo
*~
.idea/
.vscode/
# Go build artifacts
*.log
*.out
*.test
-12
View File
@@ -11,10 +11,6 @@ COPY . .
# Touch .pb.go so make does not try to regenerate via protoc (file is committed)
RUN touch mfer/mf.pb.go
# Changing value from script/cibuild; forces the check steps below to
# re-run instead of being served from a stale layer cache.
ARG CHECK_EPOCH
# Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below.
RUN make fmt-check-go
@@ -31,10 +27,6 @@ RUN yarn install --frozen-lockfile
COPY . .
# Changing value from script/cibuild; forces the check step below to
# re-run instead of being served from a stale layer cache.
ARG CHECK_EPOCH
# No make in this image; call the script entrypoint directly.
RUN script/prettier --check
@@ -55,10 +47,6 @@ COPY . .
# Touch .pb.go so make does not try to regenerate via protoc (file is committed)
RUN touch mfer/mf.pb.go
# Changing value from script/cibuild; forces the check steps below to
# re-run instead of being served from a stale layer cache.
ARG CHECK_EPOCH
RUN make test
RUN cd cmd/mfer && go build -tags urfave_cli_no_docs -o /mfer .
+64 -11
View File
@@ -1,12 +1,12 @@
# mfer
[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and
thin wrapper command-line utility written in [Go](https://golang.org) and first
published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license.
It specifies and generates `.mf` manifest files over a directory tree of files
to encapsulate metadata about them (such as cryptographic checksums or
signatures over same) to aid in archiving, downloading, and streaming, or
mirroring. The manifest files' data is serialized with Google's
[mfer](https://git.eeqj.de/sneak/mfer) is a [WTFPL](https://wtfpl.net)-licensed
(public domain) [Go](https://golang.org) library and command-line tool by
[@sneak](https://sneak.berlin) that specifies and generates `.mf` manifest files
over a directory tree to encapsulate metadata about the files — such as
cryptographic checksums and signatures over same — to aid in archiving,
downloading, streaming, and mirroring. It was first published in 2022. The
manifest files' data is serialized with Google's
[protobuf serialization format](https://developers.google.com/protocol-buffers).
The structure of these files can be found
[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto)
@@ -21,6 +21,36 @@ This project was started by [@sneak](https://sneak.berlin) to scratch an itch in
as a de-facto standard and be incorporated into other software. A compatible
javascript library is planned.
# Getting Started
`mfer` builds from source with a Go 1.23+ toolchain. The generated protobuf code
is committed, so no `protoc` toolchain is required:
```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
go build -o bin/mfer ./cmd/mfer
```
Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:
```sh
# Write .index.mf describing every file under the current directory.
bin/mfer gen .
# Verify the files on disk against the manifest. Exits nonzero if any file
# is missing or corrupted.
bin/mfer check .index.mf
# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/
```
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
for a single command's options.
# Build Status
CI runs via `script/cibuild` (`docker build .`), which executes `make check`
@@ -81,7 +111,9 @@ Any changes submitted to this project must also be
See [`REPO_POLICIES.md`](REPO_POLICIES.md) for detailed coding standards,
tooling requirements, and workflow conventions.
# Problem Statement
# Rationale
## The problem
Given a plain URL, there is no standard way to safely and programmatically
download everything "under" that URL path. `wget -r` can traverse directory
@@ -105,7 +137,7 @@ Real issues I face:
- when I download a large file via HTTP, I have no way of knowing if the file
content is what it's supposed to be
# Proposed Solution
## The solution
A standard, a manifest file format, and a tool for generating same.
@@ -137,6 +169,27 @@ The manifest file would do several important things:
- maybe a bittorrent chunklist for torrent client compatibility? perhaps a
top-level infohash for the whole manifest?
# Design
The repository is split into a reusable library and a thin command-line wrapper
around it.
- `mfer/` is the reusable library and the heart of the project: it defines the
manifest format and implements building, scanning, checking, serialization,
and signing. The protobuf schema is `mfer/mf.proto`, and the generated code it
produces (`mfer/mf.pb.go`) is committed alongside it so the library builds
with `go get` and needs no `protoc` toolchain.
- `internal/cli/` holds the command implementations — `gen`, `check`, `freshen`,
`export`, `list`, and `fetch` — that wire the library to the command-line
interface.
- `internal/log/` provides the logging used across the commands.
- `internal/bork/` provides error-handling support.
- `cmd/mfer/` is the entrypoint: its `main` package assembles the pieces above
into the `mfer` binary.
Everything under `internal/` is private to this repository; only the `mfer/`
package is intended for import by other software.
# Design Goals
- Replace SHASUMS/SHASUMS.asc files
@@ -462,9 +515,9 @@ proto `go_package` option. Which is canonical?
- Issues:
[https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues)
# Authors
# Author
- [@sneak &lt;sneak@sneak.berlin&gt;](mailto:sneak@sneak.berlin)
- [@sneak](https://sneak.berlin)
# License
+8 -2
View File
@@ -24,8 +24,14 @@ only thing left of the `chore/align-repo-policies` branch is the list below.
# Completed Steps
- 2026-09-21: made `script/cibuild` re-run the checks on an unchanged tree via a
changing `CHECK_EPOCH` build arg in each Dockerfile check stage (#89)
- 2026-09-21: added the required README sections — named author, license, and
category in the Description first line; added Getting Started (verified
install/usage block), Rationale (folding in Problem Statement and Proposed
Solution), and a Design section for the package layout; renamed Authors to
Author (#75)
- 2026-09-21: added the canonical `.editorconfig`, made `.gitignore` cover
secrets, OS, editor, and Go artifacts, and removed the dead Drone CI
references from `.gitignore` and `bin/gitrev.sh` (#72)
- 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and
`script/fmt-check` one shared prettier file set via `script/prettier`, dropped
the `|| true` that hid prettier failures, and added a node-based Dockerfile
-5
View File
@@ -1,10 +1,5 @@
#!/bin/bash
#
if [[ ! -z "$DRONE_COMMIT_SHA" ]]; then
echo "${DRONE_COMMIT_SHA:0:7}"
exit 0
fi
if [[ ! -z "$GITREV" ]]; then
echo $GITREV
else
+4 -9
View File
@@ -1,19 +1,14 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs make fmt-check-go,
# make lint, the prettier check, and make test as build steps. A bare
# `docker build .` would serve those steps from Docker's layer cache on an
# unchanged tree, exiting 0 without running them. CHECK_EPOCH is a build
# arg placed above the check steps in every check stage; passing a fresh
# value each run busts the cache for those steps (and nothing above them),
# so a green build means the checks actually ran. Dependency layers above
# the arg stay cached. The Gitea workflow runs this on push.
# script/cibuild: run the CI build. The Dockerfile runs script/check
# (via make check), so a successful build implies all checks pass.
# Generic: needs no adaptation. The Gitea workflow runs this on push.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --build-arg CHECK_EPOCH="$(date +%s)" .
docker build .
}
main "$@"