From 204f09c6469cbb88608b6bc6eb5e52c4fe18453f Mon Sep 17 00:00:00 2001 From: sneak Date: Sun, 9 Aug 2026 05:04:38 +0000 Subject: [PATCH] Configure prettier and make fmt-check cover markdown (closes #69) script/fmt ran prettier with default settings over root-level *.md and *.json, swallowing every failure with `|| true`, while script/fmt-check checked gofmt only. The formatter and the gate therefore disagreed silently: `make fmt` rewrote markdown that `make check` never looked at, including REPO_POLICIES.md, which is a verbatim copy of an authoritative upstream document that local tooling must not touch. Configuration: - .prettierrc pins the two policy deviations from prettier defaults, four-space indents and proseWrap: always. Nothing else. - .prettierignore excludes REPO_POLICIES.md so no local run can drift it from upstream again, plus .golangci.yml (user-owned, and listed even though the current file set does not reach it) and node_modules, vendor, bin. One canonical file set: - New script/prettier takes --write or --check and applies the same patterns in both modes, so script/fmt and script/fmt-check cannot drift apart by construction. The patterns are repo-wide (**/*.md, **/*.json) rather than root-only, so markdown in subdirectories such as a future docs/ is covered. - No `|| true` anywhere, and no --no-error-on-unmatched-pattern: both patterns always match tracked files, so an empty match means the glob broke and prettier should say so instead of passing vacuously. A missing prettier is a hard error naming script/bootstrap, not a silent skip. Pinned prettier: - package.json/yarn.lock pin prettier 3.9.6; the lockfile carries the integrity hash, and --frozen-lockfile enforces it. script/prettier prefers node_modules/.bin/prettier and warns on stderr when it has to fall back to a PATH prettier of unknown version. - script/bootstrap now installs node, yarn, and the locked JS deps. Its NODE_VERSION and YARN_VERSION pins already existed. Docker gate: - The golangci-lint image has no node, so the lint stage runs the new script/fmt-check-go (the Go half of fmt-check, extracted) instead of the whole thing. - The markdown half gets its own stage on a digest-pinned node image shipping exactly the node and yarn versions bootstrap pins. The builder stage takes a COPY --from dependency on it, so BuildKit cannot skip it and a markdown violation fails `docker build .` rather than being skipped somewhere nobody looks. Markdown files other than REPO_POLICIES.md are reformatted here for the first time under the policy settings. --- .dockerignore | 1 + .gitignore | 1 + .prettierignore | 14 ++ .prettierrc | 4 + AGENTS.md | 7 +- Dockerfile | 21 +- FORMAT.md | 33 ++-- Makefile | 10 +- README.md | 459 ++++++++++++++++++++++---------------------- TODO.md | 148 +++++++------- package.json | 10 + script/bootstrap | 10 +- script/fmt | 8 +- script/fmt-check | 24 +-- script/fmt-check-go | 29 +++ script/prettier | 69 +++++++ yarn.lock | 8 + 17 files changed, 501 insertions(+), 355 deletions(-) create mode 100644 .prettierignore create mode 100644 .prettierrc create mode 100644 package.json create mode 100755 script/fmt-check-go create mode 100755 script/prettier create mode 100644 yarn.lock diff --git a/.dockerignore b/.dockerignore index ca812e4..3224cdb 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,3 +1,4 @@ *.tmp *.dockerimage .git +node_modules diff --git a/.gitignore b/.gitignore index d0d14cf..b95c22c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ /bin/ /tmp +/node_modules/ *.tmp *.dockerimage /vendor diff --git a/.prettierignore b/.prettierignore new file mode 100644 index 0000000..a6ba261 --- /dev/null +++ b/.prettierignore @@ -0,0 +1,14 @@ +# REPO_POLICIES.md is a verbatim copy of an authoritative upstream +# document (sneak/prompts). Local tooling must never rewrite it: any +# reformatting is silent drift from the source of truth. +REPO_POLICIES.md + +# User-owned configuration, copied verbatim from upstream. Not matched by +# the current prettier file set, but listed so widening that set can never +# start rewriting it. +.golangci.yml + +# Dependencies and build output +node_modules/ +vendor/ +bin/ diff --git a/.prettierrc b/.prettierrc new file mode 100644 index 0000000..8af31cd --- /dev/null +++ b/.prettierrc @@ -0,0 +1,4 @@ +{ + "tabWidth": 4, + "proseWrap": "always" +} diff --git a/AGENTS.md b/AGENTS.md index a71dee0..fb82780 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,7 +13,8 @@ source for coding standards, formatting, linting, and workflow rules. - After each change, run `make fmt`, then `make test`, then `make lint`. Fix any failures before committing. -- After each change, commit only the files you've changed. Push after committing. +- After each change, commit only the files you've changed. Push after + committing. ## Attribution @@ -26,5 +27,5 @@ source for coding standards, formatting, linting, and workflow rules. - The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are committed (required for `go get` compatibility). - The format specification is in `FORMAT.md`. -- See the TODO section in `README.md` for the 1.0 implementation plan - and open design questions. +- See the TODO section in `README.md` for the 1.0 implementation plan and open + design questions. diff --git a/Dockerfile b/Dockerfile index cf7db0b..82c49a1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,15 +11,32 @@ COPY . . # Touch .pb.go so make does not try to regenerate via protoc (file is committed) RUN touch mfer/mf.pb.go -RUN make fmt-check +# Go half of fmt-check only: this image has no node, so no prettier. The +# markdown half runs in the mdfmt stage below. +RUN make fmt-check-go RUN make lint +# Markdown/JSON format stage — prettier needs node, which the Go images +# do not have. node:22.17.0-bookworm-slim (2026-08-09); ships node +# 22.17.0 and yarn 1.22.22, the versions script/bootstrap pins. +FROM node@sha256:b04ce4ae4e95b522112c2e5c52f781471a5cbc3b594527bcddedee9bc48c03a0 AS mdfmt + +WORKDIR /src +COPY package.json yarn.lock ./ +RUN yarn install --frozen-lockfile + +COPY . . + +# No make in this image; call the script entrypoint directly. +RUN script/prettier --check + # Build stage — tests and compilation # golang:1.23 (2026-03-14) FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder -# Force BuildKit to run the lint stage by creating a stage dependency +# Force BuildKit to run the lint and mdfmt stages by creating stage dependencies COPY --from=lint /src/go.sum /dev/null +COPY --from=mdfmt /src/go.sum /dev/null WORKDIR /src COPY go.mod go.sum ./ diff --git a/FORMAT.md b/FORMAT.md index ec661cb..1e4a697 100644 --- a/FORMAT.md +++ b/FORMAT.md @@ -5,9 +5,9 @@ Version 1.0 ## Overview An `.mf` file is a binary manifest that describes a directory tree of files, -including their paths, sizes, and cryptographic checksums. It supports -optional GPG signatures for integrity verification and optional timestamps -for metadata preservation. +including their paths, sizes, and cryptographic checksums. It supports optional +GPG signatures for integrity verification and optional timestamps for metadata +preservation. ## File Structure @@ -39,15 +39,15 @@ The outer message contains: ### SHA-256 Hash -The `sha256` field (104) covers the **compressed** `innerMessage` bytes. -This allows verifying data integrity before decompression. +The `sha256` field (104) covers the **compressed** `innerMessage` bytes. This +allows verifying data integrity before decompression. ## Compression -The `innerMessage` field is compressed with [Zstandard (zstd)](https://facebook.github.io/zstd/). -Implementations must enforce a decompression size limit to prevent -decompression bombs. The reference implementation limits decompressed size to -256 MB. +The `innerMessage` field is compressed with +[Zstandard (zstd)](https://facebook.github.io/zstd/). Implementations must +enforce a decompression size limit to prevent decompression bombs. The reference +implementation limits decompressed size to 256 MB. ## Inner Message (`MFFile`) @@ -114,11 +114,11 @@ Where: - `ZNAVSRFG` is the magic bytes string (literal ASCII) - `` is the hex-encoded UUID from the outer message -- `` is the hex-encoded SHA-256 hash from the outer message (covering compressed data) +- `` is the hex-encoded SHA-256 hash from the outer message (covering + compressed data) -Components are separated by hyphens. The signature is produced by GPG over -this canonical string and stored in the `signature` field of the outer -message. +Components are separated by hyphens. The signature is produced by GPG over this +canonical string and stored in the `signature` field of the outer message. ## Deterministic Serialization @@ -134,10 +134,11 @@ changed). ## MIME Type -The recommended MIME type for `.mf` files is `application/octet-stream`. -The `.mf` file extension is the canonical identifier. +The recommended MIME type for `.mf` files is `application/octet-stream`. The +`.mf` file extension is the canonical identifier. ## Reference - Proto definition: [`mfer/mf.proto`](mfer/mf.proto) -- Reference implementation: [git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) +- Reference implementation: + [git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) diff --git a/Makefile b/Makefile index d9dbfa3..e93908f 100644 --- a/Makefile +++ b/Makefile @@ -13,7 +13,7 @@ GOLDFLAGS += -X main.Version=$(VERSION) GOLDFLAGS += -X main.Gitrev=$(GITREV_BUILD) GOFLAGS := -ldflags "$(GOLDFLAGS)" -.PHONY: bootstrap setup docker default run ci test check lint fmt fmt-check hooks fixme +.PHONY: bootstrap setup docker default run ci test check lint fmt fmt-check fmt-check-go fmt-check-md hooks fixme default: fmt test @@ -44,6 +44,14 @@ check: fmt-check: @script/fmt-check +# Halves of fmt-check, for environments that have only one toolchain: +# the Docker lint stage has Go but no node, the markdown stage the reverse. +fmt-check-go: + @script/fmt-check-go + +fmt-check-md: + @script/prettier --check + hooks: @script/install-precommit diff --git a/README.md b/README.md index ea469ce..ec05d47 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,80 @@ # mfer -[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library -and thin wrapper command-line utility written in [Go](https://golang.org) -and first published in 2022 under the [WTFPL](https://wtfpl.net) (public -domain) license. It specifies and generates `.mf` manifest files over a -directory tree of files to encapsulate metadata about them (such as -cryptographic checksums or signatures over same) to aid in archiving, -downloading, and streaming, or mirroring. The manifest files' data is -serialized with Google's [protobuf serialization -format](https://developers.google.com/protocol-buffers). The structure of -these files can be found [in the format -specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto) -which is included in the [project -repository](https://git.eeqj.de/sneak/mfer). +[mfer](https://git.eeqj.de/sneak/mfer) is a reference implementation library and +thin wrapper command-line utility written in [Go](https://golang.org) and first +published in 2022 under the [WTFPL](https://wtfpl.net) (public domain) license. +It specifies and generates `.mf` manifest files over a directory tree of files +to encapsulate metadata about them (such as cryptographic checksums or +signatures over same) to aid in archiving, downloading, and streaming, or +mirroring. The manifest files' data is serialized with Google's +[protobuf serialization format](https://developers.google.com/protocol-buffers). +The structure of these files can be found +[in the format specification](https://git.eeqj.de/sneak/mfer/src/branch/main/mfer/mf.proto) +which is included in the [project repository](https://git.eeqj.de/sneak/mfer). -The current version is pre-1.0 and while the repo was published in 2022, -there has not yet been any versioned release. [SemVer](https://semver.org) -will be used for releases. +The current version is pre-1.0 and while the repo was published in 2022, there +has not yet been any versioned release. [SemVer](https://semver.org) will be +used for releases. -This project was started by [@sneak](https://sneak.berlin) to scratch an -itch in 2022 and is currently a one-person effort, though the goal is for -this to emerge as a de-facto standard and be incorporated into other -software. A compatible javascript library is planned. +This project was started by [@sneak](https://sneak.berlin) to scratch an itch in +2022 and is currently a one-person effort, though the goal is for this to emerge +as a de-facto standard and be incorporated into other software. A compatible +javascript library is planned. # Build Status -CI runs via `script/cibuild` (`docker build .`), which executes `make -check` (formatting, linting, tests). The `main` branch must always be -green. +CI runs via `script/cibuild` (`docker build .`), which executes `make check` +(formatting, linting, tests). The `main` branch must always be green. # Entrypoints This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call -them. We provide: +development workflow, and the Makefile targets are thin shims that call them. We +provide: -- `script/bootstrap` — install all dependencies (Go, golangci-lint, Go - module download), idempotently +- `script/bootstrap` — install all dependencies (Go, golangci-lint, Go module + download, and node/yarn plus the prettier version pinned in + `package.json`/`yarn.lock`), idempotently - `script/setup` — make a fresh clone ready for development: runs `script/bootstrap`, then `script/install-precommit` -- `script/projectname` — output the project name (`mfer`); used by other - scripts such as `script/docker` -- `script/test` — run the test suite (`go test`), regenerating the - protobuf code first if it is stale +- `script/projectname` — output the project name (`mfer`); used by other scripts + such as `script/docker` +- `script/test` — run the test suite (`go test`), regenerating the protobuf code + first if it is stale - `script/lint` — run `golangci-lint` and verify `gofmt` cleanliness - `script/fmt` — format all code and docs (writes): `gofumpt`, - `golangci-lint run --fix`, and prettier for JSON/Markdown -- `script/fmt-check` — check formatting without writing -- `script/check` — run `script/test`, `script/lint`, and - `script/fmt-check` + `golangci-lint run --fix`, and `script/prettier --write` +- `script/prettier` — run prettier over the repository's canonical file set + (Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or + `--check`; the single definition of that file set, so `script/fmt` and + `script/fmt-check` cannot disagree about it +- `script/fmt-check` — check formatting without writing: `script/fmt-check-go` + plus `script/prettier --check` +- `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the + Docker lint stage, whose image has no node +- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check` - `script/docker` — build the Docker image tagged with the project name -- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile - runs the checks) -- `script/precommit` — pre-commit checks: `go mod tidy` verification, - then `script/check` -- `script/install-precommit` — install the git pre-commit hook that - runs `script/precommit` +- `script/cibuild` — CI entrypoint: `docker build .` (the Dockerfile runs the + checks) +- `script/precommit` — pre-commit checks: `go mod tidy` verification, then + `script/check` +- `script/install-precommit` — install the git pre-commit hook that runs + `script/precommit` # Participation -The community is as yet nonexistent so there are no defined policies or -norms yet. Primary development happens on a privately-run Gitea instance at -[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues -are [tracked there](https://git.eeqj.de/sneak/mfer/issues). +The community is as yet nonexistent so there are no defined policies or norms +yet. Primary development happens on a privately-run Gitea instance at +[https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) and issues are +[tracked there](https://git.eeqj.de/sneak/mfer/issues). -Changes must always be formatted with a standard `go fmt`, syntactically -valid, and must pass the linting defined in the repository (presently only -the `golangci-lint` defaults), which can be run with a `make lint`. The -`main` branch is protected and all changes must be made via [pull -requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged. +Changes must always be formatted with a standard `go fmt`, syntactically valid, +and must pass the linting defined in the repository (presently only the +`golangci-lint` defaults), which can be run with a `make lint`. The `main` +branch is protected and all changes must be made via +[pull requests](https://git.eeqj.de/sneak/mfer/pulls) and pass CI to be merged. Any changes submitted to this project must also be [WTFPL-licensed](https://wtfpl.net) to be considered. @@ -81,58 +85,57 @@ tooling requirements, and workflow conventions. Given a plain URL, there is no standard way to safely and programmatically download everything "under" that URL path. `wget -r` can traverse directory -listings if they're enabled, but every server has a different format, and -this does not verify cryptographic integrity of the files, or enable them to -be fetched using a different protocol other than HTTP/s. +listings if they're enabled, but every server has a different format, and this +does not verify cryptographic integrity of the files, or enable them to be +fetched using a different protocol other than HTTP/s. -Currently, the solution that people are using are sidecar files in the -format of `SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached -signature. This is not checksum-algorithm-agnostic and the sidecar file is -not always consistently named. +Currently, the solution that people are using are sidecar files in the format of +`SHASUMS` checksum files, as well as a `SHASUMS.asc` PGP detached signature. +This is not checksum-algorithm-agnostic and the sidecar file is not always +consistently named. Real issues I face: - when I plug in an ExFAT hard drive, I don't know if any files on the filesystem are corrupted or missing - - current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files + - current ad-hoc solution are `SHASUMS`/`SHASUMS.asc` files - when I want to mirror an HTTP archive, I have to use special tools like debmirror that understand the archive format - - the debian repository metadata structure is hot garbage -- when I download a large file via HTTP, I have no way of knowing if the - file content is what it's supposed to be + - the debian repository metadata structure is hot garbage +- when I download a large file via HTTP, I have no way of knowing if the file + content is what it's supposed to be # Proposed Solution A standard, a manifest file format, and a tool for generating same. -The manifest file would be called `index.mf`, and the tool for generating such would be called `mfer`. +The manifest file would be called `index.mf`, and the tool for generating such +would be called `mfer`. The manifest file would do several important things: -- have a standard filename, so if given - `https://example.com/downloadpackage/` one could fetch - `https://example.com/downloadpackage/index.mf` to enumerate the full - directory listing. +- have a standard filename, so if given `https://example.com/downloadpackage/` + one could fetch `https://example.com/downloadpackage/index.mf` to enumerate + the full directory listing. - contain a version field for extensibility - contain structured data (protobuf, json, or cbor) -- provide an inner signed container, so that the manifest file itself can - embed a signature and a public key alongside in a single file +- provide an inner signed container, so that the manifest file itself can embed + a signature and a public key alongside in a single file - contain a list of files, each with a relative path to the manifest - contain manifest timestamp - contain ctime/mtime information for files so that file metadata can be preserved -- contain cryptographic checksums in several different algorithms for each - file - - probably encoded with multihash to indicate algo + hash - - sha256 at the minimum - - would be nice to include an IPFS/IPLD CIDv1 root hash for each file, - which likely involves doing an ipfs file object chunking - - maybe even including the complete IPFS/IPLD directory tree objects and - chunklists? - - this is because generating an `index.mf` does not imply publishing on - ipfs at that time - - maybe a bittorrent chunklist for torrent client compatibility? perhaps a - top-level infohash for the whole manifest? +- contain cryptographic checksums in several different algorithms for each file + - probably encoded with multihash to indicate algo + hash + - sha256 at the minimum + - would be nice to include an IPFS/IPLD CIDv1 root hash for each file, which + likely involves doing an ipfs file object chunking + - maybe even including the complete IPFS/IPLD directory tree objects and + chunklists? + - this is because generating an `index.mf` does not imply publishing on + ipfs at that time + - maybe a bittorrent chunklist for torrent client compatibility? perhaps a + top-level infohash for the whole manifest? # Design Goals @@ -146,37 +149,37 @@ The manifest file would do several important things: # Non-Goals - Manifest generation speed - - likely involves IPFS chunking, bittorrent chunking, and several - different cryptographic hash functions over the entirety of each and - every file + - likely involves IPFS chunking, bittorrent chunking, and several different + cryptographic hash functions over the entirety of each and every file - Small manifest file size (within reason) - - 30MiB files are "small" these days, given modern storage/bandwidth - - metadata size should not be used as an excuse to sacrifice utility (such - as providing checksums over each chunk of a large file) + - 30MiB files are "small" these days, given modern storage/bandwidth + - metadata size should not be used as an excuse to sacrifice utility (such + as providing checksums over each chunk of a large file) # Open Questions -- Should the manifest file include checksums of individual file chunks, or just for the whole assembled file? +- Should the manifest file include checksums of individual file chunks, or just + for the whole assembled file? - - If so, should the chunksize be fixed or dynamic? + - If so, should the chunksize be fixed or dynamic? - Should the manifest signature format be GnuPG signatures, or those from - OpenBSD's signify (of which there is a good [golang - implementation](https://github.com/frankbraun/gosignify)? + OpenBSD's signify (of which there is a good + [golang implementation](https://github.com/frankbraun/gosignify)? - Should the on-disk serialization format be proto3 or json? # Tool Examples - `mfer gen` / `mfer gen .` - - recurses under current directory and writes out an `index.mf` + - recurses under current directory and writes out an `index.mf` - `mfer check` / `mfer check .` - - verifies checksums of all files in manifest, displaying error and - exiting nonzero if any files are missing or corrupted + - verifies checksums of all files in manifest, displaying error and exiting + nonzero if any files are missing or corrupted - `mfer fetch https://example.com/stuff/` - - fetches `/stuff/index.mf` and downloads all files listed in manifest, - optionally resuming any that already exist locally, and assures - cryptographic integrity of downloaded files. + - fetches `/stuff/index.mf` and downloads all files listed in manifest, + optionally resuming any that already exist locally, and assures + cryptographic integrity of downloaded files. # Implementation Plan @@ -193,30 +196,31 @@ The manifest file would do several important things: # Hopes And Dreams - `aria2c https://example.com/manifestdirectory/` - - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and - checksums all files, resumes any that exist locally already) + - (fetches `https://example.com/manifestdirectory/index.mf`, downloads and + checksums all files, resumes any that exist locally already) - `mfer fetch https://example.com/manifestdirectory/` -- a command line option to zero/omit mtime/ctime, as well as manifest - timestamp, and sort all directory listings so that manifest file - generation is deterministic/reproducible -- URL format `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2` - to assert in the URL which PGP signing key should be used in the manifest, - so that shared URLs have a cryptographic trust root -- a "well-known" key in the manifest that maps well known keys (could reuse - the http spec) to specific file paths in the manifest. - - example: a `berlin.sneak.app.slideshow` key that maps to a json - slideshow config listing what image paths to show, and for how long, and - in what order +- a command line option to zero/omit mtime/ctime, as well as manifest timestamp, + and sort all directory listings so that manifest file generation is + deterministic/reproducible +- URL format + `mfer fetch https://exmaple.com/manifestdirectory/?key=5539AD00DE4C42F3AFE11575052443F4DF2A55C2` + to assert in the URL which PGP signing key should be used in the manifest, so + that shared URLs have a cryptographic trust root +- a "well-known" key in the manifest that maps well known keys (could reuse the + http spec) to specific file paths in the manifest. + - example: a `berlin.sneak.app.slideshow` key that maps to a json slideshow + config listing what image paths to show, and for how long, and in what + order # Use Cases ## Web Images I'd like to be able to put a bunch of images into a directory, generate a -manifest, and then point a slideshow client (such as an ambient display, or -a react app with the target directory in a query string arg) at that -statically hosted directory, and have it discover the full list of images -available at that URL. +manifest, and then point a slideshow client (such as an ambient display, or a +react app with the target directory in a query string arg) at that statically +hosted directory, and have it discover the full list of images available at that +URL. ## Software Distribution @@ -226,20 +230,20 @@ resumably by either HTTP or IPFS/BitTorrent without a .torrent file. ## Filesystem Archive Integrity I use filesystems that don't include data checksums, and I would like a -cryptographically signed checksum file so that I can later verify that a set -of archive files have not been modified, none are missing, and that the -checksums have not been altered in storage by a second party. +cryptographically signed checksum file so that I can later verify that a set of +archive files have not been modified, none are missing, and that the checksums +have not been altered in storage by a second party. ## Filesystem-Independent Checksums -I would like to be able to plug in a hard drive or flash drive and, if there -is an `index.mf` in the root, automatically detect missing/corrupted files, +I would like to be able to plug in a hard drive or flash drive and, if there is +an `index.mf` in the root, automatically detect missing/corrupted files, regardless of filesystem format. # Collaboration -Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your -desired username for an account on this Gitea instance. +Please email [`sneak@sneak.berlin`](mailto:sneak@sneak.berlin) with your desired +username for an account on this Gitea instance. # TODO: Remaining Work for 1.0 @@ -250,114 +254,108 @@ inline below each question. ### Format Design -**1. Should `MFFileChecksum` be simplified?** Currently it's a separate -message wrapping a single `bytes multiHash` field. Since multihash -already self-describes the algorithm, `repeated bytes hashes` directly on -`MFFilePath` would be simpler and reduce per-file protobuf overhead. Is -the extra message layer intentional (e.g. planning to add per-hash -metadata like `verified_at`)? +**1. Should `MFFileChecksum` be simplified?** Currently it's a separate message +wrapping a single `bytes multiHash` field. Since multihash already +self-describes the algorithm, `repeated bytes hashes` directly on `MFFilePath` +would be simpler and reduce per-file protobuf overhead. Is the extra message +layer intentional (e.g. planning to add per-hash metadata like `verified_at`)? > _answer:_ -**2. Should file permissions/mode be stored?** The format stores -mtime/ctime but not Unix file permissions. For archival use this may not -matter, but for software distribution or filesystem restoration it's a -gap. Should we reserve a field now (e.g. `optional uint32 mode = 305`) -even if we don't populate it yet? +**2. Should file permissions/mode be stored?** The format stores mtime/ctime but +not Unix file permissions. For archival use this may not matter, but for +software distribution or filesystem restoration it's a gap. Should we reserve a +field now (e.g. `optional uint32 mode = 305`) even if we don't populate it yet? > _answer:_ -**3. Should `atime` be removed from the schema?** Access time is -volatile, non-deterministic, and often disabled (`noatime`). Including it -means two manifests of the same directory at different times will differ, -which conflicts with the determinism goal. Remove it, or document it as -"never set by default"? +**3. Should `atime` be removed from the schema?** Access time is volatile, +non-deterministic, and often disabled (`noatime`). Including it means two +manifests of the same directory at different times will differ, which conflicts +with the determinism goal. Remove it, or document it as "never set by default"? > _answer:_ -**4. What are the path normalization rules?** The proto has `string path` -with no specification about: always forward-slash? Must be relative? No -`..` components allowed? UTF-8 NFC vs NFD normalization (macOS vs -Linux)? Max path length? This is a security issue (path traversal) and a -cross-platform compatibility issue. What rules should the spec mandate? +**4. What are the path normalization rules?** The proto has `string path` with +no specification about: always forward-slash? Must be relative? No `..` +components allowed? UTF-8 NFC vs NFD normalization (macOS vs Linux)? Max path +length? This is a security issue (path traversal) and a cross-platform +compatibility issue. What rules should the spec mandate? > _answer:_ -**5. Should we add a version byte after the magic?** Currently -`ZNAVSRFG` is followed immediately by protobuf. Adding a version byte -(`ZNAVSRFG\x01`) would allow future framing changes without requiring -protobuf parsing to detect the version. `MFFileOuter.Version` serves -this purpose but requires successful deserialization to read. Worth the -extra byte? +**5. Should we add a version byte after the magic?** Currently `ZNAVSRFG` is +followed immediately by protobuf. Adding a version byte (`ZNAVSRFG\x01`) would +allow future framing changes without requiring protobuf parsing to detect the +version. `MFFileOuter.Version` serves this purpose but requires successful +deserialization to read. Worth the extra byte? > _answer:_ **6. Should we add a length-prefix after the magic?** Protobuf is not -self-delimiting. If we ever want to concatenate manifests or append data -after the protobuf, the current framing is insufficient. Add a varint or -fixed-width length-prefix? +self-delimiting. If we ever want to concatenate manifests or append data after +the protobuf, the current framing is insufficient. Add a varint or fixed-width +length-prefix? > _answer:_ ### Signature Design -**7. What does the outer SHA-256 hash cover — compressed or uncompressed -data?** The code currently hashes compressed data (good for verifying -before decompression), but this should be explicitly documented. Which is -the intended behavior? +**7. What does the outer SHA-256 hash cover — compressed or uncompressed data?** +The code currently hashes compressed data (good for verifying before +decompression), but this should be explicitly documented. Which is the intended +behavior? > _answer:_ **8. Should `signatureString()` sign raw bytes instead of a hex-encoded -string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with -hex encoding, which adds a transformation layer. Signing the raw `sha256` -bytes (or compressed `innerMessage` directly) would be simpler. Keep the -string format or switch to raw bytes? +string?** Currently the canonical string is `MAGIC-UUID-MULTIHASH` with hex +encoding, which adds a transformation layer. Signing the raw `sha256` bytes (or +compressed `innerMessage` directly) would be simpler. Keep the string format or +switch to raw bytes? > _answer:_ **9. Should we support detached signature files (`.mf.sig`)?** Embedded -signatures are better for single-file distribution. Detached `.mf.sig` -files follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are -simpler for HTTP serving. Support both modes? +signatures are better for single-file distribution. Detached `.mf.sig` files +follow the familiar `SHASUMS`/`SHASUMS.asc` pattern and are simpler for HTTP +serving. Support both modes? > _answer:_ -**10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is -fragile (may not be installed, version-dependent output). -`github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could -use Ed25519/signify (simpler, no key management). Which direction? +**10. GPG vs pure-Go crypto for signatures?** Shelling out to `gpg` is fragile +(may not be installed, version-dependent output). +`github.com/ProtonMail/go-crypto` provides pure-Go OpenPGP, or we could use +Ed25519/signify (simpler, no key management). Which direction? > _answer:_ ### Implementation Design -**11. Should manifests be deterministic by default?** This means: sort -file entries by path, omit `createdAt` timestamp (or make it opt-in), no -`atime`. Should determinism be the default, with a -`--include-timestamps` flag to opt in? +**11. Should manifests be deterministic by default?** This means: sort file +entries by path, omit `createdAt` timestamp (or make it opt-in), no `atime`. +Should determinism be the default, with a `--include-timestamps` flag to opt in? > _answer:_ -**12. Should we consolidate or keep both scanner/checker -implementations?** There are two parallel implementations: -`mfer/scanner.go` + `mfer/checker.go` (typed with `FileSize`, -`RelFilePath`) and `internal/scanner/` + `internal/checker/` (raw -`int64`, `string`). The `mfer/` versions are superior. Delete the -`internal/` versions? +**12. Should we consolidate or keep both scanner/checker implementations?** +There are two parallel implementations: `mfer/scanner.go` + `mfer/checker.go` +(typed with `FileSize`, `RelFilePath`) and `internal/scanner/` + +`internal/checker/` (raw `int64`, `string`). The `mfer/` versions are superior. +Delete the `internal/` versions? > _answer:_ **13. Should the `manifest` type be exported?** Currently unexported with exported constructors (`NewManifestFromReader`, `NewManifestFromFile`). -Consumers can't declare `var m *mfer.manifest`. Export the type, or -define an interface? +Consumers can't declare `var m *mfer.manifest`. Export the type, or define an +interface? > _answer:_ **14. What should the Go module path be for 1.0?** Currently -`sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in -the proto `go_package` option. Which is canonical? +`sneak.berlin/go/mfer` in `go.mod` but `git.eeqj.de/sneak/mfer/mfer` in the +proto `go_package` option. Which is canonical? > _answer:_ @@ -375,39 +373,37 @@ the proto `go_package` option. Which is canonical? - [ ] Resolve proto `go_package` path inconsistency (`git.eeqj.de/sneak/mfer/mfer` vs `sneak.berlin/go/mfer`) - [ ] Specify path invariants — add proto comments requiring UTF-8, - forward-slash, relative paths, no `..`, no leading `/`; validate - in `Builder.AddFile` and `Builder.AddFileWithHash` (pending design - question answer) -- [ ] Remove or deprecate `atime` from proto (pending design question + forward-slash, relative paths, no `..`, no leading `/`; validate in + `Builder.AddFile` and `Builder.AddFileWithHash` (pending design question answer) -- [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future - file permissions (pending design question answer) -- [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version - 1 (pending design question answer) -- [ ] Write format specification document — separate from README: - magic, outer structure, compression, inner structure, path - invariants, signature scheme, canonical serialization +- [ ] Remove or deprecate `atime` from proto (pending design question answer) +- [ ] Reserve `optional uint32 mode = 305` in `MFFilePath` for future file + permissions (pending design question answer) +- [ ] Add version byte after magic — `ZNAVSRFG\x01` for format version 1 + (pending design question answer) +- [ ] Write format specification document — separate from README: magic, outer + structure, compression, inner structure, path invariants, signature + scheme, canonical serialization ### Library -- [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate - on `mfer/` package versions; update CLI code (pending design - question answer) -- [ ] Add deterministic file ordering — sort entries by path - (lexicographic, byte-order) in `Builder.Build()`; add test - asserting byte-identical output from two runs -- [ ] Add decompression size limit — `io.LimitReader` in - `deserializeInner()` with `m.pbOuter.Size` as bound -- [ ] Fix `errors.Is` dead code in checker — replace with - `os.IsNotExist(err)` or `errors.Is(err, fs.ErrNotExist)` -- [ ] Fix `AddFile` to verify size — check `totalRead == size` after - reading, return error on mismatch -- [ ] Export the `manifest` type or define a public interface (pending - design question answer) — currently consumers cannot hold a reference - to a loaded manifest in their own type declarations -- [ ] Replace GPG subprocess calls with pure-Go crypto (pending design - question answer) — current implementation shells out to `gpg` which - may not be installed +- [ ] Delete `internal/scanner/` and `internal/checker/` — consolidate on + `mfer/` package versions; update CLI code (pending design question answer) +- [ ] Add deterministic file ordering — sort entries by path (lexicographic, + byte-order) in `Builder.Build()`; add test asserting byte-identical output + from two runs +- [ ] Add decompression size limit — `io.LimitReader` in `deserializeInner()` + with `m.pbOuter.Size` as bound +- [ ] Fix `errors.Is` dead code in checker — replace with `os.IsNotExist(err)` + or `errors.Is(err, fs.ErrNotExist)` +- [ ] Fix `AddFile` to verify size — check `totalRead == size` after reading, + return error on mismatch +- [ ] Export the `manifest` type or define a public interface (pending design + question answer) — currently consumers cannot hold a reference to a loaded + manifest in their own type declarations +- [ ] Replace GPG subprocess calls with pure-Go crypto (pending design question + answer) — current implementation shells out to `gpg` which may not be + installed - [ ] Add timeout to any remaining subprocess calls ### CLI @@ -416,35 +412,33 @@ the proto `go_package` option. Which is canonical? (`--include-dotfiles`, `--follow-symlinks`) - [ ] Fix URL construction in fetch — use `BaseURL.JoinPath()` or `url.JoinPath()` instead of string concatenation -- [ ] Add progress rate-limiting to Checker — throttle to once per - second, matching Scanner -- [ ] Add `--deterministic` flag or make it default — omit `createdAt`, - sort files (pending design question answer) -- [ ] Wire `--version` flag properly (currently only a `version` - subcommand exists; top-level `--version` shows urfave/cli generic - output) -- [ ] Add retry logic to `fetch` — currently no retries on transient - HTTP errors; needs exponential backoff -- [ ] `fetch` command uses bare `http.Get` with no timeout — needs - `http.Client` with configurable timeout +- [ ] Add progress rate-limiting to Checker — throttle to once per second, + matching Scanner +- [ ] Add `--deterministic` flag or make it default — omit `createdAt`, sort + files (pending design question answer) +- [ ] Wire `--version` flag properly (currently only a `version` subcommand + exists; top-level `--version` shows urfave/cli generic output) +- [ ] Add retry logic to `fetch` — currently no retries on transient HTTP + errors; needs exponential backoff +- [ ] `fetch` command uses bare `http.Get` with no timeout — needs `http.Client` + with configurable timeout ### Testing & Robustness -- [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf - deserialization of untrusted input needs fuzz coverage -- [ ] Add integration test for `freshen` CLI command — current tests - only verify setup, not the actual freshen operation end-to-end -- [ ] Add test for `fetch` CLI command end-to-end (currently only - `downloadFile` is tested) +- [ ] Add fuzzing tests for `NewManifestFromReader` — protobuf deserialization + of untrusted input needs fuzz coverage +- [ ] Add integration test for `freshen` CLI command — current tests only verify + setup, not the actual freshen operation end-to-end +- [ ] Add test for `fetch` CLI command end-to-end (currently only `downloadFile` + is tested) ### Documentation -- [ ] Promote `FORMAT.md` as primary spec reference; README should link - to it more prominently -- [ ] Audit and update all error messages for consistency and - helpfulness -- [ ] Document the signature scheme more thoroughly (canonical string - format, verification steps) +- [ ] Promote `FORMAT.md` as primary spec reference; README should link to it + more prominently +- [ ] Audit and update all error messages for consistency and helpfulness +- [ ] Document the signature scheme more thoroughly (canonical string format, + verification steps) ### Release @@ -465,7 +459,8 @@ the proto `go_package` option. Which is canonical? ## Links - Repo: [https://git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer) -- Issues: [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues) +- Issues: + [https://git.eeqj.de/sneak/mfer/issues](https://git.eeqj.de/sneak/mfer/issues) # Authors diff --git a/TODO.md b/TODO.md index 983c596..06f9dba 100644 --- a/TODO.md +++ b/TODO.md @@ -10,102 +10,100 @@ # Status -pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists -open design questions and implementation tasks; policy compliance work is -in flight and unmerged. +pre-1.0. No git tags. README section "TODO: Remaining Work for 1.0" lists open +design questions and implementation tasks; policy compliance work is in flight +and unmerged. # Next Step -Work through the remaining compliance items folded from the 2026-07-02 -audit (the first group under Future Steps): `.editorconfig`, `.gitignore` -coverage, gofumpt-based `fmt-check`, README "Getting Started", and the -rest. `.golangci.yml` and `TODO.md` are tracked and committed as of -2026-08-07, so the only thing left of the `chore/align-repo-policies` -branch is the list below. +Work through the remaining compliance items folded from the 2026-07-02 audit +(the first group under Future Steps): `.editorconfig`, `.gitignore` coverage, +gofumpt-based `fmt-check`, README "Getting Started", and the rest. +`.golangci.yml` and `TODO.md` are tracked and committed as of 2026-08-07, so the +only thing left of the `chore/align-repo-policies` branch is the list below. # Completed Steps +- 2026-08-09: added `.prettierrc`/`.prettierignore`, gave `script/fmt` and + `script/fmt-check` one shared prettier file set via `script/prettier`, dropped + the `|| true` that hid prettier failures, and added a node-based Dockerfile + stage so a markdown formatting violation fails `docker build .` (#69) - 2026-08-07: updated golangci-lint to v2.12.2 everywhere it is pinned (`Makefile`, `Dockerfile`), added the canonical `.golangci.yml` - (`default: all`), and fixed all resulting lint findings across the - codebase -- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, - Makefile shims, README Entrypoints section -- 2026-07-03: aligned repo tooling, docs, and config with standardized - policies (7d9a138, on chore/align-repo-policies, unmerged) + (`default: all`), and fixed all resulting lint findings across the codebase +- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile + shims, README Entrypoints section +- 2026-07-03: aligned repo tooling, docs, and config with standardized policies + (7d9a138, on chore/align-repo-policies, unmerged) - 2026-06-28: moved to standardized repo policies (#56, on main) -- 2026-04-07: added 1.0 roadmap as README TODO section, removed old - TODO.md (#54) +- 2026-04-07: added 1.0 roadmap as README TODO section, removed old TODO.md + (#54) - 2026-03-20: added Gitea Actions CI workflow (#53) - 2026-03-17: added REPO_POLICIES.md, renamed CLAUDE.md to AGENTS.md (#51); removed committed .index.mf (#52) -- 2026-03-15: split Dockerfile with pre-built golangci-lint stage for - faster CI (#45) +- 2026-03-15: split Dockerfile with pre-built golangci-lint stage for faster CI + (#45) - 2026-03-01: 1.0 quality polish: code review, tests, bug fixes, docs (#32) -- 2026-02-20: deterministic file ordering in Builder.Build() (#28); - removed committed vendor/modcache archives (#35) +- 2026-02-20: deterministic file ordering in Builder.Build() (#28); removed + committed vendor/modcache archives (#35) - 2026-02-08: added --seed flag for deterministic manifest UUID # Future Steps -- Compliance (fold of TODO.md audit 2026-07-02; verify which items the - in-flight branch already closes, then check off): - - Add .editorconfig (canonical copy from sneak/prompts) - - Make .gitignore cover secrets (.env, _.key, _.pem), OS files - (.DS_Store), and editor files (_.swp, _~) - - Make fmt-check/lint verify with gofumpt, not gofmt -l, so - `make check` matches what `make fmt` writes - - Add README "Getting Started" section with copy-pasteable - install/usage block - - Move FORMAT.md from repo root to docs/ and update the AGENTS.md - reference - - Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`, - `golangci-lint@v2.12.2`) by module hash, not mutable tag - - Set `make test` timeout to 30s (currently 10s) - - Add explicit README "Rationale" heading (content exists under other - names); name the author in the README Description first line - - Reconcile root-level AGENTS.md with directory-hygiene policy (keep - or relocate) - - Add a `make build` target - - Rewrite `make hooks` to use printf or a heredoc instead of - non-portable `echo '...\n...'` +- Compliance (fold of TODO.md audit 2026-07-02; verify which items the in-flight + branch already closes, then check off): + - Add .editorconfig (canonical copy from sneak/prompts) + - Make .gitignore cover secrets (.env, _.key, _.pem), OS files (.DS_Store), + and editor files (_.swp, _~) + - Make fmt-check/lint verify with gofumpt, not gofmt -l, so `make check` + matches what `make fmt` writes + - Add README "Getting Started" section with copy-pasteable install/usage + block + - Move FORMAT.md from repo root to docs/ and update the AGENTS.md reference + - Pin Makefile-installed Go tools (`protoc-gen-go@v1.28.1`, + `golangci-lint@v2.12.2`) by module hash, not mutable tag + - Set `make test` timeout to 30s (currently 10s) + - Add explicit README "Rationale" heading (content exists under other + names); name the author in the README Description first line + - Reconcile root-level AGENTS.md with directory-hygiene policy (keep or + relocate) + - Add a `make build` target + - Rewrite `make hooks` to use printf or a heredoc instead of non-portable + `echo '...\n...'` - Answer the 14 owner design questions in the README 1.0 roadmap: - - Format: simplify MFFileChecksum; store file mode; drop atime; - specify path normalization rules; version byte after magic; - length-prefix after magic - - Signatures: hash covers compressed or uncompressed data; sign raw - bytes vs hex canonical string; detached .mf.sig support; GPG - subprocess vs pure-Go crypto - - Implementation: deterministic manifests by default; consolidate - duplicate scanner/checker implementations; export the manifest - type; canonical Go module path for 1.0 + - Format: simplify MFFileChecksum; store file mode; drop atime; specify path + normalization rules; version byte after magic; length-prefix after magic + - Signatures: hash covers compressed or uncompressed data; sign raw bytes vs + hex canonical string; detached .mf.sig support; GPG subprocess vs pure-Go + crypto + - Implementation: deterministic manifests by default; consolidate duplicate + scanner/checker implementations; export the manifest type; canonical Go + module path for 1.0 - Format and correctness: - - Resolve proto go_package vs go.mod module path inconsistency - - Specify and validate path invariants (UTF-8, forward-slash, - relative, no .., no leading /) - - Remove or deprecate atime; reserve mode field; add version byte - (all pending design answers) - - Write a standalone format specification document + - Resolve proto go_package vs go.mod module path inconsistency + - Specify and validate path invariants (UTF-8, forward-slash, relative, no + .., no leading /) + - Remove or deprecate atime; reserve mode field; add version byte (all + pending design answers) + - Write a standalone format specification document - Library: - - Delete internal/scanner and internal/checker; consolidate on the - mfer/ package versions (pending design answer) - - Add decompression size limit via io.LimitReader in - deserializeInner() - - Fix errors.Is dead code in checker; make AddFile verify - totalRead == size - - Export manifest type or define a public interface (pending) - - Replace GPG subprocess with pure-Go crypto (pending); add timeouts - to remaining subprocess calls + - Delete internal/scanner and internal/checker; consolidate on the mfer/ + package versions (pending design answer) + - Add decompression size limit via io.LimitReader in deserializeInner() + - Fix errors.Is dead code in checker; make AddFile verify totalRead == size + - Export manifest type or define a public interface (pending) + - Replace GPG subprocess with pure-Go crypto (pending); add timeouts to + remaining subprocess calls - CLI: - - Kebab-case primary flag names; fix fetch URL construction with - url.JoinPath; add http.Client timeout and retry with backoff to - fetch; rate-limit Checker progress output; add --deterministic - flag or default; wire top-level --version properly + - Kebab-case primary flag names; fix fetch URL construction with + url.JoinPath; add http.Client timeout and retry with backoff to fetch; + rate-limit Checker progress output; add --deterministic flag or default; + wire top-level --version properly - Testing: - - Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch + - Fuzz NewManifestFromReader; end-to-end tests for freshen and fetch - Documentation: - - Promote docs/FORMAT.md as primary spec reference; audit error - messages; document the signature scheme fully + - Promote docs/FORMAT.md as primary spec reference; audit error messages; + document the signature scheme fully - Release: - - Finalize module path, bump version constant, SemVer --version - output, tag v1.0.0 + - Finalize module path, bump version constant, SemVer --version output, tag + v1.0.0 diff --git a/package.json b/package.json new file mode 100644 index 0000000..cca61f6 --- /dev/null +++ b/package.json @@ -0,0 +1,10 @@ +{ + "name": "mfer", + "version": "0.1.0", + "private": true, + "description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.", + "license": "WTFPL", + "devDependencies": { + "prettier": "3.9.6" + } +} diff --git a/script/bootstrap b/script/bootstrap index c40c5e5..2f6ef3e 100755 --- a/script/bootstrap +++ b/script/bootstrap @@ -130,9 +130,13 @@ main() { if missing make; then pkg_install gnumake make make make; fi # ---- JS / docs repos ---- - # ensure_node - # ensure_yarn - # install_js_deps + # This is a Go repo, but node and yarn are required anyway: prettier + # formats the Markdown and JSON, and script/fmt-check verifies it. + # The version is pinned by package.json/yarn.lock, whose integrity + # hashes --frozen-lockfile enforces. + ensure_node + ensure_yarn + install_js_deps # ---- Go repos ---- if missing go; then pkg_install go golang go go; fi diff --git a/script/fmt b/script/fmt index 8607e7f..d4d25c8 100755 --- a/script/fmt +++ b/script/fmt @@ -2,7 +2,8 @@ # script/fmt: format all files (writes). set -eu -ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" +ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" # Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale # (mirrors the old Makefile prerequisite; the generated file is @@ -19,9 +20,8 @@ main() { ensure_pb gofumpt -l -w mfer internal cmd golangci-lint run --fix - # prettier is best-effort, as in the old Makefile (- prefix) - prettier -w *.json || true - prettier -w *.md || true + # Markdown and JSON, over the same file set script/fmt-check verifies. + "$SCRIPT_DIR/prettier" --write } main "$@" diff --git a/script/fmt-check b/script/fmt-check index 6c73704..1bd6882 100755 --- a/script/fmt-check +++ b/script/fmt-check @@ -1,28 +1,14 @@ #!/bin/sh # script/fmt-check: check formatting (read-only). Same scope as -# script/fmt, but fails instead of writing. +# script/fmt, but fails instead of writing: Go via script/fmt-check-go, +# Markdown and JSON via script/prettier. set -eu -ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" - -# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale -# (mirrors the old Makefile prerequisite; the generated file is -# committed, so this is normally a no-op). -ensure_pb() { - if [ ! -f mfer/mf.pb.go ] || - [ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then - (cd mfer && go generate .) - fi -} +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" main() { - cd "$ROOT" - ensure_pb - if [ -n "$(gofmt -l .)" ]; then - echo "gofmt: files need formatting:" >&2 - gofmt -l . >&2 - exit 1 - fi + "$SCRIPT_DIR/fmt-check-go" + "$SCRIPT_DIR/prettier" --check } main "$@" diff --git a/script/fmt-check-go b/script/fmt-check-go new file mode 100755 index 0000000..c079f59 --- /dev/null +++ b/script/fmt-check-go @@ -0,0 +1,29 @@ +#!/bin/sh +# script/fmt-check-go: check Go formatting (read-only). Split out from +# script/fmt-check so the Docker lint stage, whose image has no node and +# therefore no prettier, can run the Go half on its own. +set -eu + +ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" + +# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale +# (mirrors the old Makefile prerequisite; the generated file is +# committed, so this is normally a no-op). +ensure_pb() { + if [ ! -f mfer/mf.pb.go ] || + [ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then + (cd mfer && go generate .) + fi +} + +main() { + cd "$ROOT" + ensure_pb + if [ -n "$(gofmt -l .)" ]; then + echo "gofmt: files need formatting:" >&2 + gofmt -l . >&2 + exit 1 + fi +} + +main "$@" diff --git a/script/prettier b/script/prettier new file mode 100755 index 0000000..b2f66a3 --- /dev/null +++ b/script/prettier @@ -0,0 +1,69 @@ +#!/bin/sh +# script/prettier: run prettier over this repo's canonical file set. +# +# Takes exactly one mode argument, --write or --check, and applies the +# same patterns in both modes. script/fmt and script/fmt-check both go +# through here, so the set of files that get formatted and the set that +# get verified cannot drift apart. +# +# Failures are never swallowed: a missing prettier is an error, not a +# silent skip. A formatter that quietly does nothing is worse than one +# that fails loudly. +set -eu + +ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" + +usage() { + echo "usage: script/prettier --write|--check" >&2 + exit 2 +} + +# Prefer the version pinned by package.json/yarn.lock so that CI and +# developer machines format identically. Fall back to a prettier on PATH, +# but say so, because a different version formats differently. +find_prettier() { + if [ -x "$ROOT/node_modules/.bin/prettier" ]; then + printf '%s\n' "$ROOT/node_modules/.bin/prettier" + return 0 + fi + if command -v prettier >/dev/null 2>&1; then + echo "prettier: node_modules/.bin/prettier is absent; using the" \ + "prettier on PATH, which may be a different version than the" \ + "one pinned in package.json. Run script/bootstrap to install" \ + "the pinned version." >&2 + command -v prettier + return 0 + fi + return 1 +} + +main() { + [ "$#" -eq 1 ] || usage + case "$1" in + --write | --check) mode="$1" ;; + *) usage ;; + esac + + cd "$ROOT" + + if ! prettier_bin="$(find_prettier)"; then + echo "prettier: not found." >&2 + echo " Install it with: script/bootstrap" >&2 + echo " (installs the version pinned in package.json/yarn.lock)" >&2 + exit 1 + fi + + # Markdown and JSON, repo-wide rather than root-only, so files in + # subdirectories (docs/, once it exists) are covered too. Exclusions + # live in .prettierignore; REPO_POLICIES.md is excluded there because + # it is a verbatim copy of an upstream document. + # + # --no-error-on-unmatched-pattern is deliberately NOT used: both + # patterns always match at least one tracked file (README.md, + # package.json), so an empty match means the glob broke, and prettier + # erroring out is exactly what we want rather than a vacuous pass. + "$prettier_bin" "$mode" "**/*.md" + "$prettier_bin" "$mode" "**/*.json" +} + +main "$@" diff --git a/yarn.lock b/yarn.lock new file mode 100644 index 0000000..8f3c21a --- /dev/null +++ b/yarn.lock @@ -0,0 +1,8 @@ +# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY. +# yarn lockfile v1 + + +prettier@3.9.6: + version "3.9.6" + resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.9.6.tgz#b3ea5146515d40fc53f18aa63f74dfab1e10dbf6" + integrity sha512-OpN0zzVdiaiAhxpuuj5efpIS4sY9j7bY6uR5mnj5yPzGkdkjNKSJeUThPb60Jw29QuAZgA4o+/iB49kFiaBX6g== -- 2.49.1