sneak/mfer
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Releases Wiki Activity

Compare commits

base: sneak/mfer:ef56eeeae09650f6ca13baf1f4079d801bfa00a2
Branches Tags
sneak/mfer:main sneak/mfer:next sneak/mfer:issue-116-gomodguard
...
compare: sneak/mfer:next
Branches Tags
sneak/mfer:next sneak/mfer:issue-116-gomodguard sneak/mfer:main
3 Commits
ef56eeeae0 ... next
Author SHA1 Message Date
clawbot ce024baaed Move the CLI to urfave/cli v3 (closes #110)
check / check (push) Failing after 3s
Details 
Ported per the library's v2-to-v3 migration guide: the app is a root
cli.Command, actions take a context and the command, and flag
environment variables become value sources. -v, -q and the version flag
are local so, as before, only the commands defining them accept them.
Every command stops reading flags at its first argument, as v2 did.
ErrWriter is stdout so usage errors print with their help. The action's
context reaches the manifest download in check, export and list.
testify rises to v1.12.1, which v3 requires; the urfave_cli_no_docs
build tag, which v3 lacks, is dropped. v3 accepts a flag given under two
names, so -v --verbose gives debug output, and the test pinning the
refusal becomes a TestVerboseCount case.

Model: opus-5-5
 2026-10-04 22:02:12 +02:00
clawbot ab72692439 Pin the remaining developer tool installs (closes #68)
check / check (push) Failing after 3s
Details 
gofumpt and protoc-gen-go are tools of a separate Go module in bin/tools,
so `go tool` builds them from source checked against bin/tools/go.sum and
mfer's own module gains no dependencies. script/bootstrap downloads that
module, and unpacks protoc 33.4 into bin/protoc from its release archive
after checking the sha256 it holds for the platform. script/generate runs
that protoc with the pinned plugin, so mfer/mf.go and its go:generate line
go. script/prettier runs only the node_modules prettier, fails when it
differs from the package.json pin, finds the node bootstrap installed
through nvm by the version in .nvmrc, and runs prettier once. Comment and
package.json fixes.

Model: opus-5-5
 2026-10-04 20:48:51 +02:00
clawbot 9bb0ab3a03 fetch refuses a manifest that lists another file's temp name (closes #151)
check / check (push) Failing after 2s
Details 
fetch downloads each file to a temp name beside it and first removes
whatever is there. A manifest listing both a.txt and .a.txt.tmp had
fetch delete the second while fetching the first, then exit 0 with a
tree check rejects.

The refusal of a manifest that lists the saved manifest's own name or
temp name now covers this too: a listed file, or a directory a listed
file is in, may not sit at any name fetch writes besides the listed
files themselves. Temp names come from tempPathFor, names are compared
ignoring case as before, and the refusal still happens before the
destination is created or any file requested.

Model: opus-5-5
 2026-10-04 20:02:17 +02:00
27 changed files with 575 additions and 376 deletions
Expand all files Collapse all files
.dockerignore
+3
View File
@@ -61,3 +61,6 @@
# This repo's own host-built binary (make build). # This repo's own host-built binary (make build).
/bin/mfer /bin/mfer
# The protoc script/bootstrap unpacks for script/generate.
/bin/protoc
.gitignore
+1
View File
@@ -1,4 +1,5 @@
/bin/mfer /bin/mfer
/bin/protoc/
/tmp /tmp
/node_modules/ /node_modules/
.nvmrc
+1
View File
@@ -0,0 +1 @@
22.17.0
Dockerfile
+4 -4
View File
@@ -10,7 +10,7 @@ COPY . .
# Go half of fmt-check only: this image has no node, so no prettier. The # Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below. The image has no gofumpt # markdown half runs in the mdfmt stage below. The image has no gofumpt
# either; script/gofumpt builds the version it pins with `go run`. # either; script/gofumpt builds the version bin/tools/go.mod pins.
RUN script/gofumpt --check RUN script/gofumpt --check
# The linter directly, not `make lint`: script/lint builds this stage, and # The linter directly, not `make lint`: script/lint builds this stage, and
# there is no docker inside this build. # there is no docker inside this build.
@@ -18,7 +18,7 @@ RUN golangci-lint run --config .golangci.yml ./...
# Markdown/JSON format stage — prettier needs node, which the Go images # Markdown/JSON format stage — prettier needs node, which the Go images
# do not have. node:22.17.0-bookworm-slim (2026-08-09); ships node # do not have. node:22.17.0-bookworm-slim (2026-08-09); ships node
# 22.17.0 and yarn 1.22.22, the versions script/bootstrap pins. # 22.17.0 and yarn 1.22.22, the versions .nvmrc and script/bootstrap pin.
FROM node@sha256:b04ce4ae4e95b522112c2e5c52f781471a5cbc3b594527bcddedee9bc48c03a0 AS mdfmt FROM node@sha256:b04ce4ae4e95b522112c2e5c52f781471a5cbc3b594527bcddedee9bc48c03a0 AS mdfmt
WORKDIR /src WORKDIR /src
@@ -31,7 +31,7 @@ COPY . .
RUN script/prettier --check RUN script/prettier --check
# Build stage — tests and compilation # Build stage — tests and compilation
# golang:1.23 (2026-03-14) # golang:1.23.12, 2026-03-14
FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder
# Force BuildKit to run the lint and mdfmt stages by creating stage dependencies # Force BuildKit to run the lint and mdfmt stages by creating stage dependencies
@@ -64,7 +64,7 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
exit 1; \ exit 1; \
fi; \ fi; \
cd cmd/mfer && \ cd cmd/mfer && \
CGO_ENABLED=0 go build -tags urfave_cli_no_docs -ldflags "-X main.Gitrev=$version" -o /mfer . CGO_ENABLED=0 go build -ldflags "-X main.Gitrev=$version" -o /mfer .
# Fail unless /mfer is statically linked: scratch has no C library to run it. # Fail unless /mfer is statically linked: scratch has no C library to run it.
RUN ldd /mfer 2>&1 | grep -q 'not a dynamic executable' RUN ldd /mfer 2>&1 | grep -q 'not a dynamic executable'
README.md
+24 -18
View File
@@ -67,9 +67,13 @@ standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. We development workflow, and the Makefile targets are thin shims that call them. We
provide: provide:
- `script/bootstrap` — install all dependencies (Go, Go module download, and - `script/bootstrap` — install all dependencies, idempotently: Go and the
node/yarn plus the prettier version pinned in `package.json`/`yarn.lock`), modules of both `go.mod` and `bin/tools/go.mod`; node (the version `.nvmrc`
idempotently; golangci-lint is not installed, it runs only in Docker names, through nvm when there is no node on `PATH`) and yarn, plus the
prettier version pinned in `package.json`/`yarn.lock`; and `protoc` 33.4,
unpacked into `bin/protoc` from its release archive once the archive matches
the sha256 the script holds for this platform. golangci-lint is not installed,
it runs only in Docker
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit` `script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (`mfer`); used by other scripts - `script/projectname` — output the project name (`mfer`); used by other scripts
@@ -82,14 +86,10 @@ provide:
- `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from - `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from
`mfer/mf.proto` and record the hash of that `mfer/mf.proto` in `mfer/mf.proto` and record the hash of that `mfer/mf.proto` in
`mfer/mf.proto.sha256`; the only thing that regenerates the committed `mfer/mf.proto.sha256`; the only thing that regenerates the committed
`mfer/mf.pb.go`. It needs the exact versions that wrote the committed file, `mfer/mf.pb.go`. It runs the `protoc` that `script/bootstrap` unpacks into
and refuses to run with any other: `protoc` 33.4 (unpack `bin/protoc`, refusing any version but 33.4, and the `protoc-gen-go` that
`protoc-33.4-<platform>.zip` from `bin/tools/go.mod` pins, which `go tool` builds from source checked against
[its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4) the hashes in `bin/tools/go.sum`
and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11
(`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which
installs it in `$(go env GOPATH)/bin`; `script/generate` adds that directory
to `PATH`)
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand - `script/fuzz` — fuzz the manifest parser for one minute; run by hand
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus (`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
as ordinary tests as ordinary tests
@@ -99,14 +99,17 @@ provide:
- `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and - `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and
`script/prettier --write` `script/prettier --write`
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the - `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
given mode, `--write` or `--check`, at the one version it pins (built on given mode, `--write` or `--check`, at the version `bin/tools/go.mod` pins
demand by `go run`, so nothing installs it); `script/fmt`, `script/fmt-check` (built on demand by `go tool` from source checked against the hashes in
`bin/tools/go.sum`, so nothing installs it); `script/fmt`, `script/fmt-check`
and the Docker lint stage all go through it, so they cannot disagree about Go and the Docker lint stage all go through it, so they cannot disagree about Go
formatting formatting
- `script/prettier` — run prettier over the repository's canonical file set - `script/prettier` — run prettier over the repository's canonical file set
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or (Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
`--check`; the single definition of that file set, so `script/fmt` and `--check`, with the prettier version `yarn.lock` pins, run by the node on
`script/fmt-check` cannot disagree about it `PATH` or else the one `script/bootstrap` installed through nvm; the single
definition of that file set, so `script/fmt` and `script/fmt-check` cannot
disagree about it
- `script/fmt-check` — check formatting without writing: - `script/fmt-check` — check formatting without writing:
`script/gofumpt --check` plus `script/prettier --check` `script/gofumpt --check` plus `script/prettier --check`
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check` - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
@@ -268,9 +271,12 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
cryptographic integrity of downloaded files. A file already there with the cryptographic integrity of downloaded files. A file already there with the
size and hash the manifest lists is skipped. Once every file is in place, size and hash the manifest lists is skipped. Once every file is in place,
the manifest is saved there as `index.mf`, so `mfer check` can verify the the manifest is saved there as `index.mf`, so `mfer check` can verify the
tree later. A manifest that lists `index.mf` (in any letter case) or tree later. Each file is downloaded to a temp file beside it, such as
`.index.mf.tmp` at the top of the tree is refused before any file is `.a.txt.tmp` for `a.txt`, then moved into place. A manifest is refused
downloaded, since saving the manifest would replace it. before any file is downloaded if it lists a file where fetch writes
another: at the temp file of a listed file, or at `index.mf` or
`.index.mf.tmp` at the top of the tree. Names are compared in any letter
case.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/` - `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
- as above, but first refuses a manifest not signed by the key with that - as above, but first refuses a manifest not signed by the key with that
fingerprint, as `mfer check --require-signature` does, before downloading fingerprint, as `mfer check --require-signature` does, before downloading
bin/tools/go.mod
+22
View File
@@ -0,0 +1,22 @@
// The developer tools this repo runs with `go tool`: gofumpt for
// script/gofumpt and protoc-gen-go for script/generate. Kept out of the mfer
// module so they add nothing to what mfer's users download. `go tool` builds
// exactly the source whose hashes go.sum here records.
module sneak.berlin/go/mfer/bin/tools
go 1.26.0
tool (
google.golang.org/protobuf/cmd/protoc-gen-go
mvdan.cc/gofumpt
)
require (
golang.org/x/mod v0.40.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/tools v0.49.0 // indirect
// protoc-gen-go v1.36.11, 2026-10-04
google.golang.org/protobuf v1.36.11 // indirect
// gofumpt v0.12.0, 2026-10-04
mvdan.cc/gofumpt v0.12.0 // indirect
)
bin/tools/go.sum
+22
View File
@@ -0,0 +1,22 @@
github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474=
github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g=
github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
mvdan.cc/gofumpt v0.12.0 h1:1Lbudkz2kpM9Cjz2pL4M19u7q+GaEhCTNf7N9mfpcho=
mvdan.cc/gofumpt v0.12.0/go.mod h1:SmBHHrljiZu/uoypeKup3rFzP6eoC9UwCp2iH5E3jZA=
go.mod
+3 -9
View File
@@ -9,27 +9,21 @@ require (
github.com/klauspost/compress v1.18.2 github.com/klauspost/compress v1.18.2
github.com/multiformats/go-multihash v0.2.3 github.com/multiformats/go-multihash v0.2.3
github.com/spf13/afero v1.8.0 github.com/spf13/afero v1.8.0
github.com/stretchr/testify v1.8.1 github.com/stretchr/testify v1.12.1
github.com/urfave/cli/v2 v2.27.7 github.com/urfave/cli/v3 v3.14.0
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
google.golang.org/protobuf v1.28.1 google.golang.org/protobuf v1.28.1
) )
require ( require (
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/kr/pretty v0.2.0 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-varint v0.0.6 // indirect github.com/multiformats/go-varint v0.0.6 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
golang.org/x/sys v0.1.0 // indirect golang.org/x/sys v0.1.0 // indirect
golang.org/x/text v0.3.6 // indirect golang.org/x/text v0.3.6 // indirect
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.6 // indirect lukechampine.com/blake3 v1.1.6 // indirect
) )
go.sum
+6 -22
View File
@@ -46,8 +46,6 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk= github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cpuguy83/go-md2man/v2 v2.0.7 h1:zbFlGlXEAKlwXpmvle3d8Oe3YnkKIK4xSRTd3sHPnBo=
github.com/cpuguy83/go-md2man/v2 v2.0.7/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -134,10 +132,7 @@ github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBF
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg= github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g= github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM= github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
@@ -149,30 +144,21 @@ github.com/multiformats/go-varint v0.0.6 h1:gk85QWKxh3TazbLxED/NlDVv8+q+ReFJk7Y2
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg= github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4= github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI= github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA= github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.8.0 h1:5MmtuhAgYeU6qpa7w7bP0dv6MBYuup0vekhSpSkoq60= github.com/spf13/afero v1.8.0 h1:5MmtuhAgYeU6qpa7w7bP0dv6MBYuup0vekhSpSkoq60=
github.com/spf13/afero v1.8.0/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfAqwo= github.com/spf13/afero v1.8.0/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfAqwo=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= github.com/urfave/cli/v3 v3.14.0 h1:a8414NQlHJs0c/iBsulKLzlES0n/lEAskbL2LKpU4/s=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/urfave/cli/v3 v3.14.0/go.mod h1:vXn6HxPNccJSzQr2QvwVncOKrgYGIHU0HY5h8B2nQj4=
github.com/urfave/cli/v2 v2.27.7 h1:bH59vdhbjLv3LAvIu6gd0usJHgoTTPhCFib8qqOwXYU=
github.com/urfave/cli/v2 v2.27.7/go.mod h1:CyNAG/xg+iAOg0N4MPGZqVmv2rCoP267496AOXUZjA4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -183,6 +169,8 @@ go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw= go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -470,13 +458,9 @@ google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI= gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
internal/cli/check.go
+26 -17
View File
@@ -16,7 +16,7 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -96,8 +96,10 @@ func findManifest(fs afero.Fs, dir string) (string, error) {
// fetchManifestToTemp downloads a manifest URL to a temporary file and // fetchManifestToTemp downloads a manifest URL to a temporary file and
// returns the temp file path. The caller is responsible for removing it. // returns the temp file path. The caller is responsible for removing it.
func (mfa *CLIApp) fetchManifestToTemp(url string) (string, error) { func (mfa *CLIApp) fetchManifestToTemp(
rc, fetchErr := mfa.openManifestReader(url) ctx context.Context, url string,
) (string, error) {
rc, fetchErr := mfa.openManifestReader(ctx, url)
if fetchErr != nil { if fetchErr != nil {
return "", fetchErr return "", fetchErr
} }
@@ -208,13 +210,15 @@ func countCheckFailures(
// findExtraFiles reports files present on disk but absent from the // findExtraFiles reports files present on disk but absent from the
// manifest, and anything the search cannot read: each is a failure under // manifest, and anything the search cannot read: each is a failure under
// --no-extra-files, otherwise a warning. // --no-extra-files, otherwise a warning.
func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error { func findExtraFiles(
ctx context.Context, cmd *cli.Command, chk *mfer.Checker, failures *int64,
) error {
extraResults := make(chan mfer.Result, 1) extraResults := make(chan mfer.Result, 1)
extraDone := make(chan struct{}) extraDone := make(chan struct{})
go func() { go func() {
for result := range extraResults { for result := range extraResults {
if ctx.Bool("no-extra-files") { if cmd.Bool("no-extra-files") {
*failures++ *failures++
log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message) log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message)
@@ -226,7 +230,7 @@ func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error
close(extraDone) close(extraDone)
}() }()
err := chk.FindExtraFiles(ctx.Context, extraResults) err := chk.FindExtraFiles(ctx, extraResults)
if err != nil { if err != nil {
return fmt.Errorf("failed to check for extra files: %w", err) return fmt.Errorf("failed to check for extra files: %w", err)
} }
@@ -238,7 +242,9 @@ func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error
// runCheck runs the manifest check with progress and result reporting // runCheck runs the manifest check with progress and result reporting
// and returns the number of failures. // and returns the number of failures.
func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, error) { func runCheck(
ctx context.Context, cmd *cli.Command, chk *mfer.Checker, showProgress bool,
) (int64, error) {
// Set up results channel // Set up results channel
results := make(chan mfer.Result, 1) results := make(chan mfer.Result, 1)
@@ -264,7 +270,7 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
go countCheckFailures(results, &failures, done) go countCheckFailures(results, &failures, done)
// Run check // Run check
err := chk.Check(ctx.Context, results, progress) err := chk.Check(ctx, results, progress)
progressWg.Wait() progressWg.Wait()
@@ -275,7 +281,7 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
// Wait for results processing to complete // Wait for results processing to complete
<-done <-done
err = findExtraFiles(ctx, chk, &failures) err = findExtraFiles(ctx, cmd, chk, &failures)
if err != nil { if err != nil {
return 0, err return 0, err
} }
@@ -283,17 +289,19 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
return failures, nil return failures, nil
} }
func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) checkManifestOperation(
ctx context.Context, cmd *cli.Command,
) error {
log.Debug("checkManifestOperation()") log.Debug("checkManifestOperation()")
manifestPath, err := mfa.resolveManifestArg(ctx) manifestPath, err := mfa.resolveManifestArg(cmd)
if err != nil { if err != nil {
return fmt.Errorf("check: %w", err) return fmt.Errorf("check: %w", err)
} }
// URL manifests need to be downloaded to a temp file for the checker // URL manifests need to be downloaded to a temp file for the checker
if isHTTPURL(manifestPath) { if isHTTPURL(manifestPath) {
tmpPath, tmpErr := mfa.fetchManifestToTemp(manifestPath) tmpPath, tmpErr := mfa.fetchManifestToTemp(ctx, manifestPath)
if tmpErr != nil { if tmpErr != nil {
return fmt.Errorf("check: %w", tmpErr) return fmt.Errorf("check: %w", tmpErr)
} }
@@ -303,12 +311,13 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
manifestPath = tmpPath manifestPath = tmpPath
} }
basePath := ctx.String("base") basePath := cmd.String("base")
showProgress := ctx.Bool("progress") showProgress := cmd.Bool("progress")
log.Infof("checking manifest %s with base %s", manifestPath, basePath) log.Infof("checking manifest %s with base %s", manifestPath, basePath)
// Create checker // Create checker
//nolint:contextcheck // mfer loads a manifest without a context
chk, err := mfer.NewChecker(&mfer.CheckerOptions{ chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath, ManifestPath: manifestPath,
BasePath: basePath, BasePath: basePath,
@@ -319,9 +328,9 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
} }
// Check signature requirement // Check signature requirement
requiredSigner := ctx.String(flagRequireSignature) requiredSigner := cmd.String(flagRequireSignature)
if requiredSigner != "" { if requiredSigner != "" {
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner) err = verifyRequiredSigner(ctx, chk, requiredSigner)
if err != nil { if err != nil {
return err return err
} }
@@ -330,7 +339,7 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
log.Infof("manifest contains %d files, %s", chk.FileCount(), log.Infof("manifest contains %d files, %s", chk.FileCount(),
humanize.IBytes(safeUint64(int64(chk.TotalBytes())))) humanize.IBytes(safeUint64(int64(chk.TotalBytes()))))
failures, err := runCheck(ctx, chk, showProgress) failures, err := runCheck(ctx, cmd, chk, showProgress)
if err != nil { if err != nil {
return err return err
} }
internal/cli/entry_test.go
+21 -25
View File
@@ -18,7 +18,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2" urfcli "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -264,8 +264,9 @@ func commandsTakingVerbose() [][]string {
} }
// TestVerboseCount asserts that one -v or --verbose gives verbose output and // TestVerboseCount asserts that one -v or --verbose gives verbose output and
// two -v give debug output (issue #125). urfave/cli before v2.25.5 counted a // two -v, or -v and --verbose, give debug output (issue #125). urfave/cli
// flag given by its alias twice, so one -v gave debug output. // before v2.25.5 counted a flag given by its alias twice, so one -v gave debug
// output.
func TestVerboseCount(t *testing.T) { func TestVerboseCount(t *testing.T) {
t.Parallel() t.Parallel()
@@ -276,6 +277,7 @@ func TestVerboseCount(t *testing.T) {
{[]string{"-v"}, log.VerboseLevel}, {[]string{"-v"}, log.VerboseLevel},
{[]string{testFlagVerbose}, log.VerboseLevel}, {[]string{testFlagVerbose}, log.VerboseLevel},
{[]string{"-v", "-v"}, log.DebugLevel}, {[]string{"-v", "-v"}, log.DebugLevel},
{[]string{"-v", testFlagVerbose}, log.DebugLevel},
} }
for _, command := range commandsTakingVerbose() { for _, command := range commandsTakingVerbose() {
@@ -313,28 +315,6 @@ func TestCombinedShortVerboseRefused(t *testing.T) {
} }
} }
// TestShortAndLongVerboseRefused asserts that -v and --verbose given together
// are refused (issue #125): urfave/cli v2 refuses a flag given under two of its
// names, and one flag with an alias keeps help and parsing simple.
func TestShortAndLongVerboseRefused(t *testing.T) {
t.Parallel()
for _, command := range commandsTakingVerbose() {
args := slices.Concat(command, []string{"-v", testFlagVerbose})
t.Run(strings.Join(args, " "), func(t *testing.T) {
t.Parallel()
opts := testOpts(args, afero.NewMemMapFs())
exitCode, level := runCLIWithLevel(opts)
assert.Equal(t, 1, exitCode)
assert.Contains(t, testStderr(t, opts), "Cannot use two forms of the same flag")
assert.Equal(t, log.InfoLevel, level)
})
}
}
func TestHelpCommand(t *testing.T) { func TestHelpCommand(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1190,6 +1170,22 @@ func TestGenerateValidatesInputPaths(t *testing.T) {
}) })
} }
// TestFlagAfterArgumentIsArgument asserts that flags are read only before a
// command's first argument: after it, -v is a path, not the verbose flag.
func TestFlagAfterArgumentIsArgument(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "content")
opts := testOpts([]string{testApp, cmdGenerate, testDir, "-v"}, fs)
exitCode := runCLI(opts)
assert.Equal(t, 1, exitCode)
assert.Contains(t, testStderr(t, opts), "path does not exist: -v")
}
func TestCheckDetectsManifestCorruption(t *testing.T) { func TestCheckDetectsManifestCorruption(t *testing.T) {
t.Parallel() t.Parallel()
internal/cli/errmsg_test.go
+33 -31
View File
@@ -4,7 +4,6 @@ package cli
import ( import (
"bytes" "bytes"
"context" "context"
"flag"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os" "os"
@@ -15,7 +14,7 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2" urfcli "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -195,13 +194,17 @@ func signedChecker(t *testing.T, manifest []byte) *mfer.Checker {
func TestPathDoesNotExistMessage(t *testing.T) { func TestPathDoesNotExistMessage(t *testing.T) {
t.Parallel() t.Parallel()
set := flag.NewFlagSet("gen", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{"nope"}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()} mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil) cmd := &urfcli.Command{
Name: cmdGenerate,
Action: func(_ context.Context, c *urfcli.Command) error {
_, err := mfa.collectInputPaths(c.Args())
_, err := mfa.collectInputPaths(ctx.Args()) return err
},
}
err := cmd.Run(context.Background(), []string{cmdGenerate, "nope"})
require.ErrorIs(t, err, errPathNotExist) require.ErrorIs(t, err, errPathNotExist)
assert.EqualError(t, err, "path does not exist: nope") assert.EqualError(t, err, "path does not exist: nope")
} }
@@ -214,18 +217,22 @@ func TestOutputFileExistsMessage(t *testing.T) {
require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/d/f.txt", []byte("hi"), 0o644))
require.NoError(t, afero.WriteFile(fs, "/out.mf", []byte("old"), 0o644)) require.NoError(t, afero.WriteFile(fs, "/out.mf", []byte("old"), 0o644))
set := flag.NewFlagSet("gen", flag.ContinueOnError)
set.String("output", "", "")
set.Bool("force", false, "")
require.NoError(t, set.Parse([]string{"/d"}))
require.NoError(t, set.Set("output", "/out.mf"))
mfa := &CLIApp{Fs: fs} mfa := &CLIApp{Fs: fs}
ctx := urfcli.NewContext(nil, set, nil) cmd := &urfcli.Command{
Name: cmdGenerate,
Flags: []urfcli.Flag{
&urfcli.StringFlag{Name: "output"},
&urfcli.BoolFlag{Name: "force"},
},
Action: mfa.generateManifestOperation,
}
// generateManifestOperation writes to the process-global logger during // generateManifestOperation writes to the process-global logger during
// enumeration, so serialize with the other CLI runs. // enumeration, so serialize with the other CLI runs.
err := runLocked(func() error { return mfa.generateManifestOperation(ctx) }) err := runLocked(func() error {
return cmd.Run(context.Background(),
[]string{cmdGenerate, "--output", "/out.mf", "/d"})
})
require.ErrorIs(t, err, errOutputExists) require.ErrorIs(t, err, errOutputExists)
assert.EqualError(t, err, assert.EqualError(t, err,
"output file /out.mf already exists (use --force to overwrite)") "output file /out.mf already exists (use --force to overwrite)")
@@ -250,7 +257,7 @@ func TestUnknownCommandMessage(t *testing.T) {
err := runLocked(func() error { err := runLocked(func() error {
mfa.run([]string{testApp}) mfa.run([]string{testApp})
return mfa.app.Run([]string{testApp, "bogus"}) return mfa.app.Run(context.Background(), []string{testApp, "bogus"})
}) })
require.ErrorIs(t, err, errUnknownCommand) require.ErrorIs(t, err, errUnknownCommand)
assert.EqualError(t, err, `unknown command "bogus"`) assert.EqualError(t, err, `unknown command "bogus"`)
@@ -267,7 +274,7 @@ func TestManifestLoaderHTTPStatusMessage(t *testing.T) {
mfa := &CLIApp{Fs: afero.NewMemMapFs()} mfa := &CLIApp{Fs: afero.NewMemMapFs()}
_, err := mfa.openManifestReader(server.URL + "/foo.mf") _, err := mfa.openManifestReader(context.Background(), server.URL+"/foo.mf")
require.ErrorIs(t, err, errHTTPStatus) require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, assert.EqualError(t, err,
"failed to fetch "+server.URL+"/foo.mf: HTTP 404") "failed to fetch "+server.URL+"/foo.mf: HTTP 404")
@@ -284,17 +291,13 @@ func TestFetchManifestHTTPStatusMessage(t *testing.T) {
mfa := &CLIApp{Fs: afero.NewMemMapFs()} mfa := &CLIApp{Fs: afero.NewMemMapFs()}
set := flag.NewFlagSet(cmdFetch, flag.ContinueOnError) cmd := mfa.fetchCommand()
for _, f := range mfa.fetchCommand().Flags { cmd.Action = mfa.fetchManifestOperation
require.NoError(t, f.Apply(set))
}
require.NoError(t, set.Parse([]string{server.URL}))
ctx := urfcli.NewContext(nil, set, nil)
// fetchManifestOperation logs to the process-global logger. // fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) }) err := runLocked(func() error {
return cmd.Run(context.Background(), []string{cmdFetch, server.URL})
})
require.ErrorIs(t, err, errHTTPStatus) require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "failed to fetch manifest: HTTP 404") assert.EqualError(t, err, "failed to fetch manifest: HTTP 404")
} }
@@ -320,14 +323,13 @@ func TestFetchFileHTTPStatusMessage(t *testing.T) {
func TestURLRequiredMessage(t *testing.T) { func TestURLRequiredMessage(t *testing.T) {
t.Parallel() t.Parallel()
set := flag.NewFlagSet("fetch", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()} mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil) cmd := &urfcli.Command{Name: cmdFetch, Action: mfa.fetchManifestOperation}
// fetchManifestOperation logs to the process-global logger. // fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(ctx) }) err := runLocked(func() error {
return cmd.Run(context.Background(), []string{cmdFetch})
})
require.ErrorIs(t, err, errURLRequired) require.ErrorIs(t, err, errURLRequired)
assert.EqualError(t, err, "URL argument required") assert.EqualError(t, err, "URL argument required")
} }
internal/cli/export.go
+8 -4
View File
@@ -1,12 +1,13 @@
package cli package cli
import ( import (
"context"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"fmt" "fmt"
"time" "time"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -19,19 +20,22 @@ type ExportEntry struct {
Ctime *string `json:"ctime,omitempty"` Ctime *string `json:"ctime,omitempty"`
} }
func (mfa *CLIApp) exportManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) exportManifestOperation(
pathOrURL, err := mfa.resolveManifestArg(ctx) ctx context.Context, cmd *cli.Command,
) error {
pathOrURL, err := mfa.resolveManifestArg(cmd)
if err != nil { if err != nil {
return fmt.Errorf("export: %w", err) return fmt.Errorf("export: %w", err)
} }
rc, err := mfa.openManifestReader(pathOrURL) rc, err := mfa.openManifestReader(ctx, pathOrURL)
if err != nil { if err != nil {
return fmt.Errorf("export: %w", err) return fmt.Errorf("export: %w", err)
} }
defer func() { _ = rc.Close() }() defer func() { _ = rc.Close() }()
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(rc) manifest, err := mfer.NewManifestFromReader(rc)
if err != nil { if err != nil {
return fmt.Errorf("export: failed to parse manifest: %w", err) return fmt.Errorf("export: failed to parse manifest: %w", err)
internal/cli/fetch.go
+53 -31
View File
@@ -20,7 +20,7 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"github.com/multiformats/go-multihash" "github.com/multiformats/go-multihash"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -91,10 +91,10 @@ var (
// errHashMismatch indicates a downloaded file whose hash matches no // errHashMismatch indicates a downloaded file whose hash matches no
// manifest hash. // manifest hash.
errHashMismatch = errors.New("hash mismatch") errHashMismatch = errors.New("hash mismatch")
// errManifestNameListed indicates a manifest that lists a file where // errNameClash indicates a manifest that lists a file where fetch
// fetch saves the manifest. // writes another file.
errManifestNameListed = errors.New( errNameClash = errors.New(
"manifest lists a file where fetch saves the manifest") "manifest lists a file where fetch writes another file")
) )
// DownloadProgress reports the progress of a single file download. // DownloadProgress reports the progress of a single file download.
@@ -328,19 +328,21 @@ func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
return verifyDownloadedHash(h.Sum(nil), entry) == nil return verifyDownloadedHash(h.Sum(nil), entry) == nil
} }
func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) fetchManifestOperation(
ctx context.Context, cmd *cli.Command,
) error {
log.Debug("fetchManifestOperation()") log.Debug("fetchManifestOperation()")
if ctx.Args().Len() == 0 { if cmd.Args().Len() == 0 {
return errURLRequired return errURLRequired
} }
timeout := ctx.Duration(flagTimeout) timeout := cmd.Duration(flagTimeout)
if timeout <= 0 { if timeout <= 0 {
return errInvalidTimeout return errInvalidTimeout
} }
manifestURL, err := resolveManifestURL(ctx.Args().Get(0)) manifestURL, err := resolveManifestURL(cmd.Args().Get(0))
if err != nil { if err != nil {
return fmt.Errorf("invalid URL: %w", err) return fmt.Errorf("invalid URL: %w", err)
} }
@@ -350,7 +352,7 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
firstDelay: firstRetryDelay, firstDelay: firstRetryDelay,
} }
manifestData, files, err := fetchManifest(ctx, client, manifestURL) manifestData, files, err := fetchManifest(ctx, cmd, client, manifestURL)
if err != nil { if err != nil {
return err return err
} }
@@ -361,7 +363,7 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
return err return err
} }
dest := ctx.String(flagDest) dest := cmd.String(flagDest)
err = os.MkdirAll(dest, dirPerms) err = os.MkdirAll(dest, dirPerms)
if err != nil { if err != nil {
@@ -379,7 +381,7 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
// Download each file // Download each file
downloaded, downloadedBytes, dlErr := downloadManifestFiles( downloaded, downloadedBytes, dlErr := downloadManifestFiles(
ctx.Context, client, baseURL, dest, files, progress) ctx, client, baseURL, dest, files, progress)
close(progress) close(progress)
<-done <-done
@@ -412,11 +414,11 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
// fetchManifest downloads the manifest at manifestURL and parses it, // fetchManifest downloads the manifest at manifestURL and parses it,
// enforcing --require-signature if it is given and refusing a manifest // enforcing --require-signature if it is given and refusing a manifest
// that lists a file where it will be saved. It returns the manifest as // that lists a file where fetch writes another. It returns the manifest as
// downloaded, to be saved once the files are in place, and the files it // downloaded, to be saved once the files are in place, and the files it
// lists. // lists.
func fetchManifest( func fetchManifest(
ctx *cli.Context, client retryingClient, manifestURL string, ctx context.Context, cmd *cli.Command, client retryingClient, manifestURL string,
) ([]byte, []*mfer.MFFilePath, error) { ) ([]byte, []*mfer.MFFilePath, error) {
log.Infof("fetching manifest from %s", manifestURL) log.Infof("fetching manifest from %s", manifestURL)
@@ -425,7 +427,7 @@ func fetchManifest(
// manifest. // manifest.
var manifestData []byte var manifestData []byte
err := client.get(ctx.Context, manifestURL, func(resp *http.Response) error { err := client.get(ctx, manifestURL, func(resp *http.Response) error {
var readErr error var readErr error
manifestData, readErr = io.ReadAll(resp.Body) manifestData, readErr = io.ReadAll(resp.Body)
@@ -437,12 +439,13 @@ func fetchManifest(
} }
// Parse manifest // Parse manifest
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData)) manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData))
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("failed to parse manifest: %w", err) return nil, nil, fmt.Errorf("failed to parse manifest: %w", err)
} }
requiredSigner := ctx.String(flagRequireSignature) requiredSigner := cmd.String(flagRequireSignature)
if requiredSigner != "" { if requiredSigner != "" {
err = verifyFetchedSigner(ctx, manifestData, requiredSigner) err = verifyFetchedSigner(ctx, manifestData, requiredSigner)
if err != nil { if err != nil {
@@ -452,7 +455,7 @@ func fetchManifest(
files := manifest.Files() files := manifest.Files()
err = checkManifestNameUnlisted(files) err = checkNoNameClash(files)
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
@@ -462,19 +465,37 @@ func fetchManifest(
return manifestData, files, nil return manifestData, files, nil
} }
// checkManifestNameUnlisted returns an error if files lists a file or // checkNoNameClash returns an error if files lists a file, or a directory
// directory at the top of the tree under the name fetch saves the // a file is in, under a name where fetch writes another file: the temp
// manifest as, or under that name's temp file. Saving the manifest would // file it downloads a listed file to, or, at the top of the tree, the
// replace or remove it, or fail once every file was downloaded, leaving a // saved manifest or its temp file. fetch would remove or replace what is
// tree check rejects. Names are compared ignoring case, since on a // listed there, or fail partway, leaving a tree check rejects. Names are
// case-insensitive filesystem INDEX.MF and index.mf are one file. // compared ignoring case, since on a case-insensitive filesystem INDEX.MF
func checkManifestNameUnlisted(files []*mfer.MFFilePath) error { // and index.mf are one file.
for _, f := range files { func checkNoNameClash(files []*mfer.MFFilePath) error {
top, _, _ := strings.Cut(filepath.Clean(f.GetPath()), string(filepath.Separator)) sep := string(filepath.Separator)
if strings.EqualFold(top, defaultManifestName) || // written maps each name fetch writes, other than the listed files
strings.EqualFold(top, tempPathFor(defaultManifestName)) { // themselves, in lower case, to the file it writes there.
return fmt.Errorf("%w: %s", errManifestNameListed, f.GetPath()) written := map[string]string{
defaultManifestName: "the saved manifest",
tempPathFor(defaultManifestName): "the saved manifest's temp file",
}
for _, f := range files {
tmpPath := tempPathFor(filepath.Clean(f.GetPath()))
written[strings.ToLower(tmpPath)] = "the temp file for " + f.GetPath()
}
for _, f := range files {
// Look up each directory on the file's path, then the file itself.
parts := strings.Split(strings.ToLower(filepath.Clean(f.GetPath())), sep)
for i := range parts {
what, ok := written[strings.Join(parts[:i+1], sep)]
if ok {
return fmt.Errorf("%w: %s (%s)", errNameClash, f.GetPath(), what)
}
} }
} }
@@ -486,7 +507,7 @@ func checkManifestNameUnlisted(files []*mfer.MFFilePath) error {
// its manifest from a file, so the manifest is handed to it as a file in // its manifest from a file, so the manifest is handed to it as a file in
// memory. // memory.
func verifyFetchedSigner( func verifyFetchedSigner(
ctx *cli.Context, manifestData []byte, requiredSigner string, ctx context.Context, manifestData []byte, requiredSigner string,
) error { ) error {
memFs := afero.NewMemMapFs() memFs := afero.NewMemMapFs()
manifestPath := "/" + defaultManifestName manifestPath := "/" + defaultManifestName
@@ -496,6 +517,7 @@ func verifyFetchedSigner(
return err return err
} }
//nolint:contextcheck // mfer loads a manifest without a context
chk, err := mfer.NewChecker(&mfer.CheckerOptions{ chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath, ManifestPath: manifestPath,
BasePath: "/", BasePath: "/",
@@ -505,7 +527,7 @@ func verifyFetchedSigner(
return fmt.Errorf("failed to load manifest: %w", err) return fmt.Errorf("failed to load manifest: %w", err)
} }
return verifyRequiredSigner(ctx.Context, chk, requiredSigner) return verifyRequiredSigner(ctx, chk, requiredSigner)
} }
// saveManifest writes the fetched manifest into dest under the default // saveManifest writes the fetched manifest into dest under the default
internal/cli/fetch_test.go
+83 -24
View File
@@ -4,7 +4,6 @@ package cli
import ( import (
"bytes" "bytes"
"context" "context"
"flag"
"fmt" "fmt"
"io" "io"
"maps" "maps"
@@ -23,7 +22,6 @@ import (
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -1176,23 +1174,89 @@ func TestFetchRefusesListedManifestName(t *testing.T) {
t.Run(listed, func(t *testing.T) { t.Run(listed, func(t *testing.T) {
t.Parallel() t.Parallel()
// Built directly rather than scanned, since a scan lists no files := map[string][]byte{listed: []byte("listed")}
// hidden files and never a path starting with "./".
content := []byte("listed")
builder := mfer.NewBuilder()
_, err := builder.AddFile(mfer.RelFilePath(listed), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
require.NoError(t, err)
var manifest bytes.Buffer assertFetchRefused(t, builtManifest(t, files), files,
require.NoError(t, builder.Build(context.Background(), &manifest)) "manifest lists a file where fetch writes another file: "+listed)
assertFetchRefused(t, manifest.Bytes(), map[string][]byte{listed: content},
"manifest lists a file where fetch saves the manifest: "+listed)
}) })
} }
} }
// TestFetchRefusesListedTempName fetches manifests that list a.txt and
// .a.txt.tmp, the temp file fetch downloads a.txt to, at the top of the
// tree and in a directory. Downloading a.txt would remove .a.txt.tmp, so
// fetch must refuse the manifest before it creates the destination or
// requests any file. README with .README.tmp checks that temp names are
// compared ignoring case. A manifest that lists only one of the two is
// fetched in full.
func TestFetchRefusesListedTempName(t *testing.T) {
t.Parallel()
for _, dir := range []string{"", "sub/"} {
for file, tmp := range map[string]string{
dir + "a.txt": dir + ".a.txt.tmp",
dir + "README": dir + ".README.tmp",
} {
both := map[string][]byte{
file: []byte("a file"),
tmp: []byte("a file at its temp name"),
}
t.Run(file+" and "+tmp, func(t *testing.T) {
t.Parallel()
assertFetchRefused(t, builtManifest(t, both), both,
"manifest lists a file where fetch writes another file: "+
tmp+" (the temp file for "+file+")")
})
for listed, content := range both {
t.Run("only "+listed, func(t *testing.T) {
t.Parallel()
files := map[string][]byte{listed: content}
manifest := builtManifest(t, files)
server := httptest.NewServer(fetchTestHandler(manifest, files))
defer server.Close()
dest := t.TempDir()
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
want := maps.Clone(files)
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
})
}
}
}
}
// builtManifest returns a manifest of files, built directly rather than
// scanned, since a scan lists no hidden files and never a path starting
// with "./".
func builtManifest(t *testing.T, files map[string][]byte) []byte {
t.Helper()
builder := mfer.NewBuilder()
for p, content := range files {
_, err := builder.AddFile(mfer.RelFilePath(p), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
require.NoError(t, err)
}
var manifest bytes.Buffer
require.NoError(t, builder.Build(context.Background(), &manifest))
return manifest.Bytes()
}
// assertFetchRefused serves manifest, a manifest of files, and fetches it // assertFetchRefused serves manifest, a manifest of files, and fetches it
// with flags into a directory that does not exist yet. fetch must fail // with flags into a directory that does not exist yet. fetch must fail
// with message after requesting only the manifest, and must not create // with message after requesting only the manifest, and must not create
@@ -1256,18 +1320,13 @@ func TestFetchTimeoutFlag(t *testing.T) {
mfa := &CLIApp{Fs: afero.NewMemMapFs()} mfa := &CLIApp{Fs: afero.NewMemMapFs()}
set := flag.NewFlagSet(cmdFetch, flag.ContinueOnError) cmd := mfa.fetchCommand()
for _, f := range mfa.fetchCommand().Flags { cmd.Action = mfa.fetchManifestOperation
require.NoError(t, f.Apply(set))
}
require.NoError(t, set.Parse([]string{"--" + flagTimeout, "100ms", server.URL}))
cliCtx := urfcli.NewContext(nil, set, nil)
cliCtx.Context = ctx
// fetchManifestOperation logs to the process-global logger. // fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(cliCtx) }) err := runLocked(func() error {
return cmd.Run(ctx, []string{cmdFetch, "--" + flagTimeout, "100ms", server.URL})
})
require.Error(t, err) require.Error(t, err)
} }
internal/cli/freshen.go
+22 -19
View File
@@ -14,7 +14,7 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"github.com/multiformats/go-multihash" "github.com/multiformats/go-multihash"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -202,12 +202,12 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
// resolveFreshenManifestPath determines the manifest path from the CLI // resolveFreshenManifestPath determines the manifest path from the CLI
// arguments, searching directories for a manifest where needed. // arguments, searching directories for a manifest where needed.
func (mfa *CLIApp) resolveFreshenManifestPath(ctx *cli.Context) (string, error) { func (mfa *CLIApp) resolveFreshenManifestPath(cmd *cli.Command) (string, error) {
if ctx.Args().Len() == 0 { if cmd.Args().Len() == 0 {
return findManifest(mfa.Fs, ".") return findManifest(mfa.Fs, ".")
} }
arg := ctx.Args().Get(0) arg := cmd.Args().Get(0)
info, statErr := mfa.Fs.Stat(arg) info, statErr := mfa.Fs.Stat(arg)
if statErr == nil && info.IsDir() { if statErr == nil && info.IsDir() {
@@ -338,14 +338,14 @@ func writeFreshenedManifest(
// newFreshenBuilder constructs the manifest builder configured from CLI // newFreshenBuilder constructs the manifest builder configured from CLI
// flags. // flags.
func newFreshenBuilder(ctx *cli.Context) *mfer.Builder { func newFreshenBuilder(cmd *cli.Command) *mfer.Builder {
builder := mfer.NewBuilder() builder := mfer.NewBuilder()
if ctx.Bool("include-timestamps") { if cmd.Bool("include-timestamps") {
builder.SetIncludeTimestamps(true) builder.SetIncludeTimestamps(true)
} }
// Set up signing options if sign-key is provided // Set up signing options if sign-key is provided
if signKey := ctx.String("sign-key"); signKey != "" { if signKey := cmd.String("sign-key"); signKey != "" {
builder.SetSigningOptions(&mfer.SigningOptions{ builder.SetSigningOptions(&mfer.SigningOptions{
KeyID: mfer.GPGKeyID(signKey), KeyID: mfer.GPGKeyID(signKey),
}) })
@@ -358,13 +358,13 @@ func newFreshenBuilder(ctx *cli.Context) *mfer.Builder {
// freshenScan runs the scan phase against the loaded manifest entries // freshenScan runs the scan phase against the loaded manifest entries
// and returns the populated scanner and the count of removed files. // and returns the populated scanner and the count of removed files.
func (mfa *CLIApp) freshenScan( func (mfa *CLIApp) freshenScan(
ctx *cli.Context, manifestPath, absBase string, cmd *cli.Command, manifestPath, absBase string,
existingByPath map[string]*mfer.MFFilePath, existingByPath map[string]*mfer.MFFilePath,
) (*freshenScanner, int64, error) { ) (*freshenScanner, int64, error) {
log.Infof("scanning filesystem...") log.Infof("scanning filesystem...")
startScan := time.Now() startScan := time.Now()
showProgress := ctx.Bool("progress") showProgress := cmd.Bool("progress")
// Leave out the manifest and a temp file left by an interrupted run, // Leave out the manifest and a temp file left by an interrupted run,
// as gen does. A path that cannot be stat'd, normally because no file // as gen does. A path that cannot be stat'd, normally because no file
@@ -382,8 +382,8 @@ func (mfa *CLIApp) freshenScan(
fs: mfa.Fs, fs: mfa.Fs,
absBase: absBase, absBase: absBase,
excluded: excluded, excluded: excluded,
includeDotfiles: ctx.Bool("include-dotfiles"), includeDotfiles: cmd.Bool("include-dotfiles"),
followSymlinks: ctx.Bool("follow-symlinks"), followSymlinks: cmd.Bool("follow-symlinks"),
showProgress: showProgress, showProgress: showProgress,
existingByPath: existingByPath, existingByPath: existingByPath,
} }
@@ -433,7 +433,7 @@ func hashTotals(entries []*freshenEntry) (int64, int64) {
// runFreshenHash processes every entry through the hasher, aborting if // runFreshenHash processes every entry through the hasher, aborting if
// the context is canceled. // the context is canceled.
func runFreshenHash( func runFreshenHash(
ctx *cli.Context, hasher *freshenHasher, entries []*freshenEntry, ctx context.Context, hasher *freshenHasher, entries []*freshenEntry,
) error { ) error {
for _, e := range entries { for _, e := range entries {
select { select {
@@ -479,18 +479,21 @@ func (mfa *CLIApp) loadExistingEntries(
return existingByPath, nil return existingByPath, nil
} }
func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) freshenManifestOperation(
ctx context.Context, cmd *cli.Command,
) error {
log.Debug("freshenManifestOperation()") log.Debug("freshenManifestOperation()")
basePath := ctx.String("base") basePath := cmd.String("base")
showProgress := ctx.Bool("progress") showProgress := cmd.Bool("progress")
// Find manifest file // Find manifest file
manifestPath, err := mfa.resolveFreshenManifestPath(ctx) manifestPath, err := mfa.resolveFreshenManifestPath(cmd)
if err != nil { if err != nil {
return fmt.Errorf("freshen: %w", err) return fmt.Errorf("freshen: %w", err)
} }
//nolint:contextcheck // mfer loads a manifest without a context
existingByPath, err := mfa.loadExistingEntries(manifestPath) existingByPath, err := mfa.loadExistingEntries(manifestPath)
if err != nil { if err != nil {
return err return err
@@ -502,7 +505,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
} }
// Phase 1: Scan filesystem // Phase 1: Scan filesystem
scanner, removed, err := mfa.freshenScan(ctx, manifestPath, absBase, scanner, removed, err := mfa.freshenScan(cmd, manifestPath, absBase,
existingByPath) existingByPath)
if err != nil { if err != nil {
return err return err
@@ -524,7 +527,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
totalHashBytes: totalHashBytes, totalHashBytes: totalHashBytes,
filesToHash: filesToHash, filesToHash: filesToHash,
startHash: time.Now(), startHash: time.Now(),
builder: newFreshenBuilder(ctx), builder: newFreshenBuilder(cmd),
} }
err = runFreshenHash(ctx, hasher, scanner.entries) err = runFreshenHash(ctx, hasher, scanner.entries)
@@ -548,7 +551,7 @@ func (mfa *CLIApp) freshenManifestOperation(ctx *cli.Context) error {
} }
// Write updated manifest atomically (write to temp, then rename) // Write updated manifest atomically (write to temp, then rename)
err = writeFreshenedManifest(ctx.Context, mfa.Fs, hasher.builder, manifestPath) err = writeFreshenedManifest(ctx, mfa.Fs, hasher.builder, manifestPath)
if err != nil { if err != nil {
return err return err
} }
internal/cli/gen.go
+21 -18
View File
@@ -1,6 +1,7 @@
package cli package cli
import ( import (
"context"
"errors" "errors"
"fmt" "fmt"
"os" "os"
@@ -12,7 +13,7 @@ import (
"github.com/dustin/go-humanize" "github.com/dustin/go-humanize"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -88,12 +89,12 @@ func (mfa *CLIApp) collectInputPaths(args cli.Args) ([]string, error) {
} }
// buildScannerOptions constructs scanner options from the CLI flags. // buildScannerOptions constructs scanner options from the CLI flags.
func (mfa *CLIApp) buildScannerOptions(ctx *cli.Context) *mfer.ScannerOptions { func (mfa *CLIApp) buildScannerOptions(cmd *cli.Command) *mfer.ScannerOptions {
output := ctx.String("output") output := cmd.String("output")
opts := &mfer.ScannerOptions{ opts := &mfer.ScannerOptions{
IncludeDotfiles: ctx.Bool("include-dotfiles"), IncludeDotfiles: cmd.Bool("include-dotfiles"),
FollowSymLinks: ctx.Bool("follow-symlinks"), FollowSymLinks: cmd.Bool("follow-symlinks"),
IncludeTimestamps: ctx.Bool("include-timestamps"), IncludeTimestamps: cmd.Bool("include-timestamps"),
Fs: mfa.Fs, Fs: mfa.Fs,
// Neither a manifest being replaced nor a temp file left by an // Neither a manifest being replaced nor a temp file left by an
// interrupted run belongs in the new manifest. // interrupted run belongs in the new manifest.
@@ -101,14 +102,14 @@ func (mfa *CLIApp) buildScannerOptions(ctx *cli.Context) *mfer.ScannerOptions {
} }
// Set seed for deterministic UUID if provided // Set seed for deterministic UUID if provided
if seed := ctx.String("seed"); seed != "" { if seed := cmd.String("seed"); seed != "" {
opts.Seed = seed opts.Seed = seed
log.Infof("using deterministic seed for manifest UUID") log.Infof("using deterministic seed for manifest UUID")
} }
// Set up signing options if sign-key is provided // Set up signing options if sign-key is provided
if signKey := ctx.String("sign-key"); signKey != "" { if signKey := cmd.String("sign-key"); signKey != "" {
opts.SigningOptions = &mfer.SigningOptions{ opts.SigningOptions = &mfer.SigningOptions{
KeyID: mfer.GPGKeyID(signKey), KeyID: mfer.GPGKeyID(signKey),
} }
@@ -173,14 +174,14 @@ func (mfa *CLIApp) cleanupOnSignal(outFile afero.File, tmpPath string) chan os.S
// runEnumeratePhase enumerates all input paths with optional progress // runEnumeratePhase enumerates all input paths with optional progress
// reporting and logs the totals. // reporting and logs the totals.
func (mfa *CLIApp) runEnumeratePhase(ctx *cli.Context, s *mfer.Scanner) error { func (mfa *CLIApp) runEnumeratePhase(cmd *cli.Command, s *mfer.Scanner) error {
// Set up enumeration progress reporting // Set up enumeration progress reporting
var ( var (
enumProgress chan mfer.EnumerateStatus enumProgress chan mfer.EnumerateStatus
enumWg sync.WaitGroup enumWg sync.WaitGroup
) )
if ctx.Bool("progress") { if cmd.Bool("progress") {
enumProgress = make(chan mfer.EnumerateStatus, 1) enumProgress = make(chan mfer.EnumerateStatus, 1)
enumWg.Add(1) enumWg.Add(1)
@@ -188,7 +189,7 @@ func (mfa *CLIApp) runEnumeratePhase(ctx *cli.Context, s *mfer.Scanner) error {
go reportEnumProgress(enumProgress, &enumWg) go reportEnumProgress(enumProgress, &enumWg)
} }
err := mfa.enumerateInputs(s, ctx.Args(), enumProgress) err := mfa.enumerateInputs(s, cmd.Args(), enumProgress)
if err != nil { if err != nil {
return err return err
} }
@@ -201,22 +202,24 @@ func (mfa *CLIApp) runEnumeratePhase(ctx *cli.Context, s *mfer.Scanner) error {
return nil return nil
} }
func (mfa *CLIApp) generateManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) generateManifestOperation(
ctx context.Context, cmd *cli.Command,
) error {
log.Debug("generateManifestOperation()") log.Debug("generateManifestOperation()")
s := mfer.NewScannerWithOptions(mfa.buildScannerOptions(ctx)) s := mfer.NewScannerWithOptions(mfa.buildScannerOptions(cmd))
// Phase 1: Enumeration - collect paths and stat files // Phase 1: Enumeration - collect paths and stat files
err := mfa.runEnumeratePhase(ctx, s) err := mfa.runEnumeratePhase(cmd, s)
if err != nil { if err != nil {
return err return err
} }
showProgress := ctx.Bool("progress") showProgress := cmd.Bool("progress")
// Check if output file exists // Check if output file exists
outputPath := ctx.String("output") outputPath := cmd.String("output")
if exists, _ := afero.Exists(mfa.Fs, outputPath); exists && !ctx.Bool("force") { if exists, _ := afero.Exists(mfa.Fs, outputPath); exists && !cmd.Bool("force") {
return fmt.Errorf("output file %s %w", outputPath, errOutputExists) return fmt.Errorf("output file %s %w", outputPath, errOutputExists)
} }
@@ -259,7 +262,7 @@ func (mfa *CLIApp) generateManifestOperation(ctx *cli.Context) error {
go reportScanProgress(scanProgress, &scanWg) go reportScanProgress(scanProgress, &scanWg)
} }
err = s.ToManifest(ctx.Context, outFile, scanProgress) err = s.ToManifest(ctx, outFile, scanProgress)
scanWg.Wait() scanWg.Wait()
internal/cli/list.go
+8 -6
View File
@@ -1,33 +1,35 @@
package cli package cli
import ( import (
"context"
"fmt" "fmt"
"time" "time"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
func (mfa *CLIApp) listManifestOperation(ctx *cli.Context) error { func (mfa *CLIApp) listManifestOperation(ctx context.Context, cmd *cli.Command) error {
// Default to ErrorLevel for clean output // Default to ErrorLevel for clean output
log.SetLevel(log.ErrorLevel) log.SetLevel(log.ErrorLevel)
longFormat := ctx.Bool("long") longFormat := cmd.Bool("long")
print0 := ctx.Bool("print0") print0 := cmd.Bool("print0")
pathOrURL, err := mfa.resolveManifestArg(ctx) pathOrURL, err := mfa.resolveManifestArg(cmd)
if err != nil { if err != nil {
return fmt.Errorf("list: %w", err) return fmt.Errorf("list: %w", err)
} }
rc, err := mfa.openManifestReader(pathOrURL) rc, err := mfa.openManifestReader(ctx, pathOrURL)
if err != nil { if err != nil {
return fmt.Errorf("list: %w", err) return fmt.Errorf("list: %w", err)
} }
defer func() { _ = rc.Close() }() defer func() { _ = rc.Close() }()
//nolint:contextcheck // mfer loads a manifest without a context
manifest, err := mfer.NewManifestFromReader(rc) manifest, err := mfer.NewManifestFromReader(rc)
if err != nil { if err != nil {
return fmt.Errorf("list: failed to parse manifest: %w", err) return fmt.Errorf("list: failed to parse manifest: %w", err)
internal/cli/manifest_loader.go
+8 -8
View File
@@ -9,7 +9,7 @@ import (
"strings" "strings"
"time" "time"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
) )
// manifestFetchTimeout bounds HTTP requests made to fetch a manifest. // manifestFetchTimeout bounds HTTP requests made to fetch a manifest.
@@ -30,13 +30,13 @@ func isHTTPURL(s string) bool {
// openManifestReader opens a manifest from a path or URL and returns a ReadCloser. // openManifestReader opens a manifest from a path or URL and returns a ReadCloser.
// The caller must close the returned reader. // The caller must close the returned reader.
func (mfa *CLIApp) openManifestReader(pathOrURL string) (io.ReadCloser, error) { func (mfa *CLIApp) openManifestReader(
ctx context.Context, pathOrURL string,
) (io.ReadCloser, error) {
if isHTTPURL(pathOrURL) { if isHTTPURL(pathOrURL) {
client := &http.Client{Timeout: manifestFetchTimeout} client := &http.Client{Timeout: manifestFetchTimeout}
req, err := http.NewRequestWithContext( req, err := http.NewRequestWithContext(ctx, http.MethodGet, pathOrURL, nil)
context.Background(), http.MethodGet, pathOrURL, nil,
)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to fetch %s: %w", pathOrURL, err) return nil, fmt.Errorf("failed to fetch %s: %w", pathOrURL, err)
} }
@@ -67,9 +67,9 @@ func (mfa *CLIApp) openManifestReader(pathOrURL string) (io.ReadCloser, error) {
// resolveManifestArg resolves the manifest path from CLI arguments. // resolveManifestArg resolves the manifest path from CLI arguments.
// HTTP(S) URLs are returned as-is. Directories are searched for index.mf. // HTTP(S) URLs are returned as-is. Directories are searched for index.mf.
// If no argument is given, the current directory is searched. // If no argument is given, the current directory is searched.
func (mfa *CLIApp) resolveManifestArg(ctx *cli.Context) (string, error) { func (mfa *CLIApp) resolveManifestArg(cmd *cli.Command) (string, error) {
if ctx.Args().Len() > 0 { if cmd.Args().Len() > 0 {
arg := ctx.Args().Get(0) arg := cmd.Args().Get(0)
if isHTTPURL(arg) { if isHTTPURL(arg) {
return arg, nil return arg, nil
} }
internal/cli/mfer.go
+66 -47
View File
@@ -1,6 +1,7 @@
package cli package cli
import ( import (
"context"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -8,7 +9,7 @@ import (
"time" "time"
"github.com/spf13/afero" "github.com/spf13/afero"
"github.com/urfave/cli/v2" "github.com/urfave/cli/v3"
"sneak.berlin/go/mfer/internal/log" "sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer" "sneak.berlin/go/mfer/mfer"
) )
@@ -54,7 +55,7 @@ type CLIApp struct {
gitrev string gitrev string
startupTime time.Time startupTime time.Time
exitCode int exitCode int
app *cli.App app *cli.Command
Stdin io.Reader // Standard input stream Stdin io.Reader // Standard input stream
Stdout io.Writer // Standard output stream for normal output Stdout io.Writer // Standard output stream for normal output
@@ -105,15 +106,15 @@ func (mfa *CLIApp) printBanner() {
// urfave/cli reads a flag from the nearest command that defines it, so each // urfave/cli reads a flag from the nearest command that defines it, so each
// command in the lineage is asked. The highest -v count wins rather than the // command in the lineage is asked. The highest -v count wins rather than the
// sum, because a subcommand without its own copies reads the root's. // sum, because a subcommand without its own copies reads the root's.
func (mfa *CLIApp) setVerbosity(c *cli.Context) { func (mfa *CLIApp) setVerbosity(cmd *cli.Command) {
_, present := os.LookupEnv("MFER_DEBUG") _, present := os.LookupEnv("MFER_DEBUG")
verbosity := 0 verbosity := 0
quiet := false quiet := false
for _, ctx := range c.Lineage() { for _, c := range cmd.Lineage() {
verbosity = max(verbosity, ctx.Count("verbose")) verbosity = max(verbosity, c.Count("verbose"))
quiet = quiet || ctx.Bool("quiet") quiet = quiet || c.Bool("quiet")
} }
switch { switch {
@@ -127,23 +128,34 @@ func (mfa *CLIApp) setVerbosity(c *cli.Context) {
} }
// commonFlags returns the -v and -q flags taken by the root and by the // commonFlags returns the -v and -q flags taken by the root and by the
// generate, check, freshen and fetch subcommands. // generate, check, freshen and fetch subcommands. They are local, so the
// root's copies are not inherited by the subcommands that do not take them.
func commonFlags() []cli.Flag { func commonFlags() []cli.Flag {
return []cli.Flag{ return []cli.Flag{
&cli.BoolFlag{ &cli.BoolFlag{
Name: "verbose", Name: "verbose",
Aliases: []string{"v"}, Aliases: []string{"v"},
Usage: "Increase verbosity (-v for verbose, -v -v for debug)", Usage: "Increase verbosity (-v for verbose, -v -v for debug)",
Count: new(int), Local: true,
}, },
&cli.BoolFlag{ &cli.BoolFlag{
Name: "quiet", Name: "quiet",
Aliases: []string{"q"}, Aliases: []string{"q"},
Usage: "Suppress output except errors", Usage: "Suppress output except errors",
Local: true,
}, },
} }
} }
// stopOnFirstArg returns the StopOnNthArg setting every command uses: flags
// are read only before the command's first argument, and everything after it
// is an argument, as with urfave/cli v2. So `mfer gen d -v` names a path "-v".
func stopOnFirstArg() *int {
n := 1
return &n
}
// requireSignatureFlag returns the --require-signature flag taken by the // requireSignatureFlag returns the --require-signature flag taken by the
// check and fetch subcommands. // check and fetch subcommands.
func requireSignatureFlag() *cli.StringFlag { func requireSignatureFlag() *cli.StringFlag {
@@ -151,7 +163,7 @@ func requireSignatureFlag() *cli.StringFlag {
Name: flagRequireSignature, Name: flagRequireSignature,
Aliases: []string{"S"}, Aliases: []string{"S"},
Usage: "Require manifest to be signed by the specified GPG key ID", Usage: "Require manifest to be signed by the specified GPG key ID",
EnvVars: []string{"MFER_REQUIRE_SIGNATURE"}, Sources: cli.EnvVars("MFER_REQUIRE_SIGNATURE"),
} }
} }
@@ -161,11 +173,12 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
Aliases: []string{"gen"}, Aliases: []string{"gen"},
Usage: "Generate manifest file", Usage: "Generate manifest file",
ArgsUsage: "[path ...]", ArgsUsage: "[path ...]",
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
mfa.setVerbosity(c) Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner() mfa.printBanner()
return mfa.generateManifestOperation(c) return mfa.generateManifestOperation(ctx, cmd)
}, },
Flags: append(commonFlags(), Flags: append(commonFlags(),
&cli.BoolFlag{ &cli.BoolFlag{
@@ -199,12 +212,12 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
Name: "sign-key", Name: "sign-key",
Aliases: []string{"s"}, Aliases: []string{"s"},
Usage: "GPG key ID to sign the manifest with", Usage: "GPG key ID to sign the manifest with",
EnvVars: []string{"MFER_SIGN_KEY"}, Sources: cli.EnvVars("MFER_SIGN_KEY"),
}, },
&cli.StringFlag{ &cli.StringFlag{
Name: "seed", Name: "seed",
Usage: "Seed value for deterministic manifest UUID", Usage: "Seed value for deterministic manifest UUID",
EnvVars: []string{"MFER_SEED"}, Sources: cli.EnvVars("MFER_SEED"),
}, },
&cli.BoolFlag{ &cli.BoolFlag{
Name: "include-timestamps", Name: "include-timestamps",
@@ -220,11 +233,12 @@ func (mfa *CLIApp) checkCommand() *cli.Command {
Name: cmdCheck, Name: cmdCheck,
Usage: "Validate files using manifest file", Usage: "Validate files using manifest file",
ArgsUsage: manifestArgsUsage, ArgsUsage: manifestArgsUsage,
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
mfa.setVerbosity(c) Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner() mfa.printBanner()
return mfa.checkManifestOperation(c) return mfa.checkManifestOperation(ctx, cmd)
}, },
Flags: append(commonFlags(), Flags: append(commonFlags(),
&cli.StringFlag{ &cli.StringFlag{
@@ -252,11 +266,12 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
Name: cmdFreshen, Name: cmdFreshen,
Usage: "Update manifest with changed, new, and removed files", Usage: "Update manifest with changed, new, and removed files",
ArgsUsage: manifestArgsUsage, ArgsUsage: manifestArgsUsage,
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
mfa.setVerbosity(c) Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner() mfa.printBanner()
return mfa.freshenManifestOperation(c) return mfa.freshenManifestOperation(ctx, cmd)
}, },
Flags: append(commonFlags(), Flags: append(commonFlags(),
&cli.StringFlag{ &cli.StringFlag{
@@ -285,7 +300,7 @@ func (mfa *CLIApp) freshenCommand() *cli.Command {
Name: "sign-key", Name: "sign-key",
Aliases: []string{"s"}, Aliases: []string{"s"},
Usage: "GPG key ID to sign the manifest with", Usage: "GPG key ID to sign the manifest with",
EnvVars: []string{"MFER_SIGN_KEY"}, Sources: cli.EnvVars("MFER_SIGN_KEY"),
}, },
&cli.BoolFlag{ &cli.BoolFlag{
Name: "include-timestamps", Name: "include-timestamps",
@@ -301,10 +316,11 @@ func (mfa *CLIApp) exportCommand() *cli.Command {
Name: cmdExport, Name: cmdExport,
Usage: "Export manifest contents as JSON", Usage: "Export manifest contents as JSON",
ArgsUsage: "[manifest file or URL]", ArgsUsage: "[manifest file or URL]",
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
mfa.setVerbosity(c) Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
return mfa.exportManifestOperation(c) return mfa.exportManifestOperation(ctx, cmd)
}, },
} }
} }
@@ -313,7 +329,8 @@ func (mfa *CLIApp) versionCommand() *cli.Command {
return &cli.Command{ return &cli.Command{
Name: cmdVersion, Name: cmdVersion,
Usage: "Show version", Usage: "Show version",
Action: func(_ *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
Action: func(context.Context, *cli.Command) error {
mfa.printVersion() mfa.printVersion()
return nil return nil
@@ -327,9 +344,8 @@ func (mfa *CLIApp) listCommand() *cli.Command {
Aliases: []string{"ls"}, Aliases: []string{"ls"},
Usage: "List files in manifest", Usage: "List files in manifest",
ArgsUsage: manifestArgsUsage, ArgsUsage: manifestArgsUsage,
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
return mfa.listManifestOperation(c) Action: mfa.listManifestOperation,
},
Flags: []cli.Flag{ Flags: []cli.Flag{
&cli.BoolFlag{ &cli.BoolFlag{
Name: "long", Name: "long",
@@ -349,11 +365,12 @@ func (mfa *CLIApp) fetchCommand() *cli.Command {
Name: cmdFetch, Name: cmdFetch,
Usage: "fetch manifest and referenced files", Usage: "fetch manifest and referenced files",
ArgsUsage: "URL", ArgsUsage: "URL",
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
mfa.setVerbosity(c) Action: func(ctx context.Context, cmd *cli.Command) error {
mfa.setVerbosity(cmd)
mfa.printBanner() mfa.printBanner()
return mfa.fetchManifestOperation(c) return mfa.fetchManifestOperation(ctx, cmd)
}, },
Flags: append(commonFlags(), Flags: append(commonFlags(),
&cli.DurationFlag{ &cli.DurationFlag{
@@ -385,38 +402,40 @@ func (mfa *CLIApp) run(args []string) {
log.SetOutput(mfa.Stdout, mfa.Stderr) log.SetOutput(mfa.Stdout, mfa.Stderr)
log.Init() log.Init()
// -v means verbose, not version. urfave/cli's built-in version flag // -v means verbose, not version, at the root as on the generate, check,
// claims -v by default, which made "mfer -v --version" fail to parse and // freshen and fetch subcommands: verbose is the more common meaning of -v
// gave -v a different meaning at the root than on the generate, check, // in tools that offer both. urfave/cli's built-in version flag claims -v
// freshen and fetch subcommands, where it means verbose. Verbose is the // by default, so the version flag takes the capital -V instead.
// more common meaning of -v in tools that offer both, so -v means verbose
// at the root too and the version flag takes the capital -V.
// VersionFlag and VersionPrinter are urfave/cli package globals; run() is // VersionFlag and VersionPrinter are urfave/cli package globals; run() is
// serialized in tests, so assigning them here is safe. // serialized in tests, so assigning them here is safe.
cli.VersionFlag = &cli.BoolFlag{ cli.VersionFlag = &cli.BoolFlag{
Name: cmdVersion, Name: cmdVersion,
Aliases: []string{"V"}, Aliases: []string{"V"},
Usage: "print the version", Usage: "print the version",
Local: true,
} }
cli.VersionPrinter = func(_ *cli.Context) { mfa.printVersion() } cli.VersionPrinter = func(*cli.Command) { mfa.printVersion() }
mfa.app = &cli.App{ mfa.app = &cli.Command{
Name: mfa.appname, Name: mfa.appname,
Usage: "Manifest generator", Usage: "Manifest generator",
Version: mfa.VersionString(), Version: mfa.VersionString(),
EnableBashCompletion: true, EnableShellCompletion: true,
Writer: mfa.Stdout, Writer: mfa.Stdout,
ErrWriter: mfa.Stderr, // v3 writes its "Incorrect Usage" line to ErrWriter; v2 wrote it to
// stdout, before the help, so it stays on stdout.
ErrWriter: mfa.Stdout,
Flags: commonFlags(), Flags: commonFlags(),
Action: func(c *cli.Context) error { StopOnNthArg: stopOnFirstArg(),
if c.Args().Len() > 0 { Action: func(_ context.Context, cmd *cli.Command) error {
return fmt.Errorf("%w %q", errUnknownCommand, c.Args().First()) if cmd.Args().Len() > 0 {
return fmt.Errorf("%w %q", errUnknownCommand, cmd.Args().First())
} }
mfa.setVerbosity(c) mfa.setVerbosity(cmd)
mfa.printBanner() mfa.printBanner()
return cli.ShowAppHelp(c) return cli.ShowRootCommandHelp(cmd)
}, },
Commands: []*cli.Command{ Commands: []*cli.Command{
mfa.generateCommand(), mfa.generateCommand(),
@@ -431,7 +450,7 @@ func (mfa *CLIApp) run(args []string) {
mfa.app.HideVersion = false mfa.app.HideVersion = false
err := mfa.app.Run(args) err := mfa.app.Run(context.Background(), args)
if err != nil { if err != nil {
mfa.exitCode = 1 mfa.exitCode = 1
mfer/mf.go
-3
View File
@@ -1,3 +0,0 @@
package mfer
//go:generate protoc ./mf.proto --go_out=paths=source_relative:.
package.json
-1
View File
@@ -1,6 +1,5 @@
{ {
"name": "mfer", "name": "mfer",
"version": "0.1.0",
"private": true, "private": true,
"description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.", "description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.",
"license": "WTFPL", "license": "WTFPL",
script/bootstrap
+59 -5
View File
@@ -13,11 +13,15 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06. Never "latest" or "lts"; exact versions. # Pinned versions, 2026-07-06. Never "latest" or "lts"; exact versions.
NODE_VERSION="22.17.0" # The node version is in .nvmrc, where script/prettier reads it too.
NODE_VERSION="$(cat "$ROOT/.nvmrc")"
NVM_VERSION="0.40.3" NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz # sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0" NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22" YARN_VERSION="1.22.22"
# protoc v33.4, 2026-10-04, for script/generate. The sha256 of each
# platform's release archive is in ensure_protoc.
PROTOC_VERSION="33.4"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""
@@ -74,9 +78,11 @@ verify_sha256() {
fi fi
} }
# nvm is a bash script; run a command in a bash with nvm loaded # nvm is a bash script; run a command in a bash with nvm loaded.
# --no-use: otherwise loading nvm here switches to the version .nvmrc
# names, and fails silently while that version is not installed yet.
nvm_sh() { nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*" bash -c ". \"\$HOME/.nvm/nvm.sh\" --no-use && $*"
} }
ensure_nvm() { ensure_nvm() {
@@ -122,6 +128,48 @@ install_js_deps() {
fi fi
} }
# Unpack protoc's release archive for this platform into bin/protoc, after
# checking the archive's sha256, unless bin/protoc already holds the pinned
# version.
ensure_protoc() {
dir="$ROOT/bin/protoc"
if [ "$("$dir/bin/protoc" --version 2>/dev/null)" = \
"libprotoc $PROTOC_VERSION" ]; then
return 0
fi
case "$(uname -s) $(uname -m)" in
"Linux x86_64")
platform="linux-x86_64"
sha256="c0040ea9aef08fdeb2c74ca609b18d5fdbfc44ea0042fcfbfb38860d35f7dd66"
;;
"Linux aarch64" | "Linux arm64")
platform="linux-aarch_64"
sha256="15aa988f4a6090636525ec236a8e4b3aab41eef402751bd5bb2df6afd9b7b5a5"
;;
"Darwin x86_64")
platform="osx-x86_64"
sha256="a49bec10d039e902d3b43e49938c42526f90011467609864fa6386ac4014da58"
;;
"Darwin arm64")
platform="osx-aarch_64"
sha256="726297dcfed58592fd35620a5a6246ae020c39e88f3fd4cb1827df7bcf3dfcf1"
;;
*)
echo "bootstrap: no protoc archive pinned for $(uname -s) $(uname -m)" >&2
exit 1
;;
esac
if missing curl; then pkg_install curl curl curl curl; fi
if missing unzip; then pkg_install unzip unzip unzip unzip; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/protoc.zip" \
"https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${platform}.zip"
verify_sha256 "$tmp/protoc.zip" "$sha256"
rm -rf "$dir"
unzip -q "$tmp/protoc.zip" -d "$dir"
rm -rf "$tmp"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
@@ -132,8 +180,10 @@ main() {
# ---- JS / docs repos ---- # ---- JS / docs repos ----
# This is a Go repo, but node and yarn are required anyway: prettier # This is a Go repo, but node and yarn are required anyway: prettier
# formats the Markdown and JSON, and script/fmt-check verifies it. # formats the Markdown and JSON, and script/fmt-check verifies it.
# The version is pinned by package.json/yarn.lock, whose integrity # The version is pinned by package.json/yarn.lock: yarn checks every
# hashes --frozen-lockfile enforces. # package it fetches against its yarn.lock integrity hash, and
# --frozen-lockfile fails instead of rewriting a yarn.lock that no
# longer matches package.json.
ensure_node ensure_node
ensure_yarn ensure_yarn
install_js_deps install_js_deps
@@ -142,6 +192,10 @@ main() {
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# No golangci-lint: script/lint runs it in Docker only. # No golangci-lint: script/lint runs it in Docker only.
go mod download go mod download
# gofumpt and protoc-gen-go: bin/tools/go.mod pins them, and
# script/gofumpt and script/generate build them from there.
(cd "$ROOT/bin/tools" && go mod download)
ensure_protoc
# ---- Python repos ---- # ---- Python repos ----
# if missing python3; then pkg_install python3 python3 python3 python3; fi # if missing python3; then pkg_install python3 python3 python3 python3; fi
script/build
+1 -3
View File
@@ -12,9 +12,7 @@ main() {
# empty constant. # empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
go build -tags urfave_cli_no_docs \ go build -ldflags "-X main.Gitrev=$version" -o bin/mfer ./cmd/mfer
-ldflags "-X main.Gitrev=$version" \
-o bin/mfer ./cmd/mfer
} }
main "$@" main "$@"
script/generate
+20 -21
View File
@@ -4,26 +4,17 @@
# regenerates mf.pb.go: it is committed, so building and checking need no # regenerates mf.pb.go: it is committed, so building and checking need no
# protoc. A test fails while mf.proto no longer matches the recorded hash. # protoc. A test fails while mf.proto no longer matches the recorded hash.
# #
# Needs exactly the protoc and protoc-gen-go versions named in the header of # Runs the protoc that script/bootstrap unpacks into bin/protoc, and the
# the committed mf.pb.go (README.md says how to install them). Another # protoc-gen-go that bin/tools/go.mod pins. Another version of either
# version writes a different mf.pb.go, so the script refuses to run. # writes a different mf.pb.go.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# protoc 33.4 names itself v6.33.4 in the mf.pb.go header. # The protoc version script/bootstrap installs. protoc 33.4 names itself
# v6.33.4 in the mf.pb.go header.
PROTOC_VERSION="33.4" PROTOC_VERSION="33.4"
PROTOC_GEN_GO_VERSION="v1.36.11" PROTOC="$ROOT/bin/protoc/bin/protoc"
# require_version <command> <its exact --version output>
require_version() {
actual="$("$1" --version 2>/dev/null || true)"
if [ "$actual" != "$2" ]; then
echo "generate: needs $2 on PATH, found: ${actual:-none}" >&2
echo " README.md says how to install it." >&2
exit 1
fi
}
# sha256 <file>: print "<hash> <file>", with sha256sum, or with shasum # sha256 <file>: print "<hash> <file>", with sha256sum, or with shasum
# where there is no sha256sum. # where there is no sha256sum.
@@ -39,16 +30,24 @@ sha256() {
} }
main() { main() {
# A bin/protoc left from before the pin moved fails here, until
# script/bootstrap replaces it.
actual="$("$PROTOC" --version 2>/dev/null || true)"
if [ "$actual" != "libprotoc $PROTOC_VERSION" ]; then
echo "generate: needs protoc $PROTOC_VERSION in bin/protoc," \
"found: ${actual:-none}; run script/bootstrap" >&2
exit 1
fi
# `go tool -n` builds protoc-gen-go from bin/tools and prints where the
# binary is, without running it.
plugin="$(cd "$ROOT/bin/tools" && go tool -n protoc-gen-go)"
cd "$ROOT/mfer" cd "$ROOT/mfer"
# `go install` puts protoc-gen-go in $(go env GOPATH)/bin, which is
# often not on PATH.
PATH="$PATH:$(go env GOPATH)/bin"
require_version protoc "libprotoc $PROTOC_VERSION"
require_version protoc-gen-go "protoc-gen-go $PROTOC_GEN_GO_VERSION"
# Hashed before regenerating, so a missing hash tool stops the script # Hashed before regenerating, so a missing hash tool stops the script
# before it changes anything. Regenerating leaves mf.proto as it is. # before it changes anything. Regenerating leaves mf.proto as it is.
proto_hash="$(sha256 mf.proto)" proto_hash="$(sha256 mf.proto)"
go generate . "$PROTOC" --plugin=protoc-gen-go="$plugin" \
--go_out=paths=source_relative:. ./mf.proto
echo "$proto_hash" >mf.proto.sha256 echo "$proto_hash" >mf.proto.sha256
} }
script/gofumpt
+9 -9
View File
@@ -10,10 +10,10 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# gofumpt v0.12.0, 2026-10-04. `go run` fetches and builds exactly this # The gofumpt version is the one bin/tools/go.mod pins. `go tool` run in
# version, so neither a developer machine nor the lint image needs # bin/tools builds it from source checked against the hashes in
# gofumpt installed. # bin/tools/go.sum, so neither a developer machine nor the lint image needs
GOFUMPT="mvdan.cc/gofumpt@v0.12.0" # it installed.
usage() { usage() {
echo "usage: script/gofumpt --write|--check" >&2 echo "usage: script/gofumpt --write|--check" >&2
@@ -22,15 +22,15 @@ usage() {
main() { main() {
[ "$#" -eq 1 ] || usage [ "$#" -eq 1 ] || usage
cd "$ROOT" cd "$ROOT/bin/tools"
# Every Go file in the repo. gofumpt holds generated files, such as # Every Go file in the repo, from $ROOT down. gofumpt holds generated
# mfer/mf.pb.go, to gofmt's rules only. # files, such as mfer/mf.pb.go, to gofmt's rules only.
case "$1" in case "$1" in
--write) go run "$GOFUMPT" -l -w . ;; --write) go tool gofumpt -l -w "$ROOT" ;;
--check) --check)
# Own line: a failing command inside `[ -n "$(...)" ]` does # Own line: a failing command inside `[ -n "$(...)" ]` does
# not trip `set -e`, so a gofumpt that never ran would pass. # not trip `set -e`, so a gofumpt that never ran would pass.
unformatted="$(go run "$GOFUMPT" -l .)" unformatted="$(go tool gofumpt -l "$ROOT")"
if [ -n "$unformatted" ]; then if [ -n "$unformatted" ]; then
echo "gofumpt: files need formatting (run make fmt):" >&2 echo "gofumpt: files need formatting (run make fmt):" >&2
echo "$unformatted" >&2 echo "$unformatted" >&2
script/prettier
+24 -24
View File
@@ -18,24 +18,9 @@ usage() {
exit 2 exit 2
} }
# Prefer the version pinned by package.json/yarn.lock so that CI and # Only the prettier yarn installed from yarn.lock, never one on PATH: a
# developer machines format identically. Fall back to a prettier on PATH, # different version formats differently.
# but say so, because a different version formats differently. PRETTIER="$ROOT/node_modules/.bin/prettier"
find_prettier() {
if [ -x "$ROOT/node_modules/.bin/prettier" ]; then
printf '%s\n' "$ROOT/node_modules/.bin/prettier"
return 0
fi
if command -v prettier >/dev/null 2>&1; then
echo "prettier: node_modules/.bin/prettier is absent; using the" \
"prettier on PATH, which may be a different version than the" \
"one pinned in package.json. Run script/bootstrap to install" \
"the pinned version." >&2
command -v prettier
return 0
fi
return 1
}
main() { main() {
[ "$#" -eq 1 ] || usage [ "$#" -eq 1 ] || usage
@@ -46,10 +31,26 @@ main() {
cd "$ROOT" cd "$ROOT"
if ! prettier_bin="$(find_prettier)"; then # Where there is no node on PATH, script/bootstrap installs the version
echo "prettier: not found." >&2 # .nvmrc names through nvm, which keeps it in this directory.
echo " Install it with: script/bootstrap" >&2 if ! command -v node >/dev/null 2>&1; then
echo " (installs the version pinned in package.json/yarn.lock)" >&2 PATH="$HOME/.nvm/versions/node/v$(cat .nvmrc)/bin:$PATH"
fi
if ! command -v node >/dev/null 2>&1; then
echo "prettier: node is missing; run script/bootstrap" >&2
exit 1
fi
# node_modules keeps the old prettier after package.json moves to a new
# one, until script/bootstrap runs again, so compare the two.
if ! installed="$("$PRETTIER" --version 2>/dev/null)"; then
echo "prettier: not installed; run script/bootstrap" >&2
exit 1
fi
pinned="$(node -p 'require("./package.json").devDependencies.prettier')"
if [ "$installed" != "$pinned" ]; then
echo "prettier: package.json pins $pinned but $installed is" \
"installed; run script/bootstrap" >&2
exit 1 exit 1
fi fi
@@ -62,8 +63,7 @@ main() {
# patterns always match at least one tracked file (README.md, # patterns always match at least one tracked file (README.md,
# package.json), so an empty match means the glob broke, and prettier # package.json), so an empty match means the glob broke, and prettier
# erroring out is exactly what we want rather than a vacuous pass. # erroring out is exactly what we want rather than a vacuous pass.
"$prettier_bin" "$mode" "**/*.md" "$PRETTIER" "$mode" "**/*.md" "**/*.json"
"$prettier_bin" "$mode" "**/*.json"
} }
main "$@" main "$@"