sneak/mfer
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions Releases Wiki Activity

Compare commits

base: sneak/mfer:ef56eeeae09650f6ca13baf1f4079d801bfa00a2
Branches Tags
sneak/mfer:main sneak/mfer:next sneak/mfer:issue-116-gomodguard
..
compare: sneak/mfer:issue-116-gomodguard
Branches Tags
sneak/mfer:next sneak/mfer:issue-116-gomodguard sneak/mfer:main
1 Commits
ef56eeeae0 .. issue-116-gomodguard
Author SHA1 Message Date
sneak 114ba0a29f Disable the deprecated gomodguard linter (closes #116)
check / check (push) Failing after 2s
Details 
golangci-lint v2.12 deprecates gomodguard in favour of gomodguard_v2.
With `default: all` both ran, and every lint run printed the
deprecation warning. Disable the old name, the same way wsl is
disabled for wsl_v5; gomodguard_v2 stays enabled. The old linter had
no settings here, so there is nothing to carry over.

Model: opus-5-5
 2026-10-04 08:06:44 +00:00
52 changed files with 901 additions and 3511 deletions
Expand all files Collapse all files
.dockerignore
+3 -2
View File
@@ -59,5 +59,6 @@
**/.vscode
**/*.sublime-*
# This repo's own host-built binary (make build).
/bin/mfer
# This repo's own host-built archives (Makefile).
*.tmp
*.dockerimage
.gitignore
+7 -2
View File
@@ -1,9 +1,14 @@
/bin/mfer
/bin/
/tmp
/node_modules/
*.tmp
*.dockerimage
/vendor
vendor.tzst
modcache.tzst
# Generated manifest files
/index.mf
.index.mf
# Secrets
.env
.golangci.yml
+1
View File
@@ -16,6 +16,7 @@ linters:
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
settings:
AGENTS.md
+1 -1
View File
@@ -26,7 +26,7 @@ source for coding standards, formatting, linting, and workflow rules.
- This is a Go library + CLI tool for generating `.mf` manifest files.
- The proto definition is in `mfer/mf.proto`; generated `.pb.go` files are
committed (required for `go get` compatibility).
- The format specification is in `docs/FORMAT.md`.
- The format specification is in `FORMAT.md`.
- Open work, open design questions included, is tracked only in the repo's
issues: https://git.eeqj.de/sneak/mfer/issues. There is no `TODO.md` and no
TODO list in `README.md`; do not add either. For this repo this overrides the
Dockerfile
+9 -6
View File
@@ -8,10 +8,12 @@ RUN go mod download
COPY . .
# Touch .pb.go so make does not try to regenerate via protoc (file is committed)
RUN touch mfer/mf.pb.go
# Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below. The image has no gofumpt
# either; script/gofumpt builds the version tools/go.mod requires.
RUN script/gofumpt --check
# markdown half runs in the mdfmt stage below.
RUN make fmt-check-go
# The linter directly, not `make lint`: script/lint builds this stage, and
# there is no docker inside this build.
RUN golangci-lint run --config .golangci.yml ./...
@@ -31,7 +33,7 @@ COPY . .
RUN script/prettier --check
# Build stage — tests and compilation
# golang:1.23.12, 2026-03-14
# golang:1.23 (2026-03-14)
FROM golang@sha256:60deed95d3888cc5e4d9ff8a10c54e5edc008c6ae3fba6187be6fb592e19e8c0 AS builder
# Force BuildKit to run the lint and mdfmt stages by creating stage dependencies
@@ -44,6 +46,9 @@ RUN go mod download
COPY . .
# Touch .pb.go so make does not try to regenerate via protoc (file is committed)
RUN touch mfer/mf.pb.go
RUN make test
# A build context sent as a tar archive, as upaas sends it, keeps its files'
@@ -70,7 +75,5 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
RUN ldd /mfer 2>&1 | grep -q 'not a dynamic executable'
FROM scratch
# scratch has no CA certificates; fetch needs them to verify HTTPS servers.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /mfer /mfer
ENTRYPOINT ["/mfer"]
docs/FORMAT.md → FORMAT.md
+3 -5
View File
@@ -19,7 +19,7 @@ An `.mf` file consists of two parts, concatenated:
There is no length prefix or version byte between the magic and the protobuf
message. The protobuf message extends to the end of the file.
See [`mfer/mf.proto`](../mfer/mf.proto) for exact field numbers and types.
See [`mfer/mf.proto`](mfer/mf.proto) for exact field numbers and types.
## Outer Message (`MFFileOuter`)
@@ -49,9 +49,7 @@ The `innerMessage` field is compressed with
enforce a decompression size limit to prevent decompression bombs. The reference
implementation limits decompressed size to 256 MB. It writes zstd frames with a
window of at most 8 MiB, the largest window the zstd format recommends decoders
support, and refuses frames that ask for a larger one. It also refuses an inner
message whose file entries, hashes, timestamps and MIME types, counted at 160,
112, 64 and 16 bytes each, add up to more than 8 times its size.
support, and refuses frames that ask for a larger one.
## Inner Message (`MFFile`)
@@ -143,6 +141,6 @@ The recommended MIME type for `.mf` files is `application/octet-stream`. The
## Reference
- Proto definition: [`mfer/mf.proto`](../mfer/mf.proto)
- Proto definition: [`mfer/mf.proto`](mfer/mf.proto)
- Reference implementation:
[git.eeqj.de/sneak/mfer](https://git.eeqj.de/sneak/mfer)
Makefile
+78 -18
View File
@@ -1,8 +1,21 @@
.PHONY: bootstrap setup test lint fmt fmt-check check docker hooks build generate fuzz
export DOCKER_BUILDKIT := 1
export PROGRESS_NO_TRUNC := 1
GOPATH := $(shell go env GOPATH)
export PATH := $(PATH):$(GOPATH)/bin
PROTOC_GEN_GO := $(GOPATH)/bin/protoc-gen-go
SOURCEFILES := mfer/*.go mfer/*.proto internal/*/*.go cmd/*/*.go go.mod go.sum
ARCH := $(shell uname -m)
GITREV_BUILD := $(shell bash $(PWD)/bin/gitrev.sh 2>/dev/null || echo unknown)
APPNAME := mfer
VERSION := 0.1.0
export DOCKER_IMAGE_CACHE_DIR := $(HOME)/Library/Caches/Docker/$(APPNAME)-$(ARCH)
GOLDFLAGS += -X main.Version=$(VERSION)
GOLDFLAGS += -X main.Gitrev=$(GITREV_BUILD)
GOFLAGS := -ldflags "$(GOLDFLAGS)"
# Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
# of README.md).
.PHONY: bootstrap setup docker default run ci test fuzz check lint fmt fmt-check fmt-check-go fmt-check-md hooks fixme
default: fmt test
bootstrap:
@script/bootstrap
@@ -10,32 +23,79 @@ bootstrap:
setup:
@script/setup
run: ./bin/mfer
./$<
./$< gen
ci: test
test:
@script/test
lint:
@script/lint
fuzz:
@script/fuzz
fmt:
@script/fmt
$(PROTOC_GEN_GO):
test -e $(PROTOC_GEN_GO) || go install -v google.golang.org/protobuf/cmd/protoc-gen-go@v1.28.1
fmt-check:
@script/fmt-check
fixme:
@grep -nir fixme . | grep -v Makefile
check:
@script/check
docker:
@script/docker
fmt-check:
@script/fmt-check
# Halves of fmt-check, for environments that have only one toolchain:
# the Docker lint stage has Go but no node, the markdown stage the reverse.
fmt-check-go:
@script/fmt-check-go
fmt-check-md:
@script/prettier --check
hooks:
@script/install-precommit
build:
@script/build
mfer/mf.pb.go: mfer/mf.proto
cd mfer && go generate .
generate:
@script/generate
bin/mfer: $(SOURCEFILES) mfer/mf.pb.go
protoc --version
cd cmd/mfer && go build -tags urfave_cli_no_docs -o ../../bin/mfer $(GOFLAGS) .
fuzz:
@script/fuzz
clean:
rm -rfv mfer/*.pb.go bin/mfer cmd/mfer/mfer *.dockerimage
fmt:
@script/fmt
lint:
@script/lint
docker:
@script/docker
sneak-mfer.$(ARCH).tzst.dockerimage: $(SOURCEFILES) vendor.tzst modcache.tzst
docker build --progress plain --build-arg GITREV=$(GITREV_BUILD) -t sneak/mfer .
docker save sneak/mfer | pv | zstdmt -19 > $@
du -sh $@
godoc:
open http://127.0.0.1:6060
godoc -http=:6060
vendor.tzst: go.mod go.sum
go mod tidy
go mod vendor
cd vendor && tar -c . | pv | zstdmt -19 > $(PWD)/$@.tmp
rm -rf vendor
mv $@.tmp $@
modcache.tzst: go.mod go.sum
go mod tidy
cd $(HOME)/go/pkg && chmod -R u+rw . && rm -rf mod sumdb
go mod download -x
cd $(shell go env GOMODCACHE) && tar -c . | pv | zstdmt -19 > $(PWD)/$@.tmp
mv $@.tmp $@
README.md
+18 -50
View File
@@ -29,25 +29,23 @@ is committed, so no `protoc` toolchain is required:
```sh
git clone https://git.eeqj.de/sneak/mfer.git
cd mfer
make build
go build -o bin/mfer ./cmd/mfer
```
Generate a manifest for a directory tree, verify it later, and fetch a published
tree by URL:
```sh
# Write index.mf, a manifest of the files under the current directory.
# Write .index.mf, a manifest of the files under the current directory.
bin/mfer gen .
# Verify the files on disk against the manifest. Exits nonzero if any file
# it lists is missing or corrupted; warns about files it does not list.
bin/mfer check index.mf
# is missing or corrupted.
bin/mfer check .index.mf
# Download and cryptographically verify a tree published over HTTP into
# ./mirror: mfer fetches <url>/index.mf, downloads every file it lists,
# skipping any already there with the right hash, then saves the manifest as
# mirror/index.mf.
bin/mfer fetch --dest mirror https://example.com/tree/
# Download and cryptographically verify a tree published over HTTP: mfer
# fetches <url>/index.mf, then downloads every file it lists.
bin/mfer fetch https://example.com/tree/
```
Run `bin/mfer help` for the full command list, or `bin/mfer <command> --help`
@@ -74,42 +72,24 @@ provide:
`script/bootstrap`, then `script/install-precommit`
- `script/projectname` — output the project name (`mfer`); used by other scripts
such as `script/docker`
- `script/test` — run the test suite (`go test`); one test fails when
`mfer/mf.proto` no longer matches the hash `script/generate` recorded
- `script/build` (`make build`) — build the `mfer` binary into `bin/mfer`,
stamped with the revision `mfer version` prints: the output of
`git describe --tags --always --dirty`, as `script/docker` passes it
- `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from
`mfer/mf.proto` and record the hash of that `mfer/mf.proto` in
`mfer/mf.proto.sha256`; the only thing that regenerates the committed
`mfer/mf.pb.go`. It needs the exact versions that wrote the committed file,
and refuses to run with any other: `protoc` 33.4 (unpack
`protoc-33.4-<platform>.zip` from
[its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4)
and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11
(`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which
installs it in `$(go env GOPATH)/bin`; `script/generate` adds that directory
to `PATH`)
- `script/test` — run the test suite (`go test`), regenerating the protobuf code
first if it is stale
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
as ordinary tests
- `script/lint` — run `golangci-lint` in Docker: builds only the `lint` stage of
the `Dockerfile` (the Go format check, then the linter), uncached so it runs
every time, then removes the image
- `script/fmt` — format all code and docs (writes): `script/gofumpt --write` and
- `script/fmt` — format all code and docs (writes): `gofumpt` and
`script/prettier --write`
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
given mode, `--write` or `--check`, at the version `tools/go.mod` requires
(built on demand by `go tool` from source checked against the hashes in
`tools/go.sum`, so nothing installs it); `script/fmt`, `script/fmt-check` and
the Docker lint stage all go through it, so they cannot disagree about Go
formatting
- `script/prettier` — run prettier over the repository's canonical file set
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
`--check`; the single definition of that file set, so `script/fmt` and
`script/fmt-check` cannot disagree about it
- `script/fmt-check` — check formatting without writing:
`script/gofumpt --check` plus `script/prettier --check`
- `script/fmt-check` — check formatting without writing: `script/fmt-check-go`
plus `script/prettier --check`
- `script/fmt-check-go` — the Go half of `script/fmt-check`, on its own, for the
Docker lint stage, whose image has no node
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`
- `script/docker` — build the Docker image tagged with the project name
- `script/cibuild` — CI entrypoint: builds the image with the same command as
@@ -251,7 +231,7 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
as question 10 on [issue 82](https://git.eeqj.de/sneak/mfer/issues/82).
- Should the on-disk serialization format be proto3 or json? Settled: it is
proto3, see `docs/FORMAT.md` and `mfer/mf.proto`.
proto3, see `FORMAT.md` and `mfer/mf.proto`.
# Tool Examples
@@ -260,22 +240,10 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
- `mfer check` / `mfer check .`
- verifies checksums of all files in manifest, displaying error and exiting
nonzero if any files are missing or corrupted
- warns about each file under the base directory that the manifest does not
list, hidden files included; with `--no-extra-files` each one is a failure
instead
- `mfer fetch https://example.com/stuff/`
- fetches `/stuff/index.mf` and downloads all files listed in manifest into
the current directory, or the one given with `--dest`, and assures
cryptographic integrity of downloaded files. A file already there with the
size and hash the manifest lists is skipped. Once every file is in place,
the manifest is saved there as `index.mf`, so `mfer check` can verify the
tree later. A manifest that lists `index.mf` (in any letter case) or
`.index.mf.tmp` at the top of the tree is refused before any file is
downloaded, since saving the manifest would replace it.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
- as above, but first refuses a manifest not signed by the key with that
fingerprint, as `mfer check --require-signature` does, before downloading
any file.
- fetches `/stuff/index.mf` and downloads all files listed in manifest,
optionally resuming any that already exist locally, and assures
cryptographic integrity of downloaded files.
# Implementation Plan
bin/gitrev.sh
+7
View File
@@ -0,0 +1,7 @@
#!/bin/bash
#
if [[ ! -z "$GITREV" ]]; then
echo $GITREV
else
git describe --tags --always --dirty=-dirty
fi
bin/usage.sh
-23
View File
@@ -1,23 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
# usage.sh - Generate and check a manifest from the repo
# Run from repo root: bin/usage.sh
cleanup() {
rm -rf "$TMPDIR"
}
main() {
TMPDIR=$(mktemp -d)
MANIFEST="$TMPDIR/index.mf"
trap cleanup EXIT
echo "Building mfer..."
go build -o "$TMPDIR/mfer" ./cmd/mfer
"$TMPDIR/mfer" generate -o "$MANIFEST" .
"$TMPDIR/mfer" check --base . "$MANIFEST"
}
main "$@"
contrib/usage.sh
Executable
+19
View File
@@ -0,0 +1,19 @@
#!/bin/bash
set -euo pipefail
# usage.sh - Generate and check a manifest from the repo
# Run from repo root: ./contrib/usage.sh
TMPDIR=$(mktemp -d)
MANIFEST="$TMPDIR/index.mf"
cleanup() {
rm -rf "$TMPDIR"
}
trap cleanup EXIT
echo "Building mfer..."
go build -o "$TMPDIR/mfer" ./cmd/mfer
"$TMPDIR/mfer" generate -o "$MANIFEST" .
"$TMPDIR/mfer" check --base . "$MANIFEST"
go.mod
+15 -6
View File
@@ -3,33 +3,42 @@ module sneak.berlin/go/mfer
go 1.23
require (
github.com/apex/log v1.9.0
github.com/davecgh/go-spew v1.1.1
github.com/dustin/go-humanize v1.0.1
github.com/google/uuid v1.1.2
github.com/klauspost/compress v1.18.2
github.com/multiformats/go-multihash v0.2.3
github.com/pterm/pterm v0.12.35
github.com/spf13/afero v1.8.0
github.com/stretchr/testify v1.8.1
github.com/urfave/cli/v2 v2.27.7
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
github.com/urfave/cli/v2 v2.23.6
google.golang.org/protobuf v1.28.1
)
require (
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/atomicgo/cursor v0.0.1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect
github.com/fatih/color v1.7.0 // indirect
github.com/gookit/color v1.4.2 // indirect
github.com/klauspost/cpuid/v2 v2.0.9 // indirect
github.com/kr/pretty v0.2.0 // indirect
github.com/mattn/go-colorable v0.1.2 // indirect
github.com/mattn/go-isatty v0.0.8 // indirect
github.com/mattn/go-runewidth v0.0.13 // indirect
github.com/minio/sha256-simd v1.0.0 // indirect
github.com/mr-tron/base58 v1.2.0 // indirect
github.com/multiformats/go-varint v0.0.6 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 // indirect
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
golang.org/x/crypto v0.0.0-20220525230936-793ad666bf5e // indirect
golang.org/x/sys v0.1.0 // indirect
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 // indirect
golang.org/x/text v0.3.6 // indirect
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
lukechampine.com/blake3 v1.1.6 // indirect
)
go.sum
+83 -6
View File
@@ -38,6 +38,21 @@ cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3f
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/MarvinJWendt/testza v0.1.0/go.mod h1:7AxNvlfeHP7Z/hDQ5JtE3OKYT3XFUeLCDE2DQninSqs=
github.com/MarvinJWendt/testza v0.2.1/go.mod h1:God7bhG8n6uQxwdScay+gjm9/LnO4D3kkcZX4hv9Rp8=
github.com/MarvinJWendt/testza v0.2.8/go.mod h1:nwIcjmr0Zz+Rcwfh3/4UhBp7ePKVhuBExvZqnKYWlII=
github.com/MarvinJWendt/testza v0.2.10/go.mod h1:pd+VWsoGUiFtq+hRKSU1Bktnn+DMCSrDrXDpX2bG66k=
github.com/MarvinJWendt/testza v0.2.12 h1:/PRp/BF+27t2ZxynTiqj0nyND5PbOtfJS0SuTuxmgeg=
github.com/MarvinJWendt/testza v0.2.12/go.mod h1:JOIegYyV7rX+7VZ9r77L/eH6CfJHHzXjB69adAhzZkI=
github.com/apex/log v1.9.0 h1:FHtw/xuaM8AgmvDDTI9fiwoAL25Sq2cxojnZICUU8l0=
github.com/apex/log v1.9.0/go.mod h1:m82fZlWIuiWzWP04XCTXmnX0xRkYYbCdYn8jbJeLBEA=
github.com/apex/logs v1.0.0/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy8kCu4PNA+aP7WUV72eXWJeP9/r3/K9aLE=
github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys=
github.com/atomicgo/cursor v0.0.1 h1:xdogsqa6YYlLfM+GyClC/Lchf7aiMerFiZQn7soTOoU=
github.com/atomicgo/cursor v0.0.1/go.mod h1:cBON2QmmrysudxNBFthvMtN32r3jxVRIvzkUiF/RuIk=
github.com/aws/aws-sdk-go v1.20.6/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod h1:q/89r3U2H7sSsE2t6Kca0lfwTK8JdoNGS/yzM/4iH5I=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
@@ -46,8 +61,8 @@ github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDk
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cpuguy83/go-md2man/v2 v2.0.7 h1:zbFlGlXEAKlwXpmvle3d8Oe3YnkKIK4xSRTd3sHPnBo=
github.com/cpuguy83/go-md2man/v2 v2.0.7/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -59,9 +74,13 @@ github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1m
github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
@@ -115,15 +134,21 @@ github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLe
github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
github.com/gookit/color v1.4.2 h1:tXy44JFSFkKnELV6WaMo/lLfu/meqITX3iAV52do7lk=
github.com/gookit/color v1.4.2/go.mod h1:fqRyamkC1W8uxl+lxCQxOT09l/vYfZ+QeiX3rKQHCoQ=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af/go.mod h1:Nht3zPeWKUH0NzdCt2Blrr5ys8VGpn0CEB0cQHVjt7k=
github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
@@ -133,12 +158,22 @@ github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa02
github.com/klauspost/cpuid/v2 v2.0.9 h1:lgaqFMSdTdQYdZ04uHyN2d/eKdOMyi2YLSvlQIBFYa4=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.2.0 h1:s5hAObm+yFO5uHYt5dYjxi2rXrsnmRpJx4OYvIWUaQs=
github.com/kr/pretty v0.2.0/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
github.com/mattn/go-colorable v0.1.2 h1:/bC9yWikZXAL9uJdulbSfyVNIR3n3trXl+v8+1sx8mU=
github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-isatty v0.0.8 h1:HLtExJ+uU2HOZ+wI0Tt5DtUDrx8yhUqDcp7fYERX4CE=
github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
github.com/minio/sha256-simd v1.0.0 h1:v1ta+49hkWZyvaKwrQB8elexRqm6Y0aMLjCNsrYxo6g=
github.com/minio/sha256-simd v1.0.0/go.mod h1:OuYzVNI5vcoYIAmbIvHPl3N3jUzVedXbKy5RFepssQM=
github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
@@ -147,14 +182,32 @@ github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7B
github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM=
github.com/multiformats/go-varint v0.0.6 h1:gk85QWKxh3TazbLxED/NlDVv8+q+ReFJk7Y2W/KhfNY=
github.com/multiformats/go-varint v0.0.6/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/pterm/pterm v0.12.27/go.mod h1:PhQ89w4i95rhgE+xedAoqous6K9X+r6aSOI2eFF7DZI=
github.com/pterm/pterm v0.12.29/go.mod h1:WI3qxgvoQFFGKGjGnJR849gU0TsEOvKn5Q8LlY1U7lg=
github.com/pterm/pterm v0.12.30/go.mod h1:MOqLIyMOgmTDz9yorcYbcw+HsgoZo3BQfg2wtl3HEFE=
github.com/pterm/pterm v0.12.31/go.mod h1:32ZAWZVXD7ZfG0s8qqHXePte42kdz8ECtRyEejaWgXU=
github.com/pterm/pterm v0.12.33/go.mod h1:x+h2uL+n7CP/rel9+bImHD5lF3nM9vJj80k9ybiiTTE=
github.com/pterm/pterm v0.12.35 h1:A/vHwDM+WByn0sTPlpL2L6kOTy12xqZuwNFMF/NlA+U=
github.com/pterm/pterm v0.12.35/go.mod h1:NjiL09hFhT/vWjQHSj1athJpx6H8cjpHXNAK5bUw8T8=
github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rogpeppe/fastuuid v1.1.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6LYCDYWNEvQ=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/smartystreets/assertions v1.0.0/go.mod h1:kHHU4qYBaI3q23Pp3VPrmWhuIUrLW/7eUrw0BU5VaoM=
github.com/smartystreets/go-aws-auth v0.0.0-20180515143844-0c1422d1fdb9/go.mod h1:SnhjPscd9TpLiy1LpzGSKh3bXCfxxXuqd9xmQJy3slM=
github.com/smartystreets/gunit v1.0.0/go.mod h1:qwPWnhz6pn0NnRBP++URONOVyNkPyr4SauJk4cUOwJs=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.8.0 h1:5MmtuhAgYeU6qpa7w7bP0dv6MBYuup0vekhSpSkoq60=
@@ -162,17 +215,28 @@ github.com/spf13/afero v1.8.0/go.mod h1:CtAatgMJh6bJEIs48Ay/FOnkljP3WeGUG0MC1RfA
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/urfave/cli/v2 v2.27.7 h1:bH59vdhbjLv3LAvIu6gd0usJHgoTTPhCFib8qqOwXYU=
github.com/urfave/cli/v2 v2.27.7/go.mod h1:CyNAG/xg+iAOg0N4MPGZqVmv2rCoP267496AOXUZjA4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1 h1:gEOO8jv9F4OT7lGCjxCBTO/36wtF6j2nSip77qHd4x4=
github.com/xrash/smetrics v0.0.0-20240521201337-686a1a2994c1/go.mod h1:Ohn+xnUBiLI6FVj/9LpzZWtj1/D6lUovWYBkxHVV3aM=
github.com/tj/assert v0.0.0-20171129193455-018094318fb0/go.mod h1:mZ9/Rh9oLWpLLDRpvE+3b7gP/C2YyLFYxNmcLnPTMe0=
github.com/tj/assert v0.0.3 h1:Df/BlaZ20mq6kuai7f5z2TvPFiwC3xaWJSDQNiIS3Rk=
github.com/tj/assert v0.0.3/go.mod h1:Ne6X72Q+TB1AteidzQncjw9PabbMp4PBMZ1k+vd1Pvk=
github.com/tj/go-buffer v1.1.0/go.mod h1:iyiJpfFcR2B9sXu7KvjbT9fpM4mOelRSDTbntVj52Uc=
github.com/tj/go-elastic v0.0.0-20171221160941-36157cbbebc2/go.mod h1:WjeM0Oo1eNAjXGDx2yma7uG2XoyRZTq1uv3M/o7imD0=
github.com/tj/go-kinesis v0.0.0-20171128231115-08b17f58cb1b/go.mod h1:/yhzCV0xPfx6jb1bBgRFjl5lytqVqZXEaeqWP8lTEao=
github.com/tj/go-spin v1.1.0/go.mod h1:Mg1mzmePZm4dva8Qz60H2lHwmJ2loum4VIrLgVnKwh4=
github.com/urfave/cli/v2 v2.23.6 h1:iWmtKD+prGo1nKUtLO0Wg4z9esfBM4rAV4QRLQiEmJ4=
github.com/urfave/cli/v2 v2.23.6/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 h1:QldyIu/L63oPpyvQmHgvgickp1Yw510KJOqX7H24mg8=
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs=
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673/go.mod h1:N3UwUGtsrSj3ccvlPHLoLsHnpR27oXr4ZE984MbSER8=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
@@ -184,6 +248,7 @@ go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
@@ -227,6 +292,7 @@ golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
@@ -276,7 +342,9 @@ golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
@@ -307,11 +375,15 @@ golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211013075003-97ac67df715c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -473,8 +545,13 @@ gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
internal/cli/check.go
+29 -44
View File
@@ -11,7 +11,6 @@ import (
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/dustin/go-humanize"
@@ -76,22 +75,25 @@ func safeRateUint64(rate float64) uint64 {
return uint64(rate)
}
// findManifest returns the path of the manifest with the default name in
// dir, or an error if there is none.
// findManifest looks for a manifest file in the given directory.
// It checks for index.mf and .index.mf, returning the first one found.
func findManifest(fs afero.Fs, dir string) (string, error) {
path := filepath.Join(dir, defaultManifestName)
candidates := []string{"index.mf", ".index.mf"}
for _, name := range candidates {
path := filepath.Join(dir, name)
exists, err := afero.Exists(fs, path)
if err != nil {
return "", err
exists, err := afero.Exists(fs, path)
if err != nil {
return "", err
}
if exists {
return path, nil
}
}
if !exists {
return "", fmt.Errorf("%w in %s (looked for %s)",
errNoManifestFound, dir, defaultManifestName)
}
return path, nil
return "", fmt.Errorf(
"%w in %s (looked for index.mf and .index.mf)", errNoManifestFound, dir)
}
// fetchManifestToTemp downloads a manifest URL to a temporary file and
@@ -164,9 +166,7 @@ func verifyRequiredSigner(
}
// reportCheckProgress renders progress updates until the channel closes.
func reportCheckProgress(progress <-chan mfer.CheckStatus, wg *sync.WaitGroup) {
defer wg.Done()
func reportCheckProgress(progress <-chan mfer.CheckStatus) {
for status := range progress {
if status.ETA > 0 {
log.Progressf("Checking: %d/%d files, %s/s, ETA %s, %d failures",
@@ -206,21 +206,16 @@ func countCheckFailures(
}
// findExtraFiles reports files present on disk but absent from the
// manifest, and anything the search cannot read: each is a failure under
// --no-extra-files, otherwise a warning.
// manifest, counting each as a failure.
func findExtraFiles(ctx *cli.Context, chk *mfer.Checker, failures *int64) error {
extraResults := make(chan mfer.Result, 1)
extraDone := make(chan struct{})
go func() {
for result := range extraResults {
if ctx.Bool("no-extra-files") {
*failures++
*failures++
log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message)
} else {
log.Warnf("%s: %s (%s)", result.Status, result.Path, result.Message)
}
log.Infof("%s: %s (%s)", result.Status, result.Path, result.Message)
}
close(extraDone)
@@ -243,17 +238,11 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
results := make(chan mfer.Result, 1)
// Set up progress channel
var (
progress chan mfer.CheckStatus
progressWg sync.WaitGroup
)
var progress chan mfer.CheckStatus
if showProgress {
progress = make(chan mfer.CheckStatus, 1)
progressWg.Add(1)
go reportCheckProgress(progress, &progressWg)
go reportCheckProgress(progress)
}
// Process results in a goroutine
@@ -265,9 +254,6 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
// Run check
err := chk.Check(ctx.Context, results, progress)
progressWg.Wait()
if err != nil {
return 0, fmt.Errorf("check failed: %w", err)
}
@@ -275,9 +261,12 @@ func runCheck(ctx *cli.Context, chk *mfer.Checker, showProgress bool) (int64, er
// Wait for results processing to complete
<-done
err = findExtraFiles(ctx, chk, &failures)
if err != nil {
return 0, err
// Check for extra files if requested
if ctx.Bool("no-extra-files") {
err = findExtraFiles(ctx, chk, &failures)
if err != nil {
return 0, err
}
}
return failures, nil
@@ -309,17 +298,13 @@ func (mfa *CLIApp) checkManifestOperation(ctx *cli.Context) error {
log.Infof("checking manifest %s with base %s", manifestPath, basePath)
// Create checker
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath,
BasePath: basePath,
Fs: mfa.Fs,
})
chk, err := mfer.NewChecker(manifestPath, basePath, mfa.Fs)
if err != nil {
return fmt.Errorf("failed to load manifest: %w", err)
}
// Check signature requirement
requiredSigner := ctx.String(flagRequireSignature)
requiredSigner := ctx.String("require-signature")
if requiredSigner != "" {
err = verifyRequiredSigner(ctx.Context, chk, requiredSigner)
if err != nil {
internal/cli/entry_test.go
+28 -423
View File
@@ -8,12 +8,8 @@ import (
"io"
"math/rand"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
@@ -29,11 +25,11 @@ const (
testFile1 = "/testdir/file1.txt"
testMF = "/testdir/test.mf"
testOutput = "/output.mf"
testOutputTmp = "/output.mf.tmp"
testManifest = "/manifest.mf"
testFlagBase = "--base"
testFlagNoExtra = "--no-extra-files"
testFlagVersion = "--version"
testFlagVerbose = "--verbose"
)
var errSimulatedWrite = errors.New("simulated write failure")
@@ -46,32 +42,20 @@ var errSimulatedWrite = errors.New("simulated write failure")
var runMu sync.Mutex
// runCLI invokes RunWithOptions while holding runMu so parallel tests
// capture their own output, and returns its exit code.
// capture their own output. Before releasing the lock it points the
// process-global logger at io.Discard: other tests log outside the lock
// (manifest loads, scans), and those lines must not land in this run's
// buffers once it has returned and its test is reading them.
func runCLI(opts *RunOptions) int {
exitCode, _ := runCLIWithLevel(opts)
return exitCode
}
// runCLIWithLevel is runCLI that also returns the log level the run left
// set, read while runMu still keeps other runs from changing it. Each run
// starts at the default level, as a new process does. Before releasing the
// lock it points the process-global logger at io.Discard: other tests log
// outside the lock (manifest loads, scans), and those lines must not land in
// this run's buffers once it has returned and its test is reading them.
func runCLIWithLevel(opts *RunOptions) (int, log.Level) {
runMu.Lock()
defer runMu.Unlock()
log.SetLevel(log.InfoLevel)
exitCode := RunWithOptions(opts)
level := log.GetLevel()
log.SetOutput(io.Discard, io.Discard)
log.Init()
return exitCode, level
return exitCode
}
func TestMain(m *testing.M) {
@@ -251,90 +235,6 @@ func TestRootVerbosityFlags(t *testing.T) {
})
}
// commandsTakingVerbose returns the command lines -v can follow: the root and
// the generate, check, freshen and fetch subcommands.
func commandsTakingVerbose() [][]string {
return [][]string{
{testApp},
{testApp, cmdGenerate},
{testApp, cmdCheck},
{testApp, cmdFreshen},
{testApp, cmdFetch},
}
}
// TestVerboseCount asserts that one -v or --verbose gives verbose output and
// two -v give debug output (issue #125). urfave/cli before v2.25.5 counted a
// flag given by its alias twice, so one -v gave debug output.
func TestVerboseCount(t *testing.T) {
t.Parallel()
cases := []struct {
flags []string
want log.Level
}{
{[]string{"-v"}, log.VerboseLevel},
{[]string{testFlagVerbose}, log.VerboseLevel},
{[]string{"-v", "-v"}, log.DebugLevel},
}
for _, command := range commandsTakingVerbose() {
for _, tc := range cases {
args := slices.Concat(command, tc.flags)
t.Run(strings.Join(args, " "), func(t *testing.T) {
t.Parallel()
_, level := runCLIWithLevel(testOpts(args, afero.NewMemMapFs()))
assert.Equal(t, tc.want, level)
})
}
}
}
// TestCombinedShortVerboseRefused asserts that -vv is refused (issue #125):
// single-letter flags do not combine, so the -v help text says -v -v.
func TestCombinedShortVerboseRefused(t *testing.T) {
t.Parallel()
for _, command := range commandsTakingVerbose() {
args := slices.Concat(command, []string{"-vv"})
t.Run(strings.Join(args, " "), func(t *testing.T) {
t.Parallel()
opts := testOpts(args, afero.NewMemMapFs())
exitCode, level := runCLIWithLevel(opts)
assert.Equal(t, 1, exitCode)
assert.Contains(t, testStderr(t, opts), "flag provided but not defined: -vv")
assert.Equal(t, log.InfoLevel, level)
})
}
}
// TestShortAndLongVerboseRefused asserts that -v and --verbose given together
// are refused (issue #125): urfave/cli v2 refuses a flag given under two of its
// names, and one flag with an alias keeps help and parsing simple.
func TestShortAndLongVerboseRefused(t *testing.T) {
t.Parallel()
for _, command := range commandsTakingVerbose() {
args := slices.Concat(command, []string{"-v", testFlagVerbose})
t.Run(strings.Join(args, " "), func(t *testing.T) {
t.Parallel()
opts := testOpts(args, afero.NewMemMapFs())
exitCode, level := runCLIWithLevel(opts)
assert.Equal(t, 1, exitCode)
assert.Contains(t, testStderr(t, opts), "Cannot use two forms of the same flag")
assert.Equal(t, log.InfoLevel, level)
})
}
}
func TestHelpCommand(t *testing.T) {
t.Parallel()
@@ -394,74 +294,6 @@ func TestGenerateAndCheckCommand(t *testing.T) {
assert.Equal(t, 0, exitCode, "check failed: %s", testStderr(t, opts))
}
// sharedWriter appends to a buffer shared with other sharedWriters, so
// output written to stdout and stderr is kept in the order it was written.
// Each write first waits for delay.
type sharedWriter struct {
mu *sync.Mutex
buf *bytes.Buffer
delay time.Duration
}
func (w sharedWriter) Write(p []byte) (int, error) {
time.Sleep(w.delay)
w.mu.Lock()
defer w.mu.Unlock()
return w.buf.Write(p)
}
// TestCheckClearsProgressBeforeSummary asserts that check --progress writes
// its last progress line and clears it before it logs the summary. Progress
// writes are slowed down, so a progress goroutine that check did not wait for
// would write after the summary, or after the run has returned.
//
// Progress lines and log lines take the logger's lock, and when the progress
// goroutine gets it first the clear lands before the summary even without the
// wait. Which goroutine gets it first changes from run to run, so check runs
// ten times.
func TestCheckClearsProgressBeforeSummary(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello world")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", testMF, testDir}, fs)
require.Equal(t, 0, runCLI(opts), "generate failed: %s", testStderr(t, opts))
for range 10 {
var (
mu sync.Mutex
output bytes.Buffer
)
opts = testOpts([]string{
testApp, cmdCheck, "--progress", testFlagBase, testDir, testMF,
}, fs)
opts.Stdout = sharedWriter{mu: &mu, buf: &output, delay: 100 * time.Millisecond}
opts.Stderr = sharedWriter{mu: &mu, buf: &output}
require.Equal(t, 0, runCLI(opts))
mu.Lock()
got := output.String()
mu.Unlock()
lastProgress := strings.Index(got, "Checking: 1/1 files")
progressDone := strings.Index(got, "\r\033[K")
summary := strings.Index(got, "checked 1 files")
require.NotEqual(t, -1, lastProgress, "no last progress line in %q", got)
require.NotEqual(t, -1, progressDone, "progress line never cleared in %q", got)
require.NotEqual(t, -1, summary, "no summary in %q", got)
require.Less(t, lastProgress, progressDone,
"progress cleared before its last line in %q", got)
require.Less(t, progressDone, summary,
"summary logged before progress was cleared in %q", got)
}
}
func TestCheckCommandWithMissingFile(t *testing.T) {
t.Parallel()
@@ -573,10 +405,7 @@ func TestGenerateExcludesDotfilesByDefault(t *testing.T) {
assert.True(t, exists)
// Verify manifest only has 1 file (the non-dotfile)
manifest, err := mfer.NewManifestFromFile(&mfer.ManifestFromFileOptions{
Path: testMF,
Fs: fs,
})
manifest, err := mfer.NewManifestFromFile(fs, testMF)
require.NoError(t, err)
assert.Len(t, manifest.Files(), 1)
assert.Equal(t, "file1.txt", manifest.Files()[0].GetPath())
@@ -600,10 +429,7 @@ func TestGenerateWithIncludeDotfiles(t *testing.T) {
require.Equal(t, 0, exitCode)
// Verify manifest has 2 files (including dotfile)
manifest, err := mfer.NewManifestFromFile(&mfer.ManifestFromFileOptions{
Path: testMF,
Fs: fs,
})
manifest, err := mfer.NewManifestFromFile(fs, testMF)
require.NoError(t, err)
assert.Len(t, manifest.Files(), 2)
}
@@ -705,155 +531,30 @@ func TestNoExtraFilesWithSubdirectory(t *testing.T) {
"check should fail when extra files exist in subdirectory")
}
// TestCheckWarnsAboutUnlistedFiles adds one file the manifest does not list
// to a tree that had none: it gets one warning and the check passes, unless
// --no-extra-files makes it a failure. --quiet hides the warning, not the
// failure.
func TestCheckWarnsAboutUnlistedFiles(t *testing.T) {
func TestCheckWithoutNoExtraFilesIgnoresExtra(t *testing.T) {
t.Parallel()
for name, unlisted := range map[string]string{
"regular file": "extra.txt",
"dotfile": ".hidden",
"file in hidden directory": ".git/config",
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
fs := afero.NewMemMapFs()
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello")
// Create test file
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "hello")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
// Generate manifest
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", testManifest, testDir}, fs)
exitCode := runCLI(opts)
require.Equal(t, 0, exitCode)
check := func(flags ...string) (int, string) {
args := append([]string{testApp, cmdCheck, testFlagBase, testDir}, flags...)
opts := testOpts(append(args, testManifest), fs)
return runCLI(opts), testStderr(t, opts)
}
exitCode, stderr := check()
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
assert.NotContains(t, stderr, "not in manifest")
writeTestFile(t, fs, filepath.Join(testDir, unlisted), "unlisted")
exitCode, stderr = check()
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
assert.Equal(t, 1, strings.Count(stderr, "not in manifest"), stderr)
assert.Contains(t, stderr, unlisted)
exitCode, stderr = check("-q")
assert.Equal(t, 0, exitCode, "stderr: %s", stderr)
assert.NotContains(t, stderr, "not in manifest")
exitCode, stderr = check(testFlagNoExtra)
assert.Equal(t, 1, exitCode, "stderr: %s", stderr)
assert.Contains(t, stderr, unlisted)
exitCode, _ = check("-q", testFlagNoExtra)
assert.Equal(t, 1, exitCode)
})
}
}
// TestCheckNeverReportsManifest keeps the manifest inside the checked tree,
// under several names and path spellings, and names the tree both directly
// and through a symlink: the manifest is never reported, even under
// --no-extra-files. The manifest is recognized by file identity, which needs
// the real filesystem.
func TestCheckNeverReportsManifest(t *testing.T) {
t.Parallel()
// Manifest paths are relative to the checked tree.
for name, manifest := range map[string]string{
"default name": defaultManifestName,
"hidden name": ".index.mf",
"other name in a subdirectory": "sub/listing.mf",
"path spelled through ..": "sub/../index.mf",
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
// A temp dir holding data/tree and link, a symlink to data.
root := t.TempDir()
tree := filepath.Join(root, "data", "tree")
// Not filepath.Join, which would clean away a "..".
manifestPath := tree + "/" + manifest
fs := afero.NewOsFs()
require.NoError(t, fs.MkdirAll(filepath.Join(tree, "sub"), 0o750))
require.NoError(t,
os.Symlink(filepath.Join(root, "data"), filepath.Join(root, "link")))
writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "-o", manifestPath, tree,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
for _, base := range []string{tree, filepath.Join(root, "link", "tree")} {
opts = testOpts([]string{
testApp, cmdCheck, testFlagNoExtra, testFlagBase, base, manifestPath,
}, fs)
assert.Equal(t, 0, runCLI(opts),
"base %s, stderr: %s", base, testStderr(t, opts))
assert.NotContains(t, testStderr(t, opts), "not in manifest")
}
})
}
}
// unlistableDirFs is a filesystem on which one directory cannot be listed.
type unlistableDirFs struct {
afero.Fs
dir string
}
//nolint:ireturn // Open must return afero.File to satisfy afero.Fs.
func (f unlistableDirFs) Open(name string) (afero.File, error) {
if name == f.dir {
return nil, os.ErrPermission
}
return f.Fs.Open(name)
}
// TestCheckWithUnlistableDirectory has a directory under the base that
// cannot be listed, followed by an unlisted file: both are warned about and
// the check still passes, unless --no-extra-files is given.
func TestCheckWithUnlistableDirectory(t *testing.T) {
t.Parallel()
mem := afero.NewMemMapFs()
require.NoError(t, mem.MkdirAll("/testdir/locked", 0o755))
writeTestFile(t, mem, testFile1, "hello")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "-o", testManifest, testDir,
}, mem)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
// Directories are searched in name order, so this comes after "locked".
writeTestFile(t, mem, "/testdir/unlisted.txt", "unlisted")
fs := unlistableDirFs{Fs: mem, dir: "/testdir/locked"}
opts = testOpts([]string{testApp, cmdCheck, testFlagBase, testDir, testManifest}, fs)
assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Contains(t, testStderr(t, opts), os.ErrPermission.Error())
assert.Contains(t, testStderr(t, opts), "unlisted.txt")
// Add extra file
writeTestFile(t, fs, "/testdir/extra.txt", "extra")
// Check WITHOUT --no-extra-files (should pass - extra files ignored)
opts = testOpts([]string{
testApp, cmdCheck, testFlagNoExtra, testFlagBase, testDir, testManifest,
testApp, cmdCheck, "-q", testFlagBase, testDir, testManifest,
}, fs)
assert.Equal(t, 1, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Contains(t, testStderr(t, opts), "unlisted.txt")
exitCode = runCLI(opts)
assert.Equal(t, 0, exitCode,
"check without --no-extra-files should ignore extra files")
}
func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
@@ -876,7 +577,7 @@ func TestGenerateAtomicWriteNoTempFileOnSuccess(t *testing.T) {
assert.True(t, exists, "output file should exist")
// Verify temp file does NOT exist
tmpExists, err := afero.Exists(fs, manifestTempPath(testOutput))
tmpExists, err := afero.Exists(fs, testOutputTmp)
require.NoError(t, err)
assert.False(t, tmpExists,
"temp file should not exist after successful generation")
@@ -908,7 +609,7 @@ func TestGenerateAtomicWriteOverwriteWithForce(t *testing.T) {
"manifest should be overwritten")
// Verify temp file does NOT exist
tmpExists, err := afero.Exists(fs, manifestTempPath(testOutput))
tmpExists, err := afero.Exists(fs, testOutputTmp)
require.NoError(t, err)
assert.False(t, tmpExists,
"temp file should not exist after successful generation")
@@ -937,102 +638,6 @@ func TestGenerateFailsWithoutForceWhenOutputExists(t *testing.T) {
assert.Equal(t, "existing", string(content), "original file should be preserved")
}
// manifestPaths returns the file paths listed by the manifest at path.
func manifestPaths(t *testing.T, fs afero.Fs, path string) []string {
t.Helper()
manifest, err := mfer.NewManifestFromFile(&mfer.ManifestFromFileOptions{
Path: path,
Fs: fs,
})
require.NoError(t, err)
paths := make([]string, 0, len(manifest.Files()))
for _, f := range manifest.Files() {
paths = append(paths, f.GetPath())
}
return paths
}
// TestGenerateLeavesOutputOutOfListing overwrites an output file inside the
// scanned tree with --force: the old file is not listed, even when the tree
// is named through a symlink, while a file named index.mf in a subdirectory
// still is. The output file is recognized by file identity, which needs
// the real filesystem.
func TestGenerateLeavesOutputOutOfListing(t *testing.T) {
t.Parallel()
// Paths are relative to a temp dir holding data/tree and link, a
// symlink to data.
for name, tc := range map[string]struct{ input, output string }{
"default name": {"data/tree", "data/tree/index.mf"},
"other name in a subdirectory": {"data/tree", "data/tree/sub/listing.mf"},
"tree named through a symlink": {"link/tree", "data/tree/index.mf"},
} {
t.Run(name, func(t *testing.T) {
t.Parallel()
root := t.TempDir()
tree := filepath.Join(root, "data", "tree")
output := filepath.Join(root, tc.output)
fs := afero.NewOsFs()
require.NoError(t, fs.MkdirAll(filepath.Join(tree, "sub"), 0o750))
require.NoError(t,
os.Symlink(filepath.Join(root, "data"), filepath.Join(root, "link")))
writeTestFile(t, fs, filepath.Join(tree, testFileTxt), "hello")
writeTestFile(t, fs, filepath.Join(tree, "sub", "index.mf"), "an ordinary file")
writeTestFile(t, fs, output, "previous manifest")
opts := testOpts([]string{
testApp, cmdGenerate, "-q", "--force", "-o", output, filepath.Join(root, tc.input),
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, []string{testFileTxt, "sub/index.mf"},
manifestPaths(t, fs, output))
})
}
}
// TestGenerateDefaultOutputLeftOutOfListing runs gen with --force and no
// other arguments, so it scans the current directory and writes the
// relative path index.mf: the index.mf already there is not listed.
//
//nolint:paralleltest // changes the process-global working directory
func TestGenerateDefaultOutputLeftOutOfListing(t *testing.T) {
chdirTemp(t)
fs := afero.NewOsFs()
writeTestFile(t, fs, testFileTxt, "hello")
writeTestFile(t, fs, "index.mf", "previous manifest")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "--force"}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Equal(t, []string{testFileTxt}, manifestPaths(t, fs, "index.mf"))
}
// TestGenerateLeavesLeftoverTempFileOutOfListing runs gen where an
// interrupted run left its temp file beside the output: the leftover is
// not listed, and gen does not fail when it overwrites it.
func TestGenerateLeavesLeftoverTempFileOutOfListing(t *testing.T) {
t.Parallel()
root := t.TempDir()
output := filepath.Join(root, "index.mf")
fs := afero.NewOsFs()
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "hello")
writeTestFile(t, fs, manifestTempPath(output), "part of a manifest")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", output, root}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.Equal(t, []string{testFileTxt}, manifestPaths(t, fs, output))
}
func TestGenerateAtomicWriteUsesTemp(t *testing.T) {
t.Parallel()
@@ -1055,7 +660,7 @@ func TestGenerateAtomicWriteUsesTemp(t *testing.T) {
exists, _ := afero.Exists(fs, testOutput)
assert.True(t, exists, "output file should exist")
tmpExists, _ := afero.Exists(fs, manifestTempPath(testOutput))
tmpExists, _ := afero.Exists(fs, testOutputTmp)
assert.False(t, tmpExists, "temp file should be cleaned up")
// Verify manifest is valid (not empty)
@@ -1121,7 +726,7 @@ func TestGenerateAtomicWriteCleansUpOnError(t *testing.T) {
"output file should not exist after failed generation (atomic write)")
// Temp file should also not exist
tmpExists, _ := afero.Exists(baseFs, manifestTempPath(testOutput))
tmpExists, _ := afero.Exists(baseFs, testOutputTmp)
assert.False(t, tmpExists,
"temp file should be cleaned up after failed generation")
}
internal/cli/errmsg_test.go
+18 -44
View File
@@ -61,11 +61,7 @@ func unsignedChecker(t *testing.T) *mfer.Checker {
require.NoError(t, s.ToManifest(context.Background(), &buf, nil))
require.NoError(t, afero.WriteFile(fs, "/d/index.mf", buf.Bytes(), 0o644))
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: "/d/index.mf",
BasePath: "/d",
Fs: fs,
})
chk, err := mfer.NewChecker("/d/index.mf", "/d", fs)
require.NoError(t, err)
require.False(t, chk.IsSigned())
@@ -78,7 +74,7 @@ func TestNoManifestFoundMessage(t *testing.T) {
_, err := findManifest(afero.NewMemMapFs(), "/tmp/x")
require.ErrorIs(t, err, errNoManifestFound)
assert.EqualError(t, err,
"no manifest found in /tmp/x (looked for index.mf)")
"no manifest found in /tmp/x (looked for index.mf and .index.mf)")
}
func TestVerifyRequiredSignerMessages(t *testing.T) {
@@ -111,10 +107,9 @@ func TestVerifyRequiredSignerMessages(t *testing.T) {
// string; the required signer is a fixed value that cannot match it. Requires
// gpg and is skipped where it is absent, as the other signing tests are.
//
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
//nolint:paralleltest // signedChecker calls t.Setenv, which bars t.Parallel
func TestSignerMismatchMessage(t *testing.T) {
chk := signedChecker(t,
signedManifest(t, map[string][]byte{"f.txt": []byte("signed file")}))
chk := signedChecker(t)
embeddedFP, err := chk.ExtractEmbeddedSigningKeyFP(context.Background())
require.NoError(t, err)
@@ -126,10 +121,9 @@ func TestSignerMismatchMessage(t *testing.T) {
" does not match required "+msgFpB)
}
// signedManifest returns a manifest of files signed by a throwaway GPG key
// generated in a temporary GNUPGHOME, which it leaves set for the rest of
// the test.
func signedManifest(t *testing.T, files map[string][]byte) []byte {
// signedChecker builds a Checker over a manifest signed by a throwaway GPG
// key generated in a temporary GNUPGHOME.
func signedChecker(t *testing.T) *mfer.Checker {
t.Helper()
_, err := exec.LookPath("gpg")
@@ -161,31 +155,19 @@ func signedManifest(t *testing.T, files map[string][]byte) []byte {
b := mfer.NewBuilder()
b.SetSigningOptions(&mfer.SigningOptions{KeyID: mfer.GPGKeyID("test@mfer.test")})
for path, content := range files {
_, err = b.AddFile(mfer.RelFilePath(path), mfer.FileSize(len(content)),
mfer.ModTime{}, bytes.NewReader(content), nil)
require.NoError(t, err)
}
content := []byte("signed file")
_, err = b.AddFile("f.txt", mfer.FileSize(len(content)), mfer.ModTime{},
bytes.NewReader(content), nil)
require.NoError(t, err)
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
return buf.Bytes()
}
// signedChecker builds a Checker over manifest, a signed manifest.
func signedChecker(t *testing.T, manifest []byte) *mfer.Checker {
t.Helper()
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/index.mf", manifest, 0o644))
require.NoError(t, afero.WriteFile(fs, "/index.mf", buf.Bytes(), 0o644))
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: "/index.mf",
BasePath: "/",
Fs: fs,
})
chk, err := mfer.NewChecker("/index.mf", "/", fs)
require.NoError(t, err)
require.True(t, chk.IsSigned())
@@ -282,15 +264,10 @@ func TestFetchManifestHTTPStatusMessage(t *testing.T) {
}))
defer server.Close()
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
set := flag.NewFlagSet(cmdFetch, flag.ContinueOnError)
for _, f := range mfa.fetchCommand().Flags {
require.NoError(t, f.Apply(set))
}
set := flag.NewFlagSet("fetch", flag.ContinueOnError)
require.NoError(t, set.Parse([]string{server.URL}))
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
ctx := urfcli.NewContext(nil, set, nil)
// fetchManifestOperation logs to the process-global logger.
@@ -308,11 +285,8 @@ func TestFetchFileHTTPStatusMessage(t *testing.T) {
}))
defer server.Close()
// downloadFile logs each retry of the 500 to the process-global logger.
err := runLocked(func() error {
return downloadFile(context.Background(), testClient(), server.URL+"/x", ".", "x",
&mfer.MFFilePath{}, nil)
})
err := downloadFile(context.Background(), server.URL+"/x", "x",
&mfer.MFFilePath{}, nil)
require.ErrorIs(t, err, errHTTPStatus)
assert.EqualError(t, err, "HTTP 500")
}
@@ -365,7 +339,7 @@ func TestSizeMismatchMessage(t *testing.T) {
// finishDownload returns the size-mismatch error before it touches the
// paths, digest, or entry, so those can be zero here.
err := finishDownload("", "", "", 9, 10, nil, nil, nil, nil)
err := finishDownload("", "", 9, 10, nil, nil, nil, nil)
require.ErrorIs(t, err, errSizeMismatch)
assert.EqualError(t, err, "size mismatch: expected 10 bytes, got 9")
}
internal/cli/fetch.go
+124 -432
View File
@@ -7,19 +7,16 @@ import (
"errors"
"fmt"
"io"
"math/rand/v2"
"net"
"net/http"
"net/url"
"os"
"path"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/dustin/go-humanize"
"github.com/multiformats/go-multihash"
"github.com/spf13/afero"
"github.com/urfave/cli/v2"
"sneak.berlin/go/mfer/internal/log"
"sneak.berlin/go/mfer/mfer"
@@ -48,33 +45,12 @@ const (
bpsPerGbps = 1e9
bpsPerMbps = 1e6
bpsPerKbps = 1e3
// httpTimeout is the default time limit for one HTTP request, from
// connecting to reading the last byte of the body. It also bounds how
// long one file may take to download; fetch's --timeout changes it.
httpTimeout = 10 * time.Minute
// fetchAttempts is how many times fetch tries a request before it
// gives up on a transient failure.
fetchAttempts = 5
// firstRetryDelay is the longest fetch waits before its first retry.
// The limit doubles for each retry after that; the wait itself is
// random up to the limit.
firstRetryDelay = time.Second
// maxRetryAfter is the longest wait a server's Retry-After header can
// ask for. A server asking for longer fails the request at once.
maxRetryAfter = time.Minute
)
var (
// errURLRequired indicates the fetch command was run without a URL
// argument.
errURLRequired = errors.New("URL argument required")
// errInvalidTimeout indicates a fetch --timeout of zero or less, which
// http.Client would take as no time limit at all.
errInvalidTimeout = errors.New("--timeout must be greater than zero")
// errEmptyPath indicates an empty file path in the manifest.
errEmptyPath = errors.New("empty path")
// errAbsolutePath indicates an absolute file path in the manifest.
@@ -91,10 +67,6 @@ var (
// errHashMismatch indicates a downloaded file whose hash matches no
// manifest hash.
errHashMismatch = errors.New("hash mismatch")
// errManifestNameListed indicates a manifest that lists a file where
// fetch saves the manifest.
errManifestNameListed = errors.New(
"manifest lists a file where fetch saves the manifest")
)
// DownloadProgress reports the progress of a single file download.
@@ -106,109 +78,24 @@ type DownloadProgress struct {
ETA time.Duration // Estimated time to completion
}
// retryingClient is the HTTP client fetch makes every request with. It
// waits up to firstDelay, which must be positive, before its first retry
// of a transient failure. Tests shorten both the client's timeout and
// firstDelay.
type retryingClient struct {
client *http.Client
firstDelay time.Duration
}
// get issues a GET for rawURL and passes a 200 OK response to use.
// httpGet issues a GET request for the given URL using the provided
// context and returns the response. The caller must close the body.
//
// A connection error, a timeout, or a 5xx or 429 status is retried, up to
// fetchAttempts tries in all. Before each retry it waits a random time up
// to a limit that doubles each time, unless the server's Retry-After
// header says how long to wait. Any other failure, an error from use
// included, is returned at once and unwrapped; a non-OK status is
// returned as errHTTPStatus. use must start over each time it is called,
// since a retry calls it again with a new response.
func (c retryingClient) get(
ctx context.Context, rawURL string, use func(*http.Response) error,
) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, rawURL, nil)
// Errors are returned unwrapped: this helper replaced direct http.Get
// calls, and each caller already supplies its own context string, so
// adding one here would change user-visible messages.
func httpGet(ctx context.Context, fileURL string) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, fileURL, nil)
if err != nil {
return err
return nil, err
}
delay := c.firstDelay
for attempt := 1; ; attempt++ {
// Wait a random time up to delay, so that clients that failed
// together do not all retry together.
wait := rand.N(delay) //nolint:gosec // G404: jitter, not a secret
var retry bool
resp, err := c.client.Do(req)
switch {
case err != nil:
retry = isConnectionError(err)
case resp.StatusCode == http.StatusOK:
err = use(resp)
retry = isConnectionError(err)
_ = resp.Body.Close()
default:
_ = resp.Body.Close()
err = fmt.Errorf("%w %d", errHTTPStatus, resp.StatusCode)
retry = resp.StatusCode >= http.StatusInternalServerError ||
resp.StatusCode == http.StatusTooManyRequests
after, ok := retryAfter(resp.Header.Get("Retry-After"))
if ok {
wait = after
retry = retry && after <= maxRetryAfter
}
}
if !retry || attempt == fetchAttempts || ctx.Err() != nil {
return err
}
log.Warnf("%s: %s, retrying in %s", rawURL, err, wait.Round(time.Millisecond))
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(wait):
}
delay *= 2
}
}
// isConnectionError reports whether err is a connection error or a
// timeout: a connection that could not be made, was reset, or closed
// early, or a request that ran out of time.
func isConnectionError(err error) bool {
var (
opErr *net.OpError
netErr net.Error
)
return errors.As(err, &opErr) ||
(errors.As(err, &netErr) && netErr.Timeout()) ||
errors.Is(err, io.EOF) || errors.Is(err, io.ErrUnexpectedEOF)
}
// retryAfter returns the wait a Retry-After header value asks for, given
// either in seconds or as an HTTP date. ok is false if there is none.
func retryAfter(value string) (time.Duration, bool) {
seconds, err := strconv.Atoi(value)
if err == nil {
return time.Duration(seconds) * time.Second, true
resp, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
when, err := http.ParseTime(value)
if err == nil {
return time.Until(when), true
}
return 0, false
return resp, nil
}
// reportDownloadProgress renders download progress until the channel
@@ -232,100 +119,46 @@ func reportDownloadProgress(progress <-chan DownloadProgress, done chan<- struct
}
// manifestBaseURL returns the URL of the directory containing the
// manifest.
// manifest, with a trailing slash.
func manifestBaseURL(manifestURL string) (*url.URL, error) {
parsed, err := url.Parse(manifestURL)
baseURL, err := url.Parse(manifestURL)
if err != nil {
return nil, fmt.Errorf("fetch: invalid manifest URL: %w", err)
}
// JoinPath cleans the path it builds, so ".." drops the manifest's
// file name.
return parsed.JoinPath(".."), nil
baseURL.Path = path.Dir(baseURL.Path)
if !strings.HasSuffix(baseURL.Path, "/") {
baseURL.Path += "/"
}
return baseURL, nil
}
// downloadManifestFiles downloads every file in the manifest into dest,
// reporting progress on the progress channel. A file already present in
// dest is skipped. It returns how many files it downloaded and their
// total size.
// downloadManifestFiles downloads every file in the manifest, reporting
// progress on the progress channel.
func downloadManifestFiles(
ctx context.Context,
client retryingClient,
baseURL *url.URL,
dest string,
files []*mfer.MFFilePath,
progress chan<- DownloadProgress,
) (int, int64, error) {
var (
downloaded int
downloadedBytes int64
)
) error {
for _, f := range files {
// Sanitize the path to prevent path traversal attacks
localPath, err := sanitizePath(f.GetPath())
if err != nil {
return 0, 0, fmt.Errorf("invalid path in manifest: %w", err)
return fmt.Errorf("invalid path in manifest: %w", err)
}
if alreadyPresent(dest, localPath, f) {
log.Infof("skipping %s: already present", f.GetPath())
continue
}
// JoinPath takes escaped path text, so a name such as "100%.txt"
// must be escaped first.
fileURL := baseURL.JoinPath(encodeFilePath(f.GetPath())).String()
fileURL := baseURL.String() + encodeFilePath(f.GetPath())
log.Infof("fetching %s", f.GetPath())
err = downloadFile(ctx, client, fileURL, dest, localPath, f, progress)
err = downloadFile(ctx, fileURL, localPath, f, progress)
if err != nil {
return 0, 0, fmt.Errorf("failed to download %s: %w", f.GetPath(), err)
return fmt.Errorf("failed to download %s: %w", f.GetPath(), err)
}
downloaded++
downloadedBytes += f.GetSize()
}
return downloaded, downloadedBytes, nil
}
// alreadyPresent reports whether localPath under dest is a regular file
// with the size and one of the hashes the manifest lists for entry. It
// hashes the whole file, since a matching size alone would accept a
// corrupted or partly written one. A file it cannot read, or reaches only
// through a symlink, is not present: fetch downloads it, and the download
// reports the problem.
func alreadyPresent(dest, localPath string, entry *mfer.MFFilePath) bool {
if checkNoSymlinks(dest, localPath) != nil {
return false
}
path := filepath.Join(dest, localPath)
info, err := os.Lstat(path)
if err != nil || !info.Mode().IsRegular() || info.Size() != entry.GetSize() {
return false
}
// G304: localPath is a relative path that sanitizePath keeps inside
// dest as text, and checkNoSymlinks just found no symlink in it.
f, err := os.Open(path) //nolint:gosec // G304: see comment above
if err != nil {
return false
}
defer func() { _ = f.Close() }()
h := sha256.New()
_, err = io.Copy(h, f)
if err != nil {
return false
}
return verifyDownloadedHash(h.Sum(nil), entry) == nil
return nil
}
func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
@@ -335,37 +168,47 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
return errURLRequired
}
timeout := ctx.Duration(flagTimeout)
if timeout <= 0 {
return errInvalidTimeout
}
inputURL := ctx.Args().Get(0)
manifestURL, err := resolveManifestURL(ctx.Args().Get(0))
manifestURL, err := resolveManifestURL(inputURL)
if err != nil {
return fmt.Errorf("invalid URL: %w", err)
}
client := retryingClient{
client: &http.Client{Timeout: timeout},
firstDelay: firstRetryDelay,
log.Infof("fetching manifest from %s", manifestURL)
// Fetch manifest
resp, err := httpGet(ctx.Context, manifestURL)
if err != nil {
return fmt.Errorf("failed to fetch manifest: %w", err)
}
manifestData, files, err := fetchManifest(ctx, client, manifestURL)
if err != nil {
return err
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("failed to fetch manifest: %w %d",
errHTTPStatus, resp.StatusCode)
}
// Parse manifest
manifest, err := mfer.NewManifestFromReader(resp.Body)
if err != nil {
return fmt.Errorf("failed to parse manifest: %w", err)
}
files := manifest.Files()
log.Infof("manifest contains %d files", len(files))
// Compute base URL (directory containing manifest)
baseURL, err := manifestBaseURL(manifestURL)
if err != nil {
return err
}
dest := ctx.String(flagDest)
err = os.MkdirAll(dest, dirPerms)
if err != nil {
return fmt.Errorf("failed to create destination directory %s: %w", dest, err)
// Calculate total bytes to download
var totalBytes int64
for _, f := range files {
totalBytes += f.GetSize()
}
// Create progress channel and start progress reporter goroutine
@@ -378,8 +221,7 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
startTime := time.Now()
// Download each file
downloaded, downloadedBytes, dlErr := downloadManifestFiles(
ctx.Context, client, baseURL, dest, files, progress)
dlErr := downloadManifestFiles(ctx.Context, baseURL, files, progress)
close(progress)
<-done
@@ -388,150 +230,15 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
return dlErr
}
// Saved only now that every file is in place and verified, so that
// "mfer check" can verify the tree later.
err = saveManifest(dest, manifestData)
if err != nil {
return fmt.Errorf("failed to save manifest: %w", err)
}
// Print summary
elapsed := time.Since(startTime)
avgBytesPerSec := float64(downloadedBytes) / elapsed.Seconds()
avgBytesPerSec := float64(totalBytes) / elapsed.Seconds()
avgRate := formatBitrate(avgBytesPerSec * bitsPerByte)
log.Infof("downloaded %d files (%s) in %.1fs (%s avg), skipped %d already present",
downloaded,
humanize.IBytes(safeUint64(downloadedBytes)),
log.Infof("downloaded %d files (%s) in %.1fs (%s avg)",
len(files),
humanize.IBytes(safeUint64(totalBytes)),
elapsed.Seconds(),
avgRate,
len(files)-downloaded)
log.Infof("saved manifest to %s", filepath.Join(dest, defaultManifestName))
return nil
}
// fetchManifest downloads the manifest at manifestURL and parses it,
// enforcing --require-signature if it is given and refusing a manifest
// that lists a file where it will be saved. It returns the manifest as
// downloaded, to be saved once the files are in place, and the files it
// lists.
func fetchManifest(
ctx *cli.Context, client retryingClient, manifestURL string,
) ([]byte, []*mfer.MFFilePath, error) {
log.Infof("fetching manifest from %s", manifestURL)
// Read the whole manifest before parsing it, so that a connection
// lost partway through is retried rather than reported as a bad
// manifest.
var manifestData []byte
err := client.get(ctx.Context, manifestURL, func(resp *http.Response) error {
var readErr error
manifestData, readErr = io.ReadAll(resp.Body)
return readErr
})
if err != nil {
return nil, nil, fmt.Errorf("failed to fetch manifest: %w", err)
}
// Parse manifest
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(manifestData))
if err != nil {
return nil, nil, fmt.Errorf("failed to parse manifest: %w", err)
}
requiredSigner := ctx.String(flagRequireSignature)
if requiredSigner != "" {
err = verifyFetchedSigner(ctx, manifestData, requiredSigner)
if err != nil {
return nil, nil, err
}
}
files := manifest.Files()
err = checkManifestNameUnlisted(files)
if err != nil {
return nil, nil, err
}
log.Infof("manifest contains %d files", len(files))
return manifestData, files, nil
}
// checkManifestNameUnlisted returns an error if files lists a file or
// directory at the top of the tree under the name fetch saves the
// manifest as, or under that name's temp file. Saving the manifest would
// replace or remove it, or fail once every file was downloaded, leaving a
// tree check rejects. Names are compared ignoring case, since on a
// case-insensitive filesystem INDEX.MF and index.mf are one file.
func checkManifestNameUnlisted(files []*mfer.MFFilePath) error {
for _, f := range files {
top, _, _ := strings.Cut(filepath.Clean(f.GetPath()), string(filepath.Separator))
if strings.EqualFold(top, defaultManifestName) ||
strings.EqualFold(top, tempPathFor(defaultManifestName)) {
return fmt.Errorf("%w: %s", errManifestNameListed, f.GetPath())
}
}
return nil
}
// verifyFetchedSigner enforces --require-signature on the fetched manifest
// exactly as check does. verifyRequiredSigner takes a Checker, which loads
// its manifest from a file, so the manifest is handed to it as a file in
// memory.
func verifyFetchedSigner(
ctx *cli.Context, manifestData []byte, requiredSigner string,
) error {
memFs := afero.NewMemMapFs()
manifestPath := "/" + defaultManifestName
err := afero.WriteFile(memFs, manifestPath, manifestData, filePerms)
if err != nil {
return err
}
chk, err := mfer.NewChecker(&mfer.CheckerOptions{
ManifestPath: manifestPath,
BasePath: "/",
Fs: memFs,
})
if err != nil {
return fmt.Errorf("failed to load manifest: %w", err)
}
return verifyRequiredSigner(ctx.Context, chk, requiredSigner)
}
// saveManifest writes the fetched manifest into dest under the default
// manifest name, the way fetch writes every file: to a new temp file that
// is then renamed into place.
func saveManifest(dest string, manifestData []byte) error {
tmpPath := tempPathFor(defaultManifestName)
out, err := createTempFile(dest, tmpPath)
if err != nil {
return err
}
_, writeErr := out.Write(manifestData)
closeErr := out.Close()
err = errors.Join(writeErr, closeErr)
if err == nil {
err = moveIntoPlace(dest, tmpPath, defaultManifestName)
}
if err != nil {
_ = os.Remove(filepath.Join(dest, tmpPath))
return err
}
avgRate)
return nil
}
@@ -575,15 +282,14 @@ func sanitizePath(p string) (string, error) {
return cleaned, nil
}
// checkNoSymlinks returns an error if any part of p, a path relative to
// dest, already exists under dest as a symlink. dest itself is the user's
// choice and may be one. sanitizePath checks p only as text, so without
// this a symlink inside dest could send a write to p outside of it. Parts
// that do not exist yet are fine: fetch creates them as plain directories
// and files. Call it immediately before each write: a symlink created
// after it returns is not caught.
func checkNoSymlinks(dest, p string) error {
current := dest
// checkNoSymlinks returns an error if any part of the relative path p
// already exists as a symlink. sanitizePath checks p only as text, so
// without this a symlink inside the target directory could send a write
// to p outside of it. Parts that do not exist yet are fine: fetch creates
// them as plain directories and files. Call it immediately before each
// write: a symlink created after it returns is not caught.
func checkNoSymlinks(p string) error {
current := ""
for _, part := range strings.Split(p, string(filepath.Separator)) {
current = filepath.Join(current, part)
@@ -607,7 +313,7 @@ func checkNoSymlinks(dest, p string) error {
// resolveManifestURL takes a URL and returns the manifest URL.
// If the URL already ends with .mf, it's returned as-is.
// Otherwise, the default manifest name is appended.
// Otherwise, index.mf is appended.
func resolveManifestURL(inputURL string) (string, error) {
parsed, err := url.Parse(inputURL)
if err != nil {
@@ -619,7 +325,15 @@ func resolveManifestURL(inputURL string) (string, error) {
return inputURL, nil
}
return parsed.JoinPath(defaultManifestName).String(), nil
// Ensure path ends with /
if !strings.HasSuffix(parsed.Path, "/") {
parsed.Path += "/"
}
// Append index.mf
parsed.Path += "index.mf"
return parsed.String(), nil
}
// progressWriter wraps an io.Writer and reports progress to a channel.
@@ -723,14 +437,12 @@ func verifyDownloadedHash(digest []byte, entry *mfer.MFFilePath) error {
return errHashMismatch
}
// downloadFile downloads a URL to localPath, a path relative to dest, with
// hash verification. It downloads to a temporary file, verifies the hash,
// then renames to the final path. Progress is reported via the progress
// channel.
// downloadFile downloads a URL to a local file path with hash verification.
// It downloads to a temporary file, verifies the hash, then renames to the final path.
// Progress is reported via the progress channel.
func downloadFile(
ctx context.Context,
client retryingClient,
fileURL, dest, localPath string,
fileURL, localPath string,
entry *mfer.MFFilePath,
progress chan<- DownloadProgress,
) error {
@@ -744,13 +456,11 @@ func downloadFile(
// Create parent directories if needed
dir := filepath.Dir(localPath)
if dir != "" && dir != "." {
err = checkNoSymlinks(dest, dir)
err = checkNoSymlinks(dir)
if err != nil {
return err
}
dir = filepath.Join(dest, dir)
err = os.MkdirAll(dir, dirPerms)
if err != nil {
return fmt.Errorf("failed to create directory %s: %w", dir, err)
@@ -759,65 +469,18 @@ func downloadFile(
tmpPath := tempPathFor(localPath)
return client.get(ctx, fileURL, func(resp *http.Response) error {
return saveResponse(resp, dest, tmpPath, localPath, entry, progress)
})
}
// createTempFile creates tmpPath, a path relative to dest, as a new empty
// file.
func createTempFile(dest, tmpPath string) (*os.File, error) {
err := checkNoSymlinks(dest, tmpPath)
// Fetch file
resp, err := httpGet(ctx, fileURL)
if err != nil {
return nil, err
return fmt.Errorf("HTTP request failed: %w", err)
}
path := filepath.Join(dest, tmpPath)
defer func() { _ = resp.Body.Close() }()
// Remove whatever is at tmpPath, such as a leftover from an
// interrupted run, rather than write into it: it may be a hard link
// to a file outside dest, and removing a hard link removes only this
// name. If the removal fails, O_EXCL below makes the create fail.
_ = os.Remove(path)
// Create the temp file only if nothing is at tmpPath (O_EXCL).
//
// G304: tmpPath is a relative path that sanitizePath keeps inside dest
// as text, and checkNoSymlinks just found no symlink in it.
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
path, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
if err != nil {
return nil, fmt.Errorf("failed to create temp file: %w", err)
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("%w %d", errHTTPStatus, resp.StatusCode)
}
return out, nil
}
// moveIntoPlace renames tmpPath to localPath, both relative to dest.
func moveIntoPlace(dest, tmpPath, localPath string) error {
err := checkNoSymlinks(dest, localPath)
if err != nil {
return err
}
err = os.Rename(filepath.Join(dest, tmpPath), filepath.Join(dest, localPath))
if err != nil {
return fmt.Errorf("failed to rename temp file: %w", err)
}
return nil
}
// saveResponse writes resp's body to tmpPath, verifies it against entry,
// and renames it to localPath, both paths relative to dest. It starts a
// new temp file each time and removes it on failure, so a retry after a
// failed try never appends to or keeps a partial file.
func saveResponse(
resp *http.Response,
dest, tmpPath, localPath string,
entry *mfer.MFFilePath,
progress chan<- DownloadProgress,
) error {
// Determine expected size
expectedSize := entry.GetSize()
@@ -826,11 +489,29 @@ func saveResponse(
totalBytes = expectedSize
}
out, err := createTempFile(dest, tmpPath)
err = checkNoSymlinks(tmpPath)
if err != nil {
return err
}
// Remove whatever is at tmpPath, such as a leftover from an
// interrupted run, rather than write into it: it may be a hard link
// to a file outside the target directory, and removing a hard link
// removes only this name. If the removal fails, O_EXCL below makes
// the create fail.
_ = os.Remove(tmpPath)
// Create the temp file only if nothing is at tmpPath (O_EXCL).
//
// G304: tmpPath is a relative path that sanitizePath keeps inside the
// target directory as text, and checkNoSymlinks just found no symlink
// in it.
out, err := os.OpenFile( //nolint:gosec // G304: see comment above
tmpPath, os.O_RDWR|os.O_CREATE|os.O_EXCL, filePerms)
if err != nil {
return fmt.Errorf("failed to create temp file: %w", err)
}
// Set up hash computation
h := sha256.New()
@@ -850,10 +531,10 @@ func saveResponse(
closeErr := out.Close()
err = finishDownload(
dest, tmpPath, localPath, written, expectedSize, h.Sum(nil), entry,
tmpPath, localPath, written, expectedSize, h.Sum(nil), entry,
copyErr, closeErr)
if err != nil {
_ = os.Remove(filepath.Join(dest, tmpPath))
_ = os.Remove(tmpPath)
return err
}
@@ -864,7 +545,7 @@ func saveResponse(
// finishDownload validates the copy result, verifies size and hash, and
// moves the temp file into place. On error the caller removes tmpPath.
func finishDownload(
dest, tmpPath, localPath string,
tmpPath, localPath string,
written, expectedSize int64,
digest []byte,
entry *mfer.MFFilePath,
@@ -890,5 +571,16 @@ func finishDownload(
return err
}
return moveIntoPlace(dest, tmpPath, localPath)
err = checkNoSymlinks(localPath)
if err != nil {
return err
}
// Rename temp file to final path
err = os.Rename(tmpPath, localPath)
if err != nil {
return fmt.Errorf("failed to rename temp file: %w", err)
}
return nil
}
internal/cli/fetch_test.go
+26 -799
View File
@@ -4,26 +4,16 @@ package cli
import (
"bytes"
"context"
"flag"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"slices"
"strconv"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
urfcli "github.com/urfave/cli/v2"
"sneak.berlin/go/mfer/mfer"
)
@@ -224,68 +214,6 @@ func fetchTestHandler(
}
}
// manifestOf scans a tree holding files and returns its manifest bytes.
func manifestOf(t *testing.T, files map[string][]byte) []byte {
t.Helper()
sourceFs := afero.NewMemMapFs()
for p, content := range files {
require.NoError(t, sourceFs.MkdirAll(filepath.Dir("/"+p), 0o755))
require.NoError(t, afero.WriteFile(sourceFs, "/"+p, content, 0o644))
}
return scanToManifest(t, sourceFs)
}
// filesUnder returns the content of every file under dir, by its path
// relative to dir.
func filesUnder(t *testing.T, dir string) map[string][]byte {
t.Helper()
files := map[string][]byte{}
err := filepath.WalkDir(dir, func(path string, entry os.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
rel, err := filepath.Rel(dir, path)
if err != nil {
return err
}
content, err := os.ReadFile(path) //nolint:gosec // test-controlled path
files[filepath.ToSlash(rel)] = content
return err
})
require.NoError(t, err)
return files
}
// testClient returns the client tests download with. Its retries wait
// milliseconds rather than seconds.
func testClient() retryingClient {
return retryingClient{
client: &http.Client{Timeout: 10 * time.Second},
firstDelay: time.Millisecond,
}
}
// getNothing calls client.get for rawURL with a use that reads nothing,
// under a context that expires after timeout. It holds runMu, since get
// logs each retry to the process-global logger, and starts the timeout
// only once it has the lock.
func getNothing(client retryingClient, rawURL string, timeout time.Duration) error {
return runLocked(func() error {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
return client.get(ctx, rawURL, func(*http.Response) error { return nil })
})
}
//nolint:paralleltest // changes the process-global working directory
func TestFetchFromHTTP(t *testing.T) {
// Create source filesystem with test files
@@ -338,8 +266,7 @@ func TestFetchFromHTTP(t *testing.T) {
require.NoError(t, err)
fileURL := baseURL + f.GetPath()
err = downloadFile(context.Background(), testClient(),
fileURL, ".", localPath, f, progress)
err = downloadFile(context.Background(), fileURL, localPath, f, progress)
require.NoError(t, err, "failed to download %s", f.GetPath())
}
@@ -386,8 +313,8 @@ func TestFetchHashMismatch(t *testing.T) {
chdirTemp(t)
// Try to download - should fail with hash mismatch
err = downloadFile(context.Background(), testClient(),
server.URL+"/file.txt", ".", testFileTxt, files[0], nil)
err = downloadFile(context.Background(),
server.URL+"/file.txt", testFileTxt, files[0], nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "mismatch")
@@ -432,8 +359,8 @@ func TestFetchSizeMismatch(t *testing.T) {
chdirTemp(t)
// Try to download - should fail with size mismatch
err = downloadFile(context.Background(), testClient(),
server.URL+"/file.txt", ".", testFileTxt, files[0], nil)
err = downloadFile(context.Background(),
server.URL+"/file.txt", testFileTxt, files[0], nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "size mismatch")
@@ -490,8 +417,8 @@ func TestFetchProgress(t *testing.T) {
}()
// Download
err = downloadFile(context.Background(), testClient(),
server.URL+"/large.txt", ".", "large.txt", files[0], progress)
err = downloadFile(context.Background(),
server.URL+"/large.txt", "large.txt", files[0], progress)
close(progress)
<-done
@@ -514,51 +441,24 @@ func TestFetchProgress(t *testing.T) {
assert.Equal(t, content, downloaded)
}
// TestFetchRefusesSymlinks runs fetch with --dest naming a directory other
// than the current one, which holds a symlink pointing outside it, in each
// place fetch writes: a parent directory, the temp file, the file itself,
// which the temp file is renamed onto, and the saved manifest's temp file
// and final name; and once as a directory inside a plain directory. The
// fetch must fail, and neither the outside directory nor the current one
// may change.
// TestFetchRefusesSymlinks runs fetch into a destination directory that
// holds a symlink pointing outside it, in each of the three places fetch
// writes: a parent directory, the temp file, and the file itself, which
// the temp file is renamed onto; and once as a directory inside a plain
// directory. The fetch must fail and nothing outside may change.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchRefusesSymlinks(t *testing.T) {
// What a link standing for a file points to: a file outside that does
// not exist yet.
const newFile = "new.txt"
tests := []struct {
name string
entry string // the manifest's only file
link string // symlink placed in the destination directory
target string // what link points to, relative to the outside directory
failure string // what fetch reports it was doing when it found link
name string
entry string // the manifest's only file
link string // symlink placed in the destination directory
target string // what link points to, relative to the outside directory
}{
{
"parent directory", "sub/deeper/file.txt", "sub", ".",
"failed to download sub/deeper/file.txt",
},
{
"directory inside a plain directory", "docs/data/passwd", "docs/data", ".",
"failed to download docs/data/passwd",
},
{
"temp file", testFileTxt, ".file.txt.tmp", newFile,
"failed to download " + testFileTxt,
},
{
"file", testFileTxt, testFileTxt, newFile,
"failed to download " + testFileTxt,
},
{
"manifest temp file", testFileTxt, tempPathFor(defaultManifestName), newFile,
"failed to save manifest",
},
{
"manifest", testFileTxt, defaultManifestName, newFile,
"failed to save manifest",
},
{"parent directory", "sub/deeper/file.txt", "sub", "."},
{"directory inside a plain directory", "docs/data/passwd", "docs/data", "."},
{"temp file", testFileTxt, ".file.txt.tmp", "new.txt"},
{"file", testFileTxt, testFileTxt, "new.txt"},
}
for _, tt := range tests {
@@ -573,61 +473,23 @@ func TestFetchRefusesSymlinks(t *testing.T) {
defer server.Close()
outside := t.TempDir()
cwd := chdirTemp(t)
dest := t.TempDir()
link := filepath.Join(dest, tt.link)
require.NoError(t, os.MkdirAll(filepath.Dir(link), 0o750))
require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), link))
chdirTemp(t)
require.NoError(t, os.MkdirAll(filepath.Dir(tt.link), 0o750))
require.NoError(t, os.Symlink(filepath.Join(outside, tt.target), tt.link))
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
}, afero.NewOsFs())
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
tt.failure+": symlink in path not allowed: "+link)
assert.Contains(t, testStderr(t, opts), "failed to download "+tt.entry+
": symlink in path not allowed: "+tt.link)
written, err := os.ReadDir(outside)
require.NoError(t, err)
assert.Empty(t, written, "fetch wrote outside the destination")
written, err = os.ReadDir(cwd)
require.NoError(t, err)
assert.Empty(t, written, "fetch wrote to the current directory")
})
}
}
// TestFetchDoesNotSkipThroughSymlink runs fetch with --dest holding a
// symlink to a directory outside it, where the file the manifest lists
// through that symlink already sits with the listed content. fetch must
// not take that file as already present: it must fail on the symlink, as
// the download would, and leave the outside file alone.
func TestFetchDoesNotSkipThroughSymlink(t *testing.T) {
t.Parallel()
content := []byte("fetched")
files := map[string][]byte{"sub/" + testFileTxt: content}
server := httptest.NewServer(fetchTestHandler(manifestOf(t, files), files))
defer server.Close()
outside := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(outside, testFileTxt), content, 0o600))
dest := t.TempDir()
link := filepath.Join(dest, "sub")
require.NoError(t, os.Symlink(outside, link))
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
}, afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts),
"failed to download sub/"+testFileTxt+": symlink in path not allowed: "+link)
assert.Equal(t, map[string][]byte{testFileTxt: content}, filesUnder(t, outside))
}
// TestFetchReplacesHardLinkAtTempName runs fetch into a destination
// directory that holds, at the temp file's name, a hard link to a file
// outside it. To fetch that is an ordinary leftover from an interrupted
@@ -661,638 +523,3 @@ func TestFetchReplacesHardLinkAtTempName(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, "outside", string(outside), "fetch wrote outside the destination")
}
// TestGetRetriesTransientStatusesOnly answers every request with one
// status and counts the requests: a 5xx or 429 is retried until
// fetchAttempts runs out, and any other status fails at once.
func TestGetRetriesTransientStatusesOnly(t *testing.T) {
t.Parallel()
tests := []struct {
status int
requests int32
}{
{http.StatusNotFound, 1},
{http.StatusForbidden, 1},
{http.StatusInternalServerError, fetchAttempts},
{http.StatusServiceUnavailable, fetchAttempts},
{http.StatusTooManyRequests, fetchAttempts},
}
for _, tt := range tests {
t.Run(http.StatusText(tt.status), func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
requests.Add(1)
w.WriteHeader(tt.status)
}))
defer server.Close()
err := getNothing(testClient(), server.URL, 10*time.Second)
require.ErrorIs(t, err, errHTTPStatus)
require.EqualError(t, err, fmt.Sprintf("HTTP %d", tt.status))
assert.Equal(t, tt.requests, requests.Load())
})
}
}
// TestGetTimesOutOnStalledServer points get at a server that takes a
// request and never answers it. The try must give up at the client's
// timeout and be retried; when every try stalls, get must return a
// timeout error rather than hang.
func TestGetTimesOutOnStalledServer(t *testing.T) {
t.Parallel()
tests := []struct {
name string
stallEvery bool
}{
{"first request stalls", false},
{"every request stalls", true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
stop := make(chan struct{})
server := httptest.NewServer(
http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
if requests.Add(1) == 1 || tt.stallEvery {
select {
case <-r.Context().Done():
case <-stop:
}
}
}))
defer server.Close()
defer close(stop)
client := testClient()
client.client.Timeout = 50 * time.Millisecond
err := getNothing(client, server.URL, 10*time.Second)
if !tt.stallEvery {
require.NoError(t, err)
return
}
var netErr net.Error
require.ErrorAs(t, err, &netErr)
assert.True(t, netErr.Timeout(), "want a timeout, got %v", err)
})
}
}
// TestGetHonorsRetryAfter answers the first request with a 503 and a
// Retry-After header, and any later one with 200 OK. The client's own
// wait before a retry is an hour, so get finishes in time only if it
// waits as long as Retry-After says instead. A Retry-After longer than
// maxRetryAfter must fail at once.
func TestGetHonorsRetryAfter(t *testing.T) {
t.Parallel()
tests := []struct {
name string
value string
requests int32
wantErr bool
}{
{"seconds", "0", 2, false},
{"HTTP date", time.Now().Add(-time.Minute).UTC().Format(http.TimeFormat), 2, false},
{"too long", strconv.Itoa(int((maxRetryAfter + time.Second).Seconds())), 1, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
var requests atomic.Int32
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
if requests.Add(1) == 1 {
w.Header().Set("Retry-After", tt.value)
w.WriteHeader(http.StatusServiceUnavailable)
}
}))
defer server.Close()
client := testClient()
client.firstDelay = time.Hour
err := getNothing(client, server.URL, 10*time.Second)
if tt.wantErr {
require.ErrorIs(t, err, errHTTPStatus)
} else {
require.NoError(t, err)
}
assert.Equal(t, tt.requests, requests.Load())
})
}
}
// TestGetStopsWaitingWhenCanceled gives get a server that always answers
// 503 and an hour's wait before each retry, then lets the context expire
// during that wait. get returning at all shows it stopped waiting.
func TestGetStopsWaitingWhenCanceled(t *testing.T) {
t.Parallel()
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusServiceUnavailable)
}))
defer server.Close()
client := testClient()
client.firstDelay = time.Hour
err := getNothing(client, server.URL, 50*time.Millisecond)
require.ErrorIs(t, err, context.DeadlineExceeded)
}
// TestDownloadFileRetriesToSuccess serves a file that fails twice before
// it succeeds: first with a 503, then with a body cut off halfway. The
// download must end with the whole, verified file and no temp file.
//
//nolint:paralleltest // changes the process-global working directory
func TestDownloadFileRetriesToSuccess(t *testing.T) {
content := []byte("the whole file, every byte of it")
manifest, err := mfer.NewManifestFromReader(bytes.NewReader(
manifestOf(t, map[string][]byte{testFileTxt: content})))
require.NoError(t, err)
var requests atomic.Int32
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
switch requests.Add(1) {
case 1:
w.WriteHeader(http.StatusServiceUnavailable)
case 2:
// Promise the whole file but send half of it; the server
// then closes the connection.
w.Header().Set("Content-Length", strconv.Itoa(len(content)))
_, _ = w.Write(content[:len(content)/2])
default:
_, _ = w.Write(content)
}
}))
defer server.Close()
chdirTemp(t)
err = downloadFile(context.Background(), testClient(),
server.URL+"/"+testFileTxt, ".", testFileTxt, manifest.Files()[0], nil)
require.NoError(t, err)
assert.Equal(t, int32(3), requests.Load())
fetched, err := os.ReadFile(testFileTxt)
require.NoError(t, err)
assert.Equal(t, content, fetched)
_, err = os.Stat(".file.txt.tmp")
assert.True(t, os.IsNotExist(err), "temp file left behind")
}
// TestFetchBaseURLForms fetches one tree by its directory URL with and
// without a trailing slash, with a query string, and by its manifest URL.
// Each must request the same manifest and file paths.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchBaseURLForms(t *testing.T) {
files := map[string][]byte{"one.txt": []byte("1"), "sub/two.txt": []byte("2")}
tree := http.StripPrefix("/tree", fetchTestHandler(manifestOf(t, files), files))
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
inputs := []string{"/tree", "/tree/", "/tree?key=value", "/tree/index.mf"}
for _, input := range inputs {
t.Run(input, func(t *testing.T) {
mu.Lock()
requested = nil
mu.Unlock()
chdirTemp(t)
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL + input},
afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
mu.Lock()
defer mu.Unlock()
assert.ElementsMatch(t,
[]string{"/tree/index.mf", "/tree/one.txt", "/tree/sub/two.txt"}, requested)
})
}
}
// TestFetchEscapedPaths fetches files whose names need escaping in a URL,
// one of them a name that is itself an escape sequence. Each must arrive
// under its own name with its own content.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchEscapedPaths(t *testing.T) {
files := map[string][]byte{
"a b.txt": []byte("space"),
"a%20b.txt": []byte("percent sign, two, zero"),
"100%.txt": []byte("percent sign"),
"q?x=1#frag.txt": []byte("question mark and hash"),
"dir with space/sub.txt": []byte("directory with a space"),
}
server := httptest.NewServer(fetchTestHandler(manifestOf(t, files), files))
defer server.Close()
chdirTemp(t)
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
for name, content := range files {
fetched, err := os.ReadFile(name) //nolint:gosec // test-controlled path
require.NoError(t, err)
assert.Equal(t, content, fetched, name)
}
}
// TestFetchTree runs fetch on a tree with nested directories. Every file
// the manifest lists must land under its own path with its own content,
// beside the manifest, and nothing else may be left in the destination.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchTree(t *testing.T) {
files := map[string][]byte{
"top.txt": []byte("at the top"),
"sub/one.txt": []byte("one level down"),
"sub/deeper/two.txt": []byte("two levels down"),
"other/deep/est.txt": []byte("in a second directory"),
}
manifest := manifestOf(t, files)
server := httptest.NewServer(fetchTestHandler(manifest, files))
defer server.Close()
dest := chdirTemp(t)
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
want := maps.Clone(files)
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
}
// TestFetchFailsOnHashMismatch runs fetch against a server that serves a
// file with the size the manifest lists but different content. fetch must
// exit non-zero and leave no file in the destination.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchFailsOnHashMismatch(t *testing.T) {
listed := map[string][]byte{testDirFile: []byte("original")}
served := map[string][]byte{testDirFile: []byte("tampered")}
server := httptest.NewServer(fetchTestHandler(manifestOf(t, listed), served))
defer server.Close()
dest := chdirTemp(t)
opts := testOpts([]string{testApp, cmdFetch, "-q", server.URL}, afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Empty(t, filesUnder(t, dest))
}
// TestFetchIntoPartlyFilledDestination runs fetch where an interrupted
// fetch of an older version of the tree left one file current, one file
// out of date and one half written to its temp file, beside a file the
// manifest does not list. fetch skips the current file and downloads the
// other two. The out-of-date file has the same size as the new version,
// so only its hash shows that it must be replaced. The file the manifest
// does not list is left alone, and the manifest is saved beside the files.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchIntoPartlyFilledDestination(t *testing.T) {
files := map[string][]byte{
"current.txt": []byte("already fetched"),
"sub/changed.txt": []byte("new version"),
"sub/partial.txt": []byte("cut off partway"),
}
unlisted := []byte("not in the manifest")
manifest := manifestOf(t, files)
tree := fetchTestHandler(manifest, files)
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
dest := chdirTemp(t)
require.NoError(t, os.MkdirAll("sub", 0o750))
require.NoError(t, os.WriteFile("current.txt", files["current.txt"], 0o600))
require.NoError(t, os.WriteFile("sub/changed.txt", []byte("old version"), 0o600))
require.NoError(t,
os.WriteFile(tempPathFor("sub/partial.txt"), []byte("cut off"), 0o600))
require.NoError(t, os.WriteFile("unlisted.txt", unlisted, 0o600))
opts := testOpts([]string{testApp, cmdFetch, server.URL}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
assert.Contains(t, testStderr(t, opts), "skipping current.txt: already present")
want := maps.Clone(files)
want["unlisted.txt"] = unlisted
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
mu.Lock()
defer mu.Unlock()
assert.ElementsMatch(t, []string{
"/" + defaultManifestName, "/sub/changed.txt", "/sub/partial.txt",
}, requested)
}
// TestFetchIntoDest fetches a tree with --dest into a directory that does
// not exist yet. The files and the manifest must land there and nowhere
// else, and check must pass on the result with no extra files. A second
// fetch into the same directory must download nothing but the manifest.
//
//nolint:paralleltest // changes the process-global working directory
func TestFetchIntoDest(t *testing.T) {
files := map[string][]byte{
"top.txt": []byte("at the top"),
"sub/one.txt": []byte("one level down"),
}
manifest := manifestOf(t, files)
tree := fetchTestHandler(manifest, files)
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
cwd := chdirTemp(t)
dest := filepath.Join(t.TempDir(), "mirror")
fetch := []string{testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL}
opts := testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
want := maps.Clone(files)
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
assert.Empty(t, filesUnder(t, cwd), "fetch wrote outside --dest")
check := testOpts([]string{
testApp, cmdCheck, "-q", testFlagBase, dest, testFlagNoExtra,
filepath.Join(dest, defaultManifestName),
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(check), testStderr(t, check))
mu.Lock()
requested = nil
mu.Unlock()
opts = testOpts(fetch, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
assert.Equal(t, want, filesUnder(t, dest))
mu.Lock()
defer mu.Unlock()
assert.Equal(t, []string{"/" + defaultManifestName}, requested)
}
// TestFetchRequireSignature runs fetch with --require-signature. A
// manifest that is unsigned, or signed by another key, must stop fetch
// with check's message before it downloads or writes anything; the
// required key lets it through. The signed cases need gpg and are skipped
// without it, as the other signing tests are.
//
//nolint:paralleltest // signedManifest calls t.Setenv, which bars t.Parallel
func TestFetchRequireSignature(t *testing.T) {
files := map[string][]byte{testFileTxt: []byte("signed file")}
t.Run("unsigned", func(t *testing.T) {
assertFetchRefused(t, manifestOf(t, files), files,
"manifest is not signed, but signature from "+msgFpA+" is required",
"--"+flagRequireSignature, msgFpA)
})
t.Run("signed", func(t *testing.T) {
manifest := signedManifest(t, files)
signer, err := signedChecker(t, manifest).
ExtractEmbeddedSigningKeyFP(context.Background())
require.NoError(t, err)
assertFetchRefused(t, manifest, files,
"embedded signing key fingerprint "+signer+" does not match required "+msgFpB,
"--"+flagRequireSignature, msgFpB)
server := httptest.NewServer(fetchTestHandler(manifest, files))
defer server.Close()
dest := t.TempDir()
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest,
"--" + flagRequireSignature, signer, server.URL,
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
assert.Equal(t, files[testFileTxt], filesUnder(t, dest)[testFileTxt])
})
}
// TestFetchRefusesListedManifestName fetches manifests that list, at the
// top of the tree, the name fetch saves the manifest under or that name's
// temp file: as a file, as a directory, in capitals, and with a leading
// "./". Saving the manifest would replace or remove what is listed there,
// so fetch must refuse the manifest before it creates the destination or
// requests any file.
func TestFetchRefusesListedManifestName(t *testing.T) {
t.Parallel()
for _, listed := range []string{
defaultManifestName,
tempPathFor(defaultManifestName),
defaultManifestName + "/" + testFileTxt,
"INDEX.MF",
"./" + defaultManifestName,
} {
t.Run(listed, func(t *testing.T) {
t.Parallel()
// Built directly rather than scanned, since a scan lists no
// hidden files and never a path starting with "./".
content := []byte("listed")
builder := mfer.NewBuilder()
_, err := builder.AddFile(mfer.RelFilePath(listed), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
require.NoError(t, err)
var manifest bytes.Buffer
require.NoError(t, builder.Build(context.Background(), &manifest))
assertFetchRefused(t, manifest.Bytes(), map[string][]byte{listed: content},
"manifest lists a file where fetch saves the manifest: "+listed)
})
}
}
// assertFetchRefused serves manifest, a manifest of files, and fetches it
// with flags into a directory that does not exist yet. fetch must fail
// with message after requesting only the manifest, and must not create
// the directory.
func assertFetchRefused(
t *testing.T, manifest []byte, files map[string][]byte,
message string, flags ...string,
) {
t.Helper()
tree := fetchTestHandler(manifest, files)
var (
mu sync.Mutex
requested []string
)
server := httptest.NewServer(
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
requested = append(requested, r.URL.Path)
mu.Unlock()
tree.ServeHTTP(w, r)
}))
defer server.Close()
dest := filepath.Join(t.TempDir(), "mirror")
opts := testOpts(slices.Concat(
[]string{testApp, cmdFetch, "-q", "--" + flagDest, dest}, flags, []string{server.URL},
), afero.NewOsFs())
assert.Equal(t, 1, runCLI(opts))
assert.Contains(t, testStderr(t, opts), message)
assert.NoDirExists(t, dest, "fetch created the destination before refusing")
mu.Lock()
defer mu.Unlock()
assert.Equal(t, []string{"/" + defaultManifestName}, requested)
}
// TestFetchTimeoutFlag runs fetch with --timeout against a server that
// never answers. Without the flag's limit the request would wait forever;
// once fetch gives up on it, the server cancels fetch's context so that
// fetch returns instead of retrying.
func TestFetchTimeoutFlag(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
server := httptest.NewServer(
http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
<-r.Context().Done() // fetch gave up on this request
cancel()
}))
defer server.Close()
mfa := &CLIApp{Fs: afero.NewMemMapFs()}
set := flag.NewFlagSet(cmdFetch, flag.ContinueOnError)
for _, f := range mfa.fetchCommand().Flags {
require.NoError(t, f.Apply(set))
}
require.NoError(t, set.Parse([]string{"--" + flagTimeout, "100ms", server.URL}))
cliCtx := urfcli.NewContext(nil, set, nil)
cliCtx.Context = ctx
// fetchManifestOperation logs to the process-global logger.
err := runLocked(func() error { return mfa.fetchManifestOperation(cliCtx) })
require.Error(t, err)
}
// TestFetchRejectsTimeoutOfZeroOrLess runs fetch with a --timeout of zero
// and of less than zero. http.Client takes either as no time limit at all,
// so fetch must refuse it before making any request.
func TestFetchRejectsTimeoutOfZeroOrLess(t *testing.T) {
t.Parallel()
var requests atomic.Int32
server := httptest.NewServer(
http.HandlerFunc(func(http.ResponseWriter, *http.Request) {
requests.Add(1)
}))
defer server.Close()
for _, timeout := range []string{"0", "-1s"} {
opts := testOpts(
[]string{testApp, cmdFetch, "--" + flagTimeout + "=" + timeout, server.URL},
afero.NewMemMapFs())
assert.Equal(t, 1, runCLI(opts), timeout)
assert.Contains(t, testStderr(t, opts), errInvalidTimeout.Error(), timeout)
}
assert.Zero(t, requests.Load(), "fetch made a request")
}
internal/cli/freshen.go
+10 -35
View File
@@ -7,7 +7,6 @@ import (
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"time"
@@ -57,7 +56,7 @@ type freshenEntry struct {
type freshenScanner struct {
fs afero.Fs
absBase string
excluded []fs.FileInfo // files left out of the listing
manifestBase string
includeDotfiles bool
followSymlinks bool
showProgress bool
@@ -157,6 +156,11 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
"freshen: failed to compute relative path for %s: %w", path, err)
}
// Skip the manifest file itself
if relPath == s.manifestBase || relPath == "."+s.manifestBase {
return nil
}
// Handle dotfiles
if !s.includeDotfiles && mfer.IsHiddenPath(filepath.ToSlash(relPath)) {
if info.IsDir() {
@@ -181,12 +185,6 @@ func (s *freshenScanner) walk(path string, info fs.FileInfo, walkErr error) erro
info = realInfo
}
for _, excluded := range s.excluded {
if os.SameFile(info, excluded) {
return nil
}
}
s.scanCount++
// Check against existing manifest
@@ -309,7 +307,7 @@ func (h *freshenHasher) processEntry(e *freshenEntry) error {
func writeFreshenedManifest(
ctx context.Context, afs afero.Fs, builder *mfer.Builder, manifestPath string,
) error {
tmpPath := manifestTempPath(manifestPath)
tmpPath := manifestPath + ".tmp"
outFile, err := afs.Create(tmpPath)
if err != nil {
@@ -366,22 +364,10 @@ func (mfa *CLIApp) freshenScan(
startScan := time.Now()
showProgress := ctx.Bool("progress")
// Leave out the manifest and a temp file left by an interrupted run,
// as gen does. A path that cannot be stat'd, normally because no file
// is there, needs no leaving out.
var excluded []fs.FileInfo
for _, p := range []string{manifestPath, manifestTempPath(manifestPath)} {
info, err := mfa.Fs.Stat(p)
if err == nil {
excluded = append(excluded, info)
}
}
scanner := &freshenScanner{
fs: mfa.Fs,
absBase: absBase,
excluded: excluded,
manifestBase: filepath.Base(manifestPath),
includeDotfiles: ctx.Bool("include-dotfiles"),
followSymlinks: ctx.Bool("follow-symlinks"),
showProgress: showProgress,
@@ -459,10 +445,7 @@ func (mfa *CLIApp) loadExistingEntries(
log.Infof("loading manifest from %s", manifestPath)
// Load existing manifest
manifest, err := mfer.NewManifestFromFile(&mfer.ManifestFromFileOptions{
Path: manifestPath,
Fs: mfa.Fs,
})
manifest, err := mfer.NewManifestFromFile(mfa.Fs, manifestPath)
if err != nil {
return nil, fmt.Errorf("failed to load manifest: %w", err)
}
@@ -629,15 +612,7 @@ func addExistingToBuilder(b *mfer.Builder, entry *mfer.MFFilePath) error {
return nil
}
err := b.AddFileWithHash(mfer.RelFilePath(entry.GetPath()),
return b.AddFileWithHash(mfer.RelFilePath(entry.GetPath()),
mfer.FileSize(entry.GetSize()), mfer.ModTime(mtime),
entry.GetHashes()[0].GetMultiHash())
if err != nil {
return fmt.Errorf(
"manifest entry %s: %w (regenerate the manifest with mfer generate)",
entry.GetPath(), err,
)
}
return nil
}
internal/cli/freshen_test.go
+46 -244
View File
@@ -2,19 +2,15 @@
package cli
import (
"crypto/sha256"
"maps"
"bytes"
"context"
"os"
"path/filepath"
"slices"
"testing"
"time"
"github.com/multiformats/go-multihash"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/proto"
"sneak.berlin/go/mfer/mfer"
)
@@ -32,236 +28,71 @@ func (s stubFileInfo) ModTime() time.Time { return s.mtime }
func (s stubFileInfo) IsDir() bool { return false }
func (s stubFileInfo) Sys() any { return nil }
// setupFreshenDir writes files, by path and content, into a fresh temp
// dir and runs gen on it. It returns the temp dir and the path of the
// manifest gen wrote there.
func setupFreshenDir(
t *testing.T, fs afero.Fs, files map[string]string,
) (string, string) {
// setupFreshenDir populates /testdir with two files, scans it, and
// writes the resulting manifest to /testdir/.index.mf.
func setupFreshenDir(t *testing.T, fs afero.Fs) {
t.Helper()
root := t.TempDir()
manifestPath := filepath.Join(root, defaultManifestName)
require.NoError(t, fs.MkdirAll(testDir, 0o755))
writeTestFile(t, fs, testFile1, "content1")
writeTestFile(t, fs, "/testdir/file2.txt", "content2")
for path, content := range files {
require.NoError(t, fs.MkdirAll(filepath.Dir(filepath.Join(root, path)), 0o750))
writeTestFile(t, fs, filepath.Join(root, path), content)
}
// Generate initial manifest
opts := &mfer.ScannerOptions{Fs: fs}
s := mfer.NewScannerWithOptions(opts)
require.NoError(t, s.EnumeratePath(testDir, nil))
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", manifestPath, root}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
var manifestBuf bytes.Buffer
return root, manifestPath
require.NoError(t, s.ToManifest(context.Background(), &manifestBuf, nil))
// Write manifest to filesystem
require.NoError(t,
afero.WriteFile(fs, "/testdir/.index.mf", manifestBuf.Bytes(), 0o644))
}
// runFreshen runs freshen on the manifest at manifestPath for the tree
// at root and requires it to succeed.
func runFreshen(t *testing.T, fs afero.Fs, root, manifestPath string) {
t.Helper()
opts := testOpts([]string{
testApp, cmdFreshen, "-q", testFlagBase, root, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
}
// manifestFiles returns the file entries of the manifest at path.
func manifestFiles(t *testing.T, fs afero.Fs, path string) []*mfer.MFFilePath {
t.Helper()
manifest, err := mfer.NewManifestFromFile(&mfer.ManifestFromFileOptions{
Path: path,
Fs: fs,
})
require.NoError(t, err)
return manifest.Files()
}
// TestFreshenUnchanged freshens a tree that has not changed since gen
// made its manifest: the manifest must list the same entries as before.
func TestFreshenUnchanged(t *testing.T) {
t.Parallel()
fs := afero.NewOsFs()
root, manifestPath := setupFreshenDir(t, fs,
map[string]string{testFileTxt: "content1", testDirFile: "content2"})
before := manifestFiles(t, fs, manifestPath)
fs := afero.NewMemMapFs()
setupFreshenDir(t, fs)
runFreshen(t, fs, root, manifestPath)
after := manifestFiles(t, fs, manifestPath)
require.Len(t, after, len(before))
for i := range before {
assert.True(t, proto.Equal(before[i], after[i]),
"entry for %s changed", before[i].GetPath())
}
// Parse manifest to verify
manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf")
require.NoError(t, err)
assert.Len(t, manifest.Files(), 2)
}
// assertManifestLists asserts that the manifest at manifestPath lists
// exactly the files in want, each with the size and SHA-256 hash of its
// content in want and the mtime of the file of that name under root.
func assertManifestLists(
t *testing.T, fs afero.Fs, root, manifestPath string, want map[string]string,
) {
t.Helper()
files := manifestFiles(t, fs, manifestPath)
listed := make([]string, 0, len(files))
for _, f := range files {
listed = append(listed, f.GetPath())
content, ok := want[f.GetPath()]
if !ok {
continue // reported by the ElementsMatch below
}
digest := sha256.Sum256([]byte(content))
hash, err := multihash.Encode(digest[:], multihash.SHA2_256)
require.NoError(t, err)
assert.Equal(t, int64(len(content)), f.GetSize(), f.GetPath())
require.NotEmpty(t, f.GetHashes(), f.GetPath())
assert.Equal(t, hash, f.GetHashes()[0].GetMultiHash(), f.GetPath())
info, err := fs.Stat(filepath.Join(root, f.GetPath()))
require.NoError(t, err)
mtime, ok := entryMtime(f)
assert.True(t, ok && mtime.Equal(info.ModTime()),
"%s: manifest has mtime %v, file has %v",
f.GetPath(), mtime, info.ModTime())
}
assert.ElementsMatch(t, slices.Collect(maps.Keys(want)), listed)
}
// TestFreshenWithChanges makes one change to a tree after gen made its
// manifest, then freshens the manifest. The rewritten manifest must list
// exactly the files now in the tree, and check must pass on the tree.
func TestFreshenWithChanges(t *testing.T) {
t.Parallel()
tree := map[string]string{testFileTxt: "content1", testDirFile: "content2"}
fs := afero.NewMemMapFs()
setupFreshenDir(t, fs)
// Every file a case writes gets this mtime, which differs from the one
// gen recorded, so an edit that keeps the size is told apart by its
// mtime whatever the filesystem's clock resolution.
writtenMtime := time.Unix(1_700_000_000, 0)
// Verify initial manifest has 2 files
manifest, err := mfer.NewManifestFromFile(fs, "/testdir/.index.mf")
require.NoError(t, err)
assert.Len(t, manifest.Files(), 2)
for _, tc := range []struct {
name string
write map[string]string // files to write, by path and content
remove string // file to delete, if any
}{
{
name: "modified file",
write: map[string]string{testDirFile: "modified content2"},
},
{
name: "modified file, same size",
write: map[string]string{testDirFile: "CONTENT2"},
},
{
name: "new file",
write: map[string]string{"dir/new.txt": "content3"},
},
{
name: "deleted file",
remove: testFileTxt,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
// Add a new file
writeTestFile(t, fs, "/testdir/file3.txt", "content3")
fs := afero.NewOsFs()
root, manifestPath := setupFreshenDir(t, fs, tree)
// Modify file2 (change content and size)
writeTestFile(t, fs, "/testdir/file2.txt", "modified content2")
// want is the tree as it is after the change.
want := maps.Clone(tree)
// Remove file1
require.NoError(t, fs.Remove(testFile1))
for path, content := range tc.write {
writeTestFile(t, fs, filepath.Join(root, path), content)
require.NoError(t, fs.Chtimes(
filepath.Join(root, path), writtenMtime, writtenMtime))
want[path] = content
}
// Note: The freshen operation would need to be run here
// For now, we just verify the test setup is correct
exists, _ := afero.Exists(fs, testFile1)
assert.False(t, exists)
if tc.remove != "" {
require.NoError(t, fs.Remove(filepath.Join(root, tc.remove)))
delete(want, tc.remove)
}
exists, _ = afero.Exists(fs, "/testdir/file3.txt")
assert.True(t, exists)
runFreshen(t, fs, root, manifestPath)
assertManifestLists(t, fs, root, manifestPath, want)
opts := testOpts([]string{
testApp, cmdCheck, "-q", testFlagNoExtra, testFlagBase, root, manifestPath,
}, fs)
assert.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
})
}
}
// TestFreshenLeavesManifestOutOfListing freshens a manifest kept in a
// subdirectory of the tree it lists: the manifest is not listed, while an
// ordinary file of the same name at the top of the tree is. The manifest
// is recognized by file identity, which needs the real filesystem.
func TestFreshenLeavesManifestOutOfListing(t *testing.T) {
t.Parallel()
root := t.TempDir()
manifestPath := filepath.Join(root, "sub", "listing.mf")
fs := afero.NewOsFs()
require.NoError(t, fs.MkdirAll(filepath.Join(root, "sub"), 0o750))
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "hello")
writeTestFile(t, fs, filepath.Join(root, "listing.mf"), "an ordinary file")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", manifestPath, root}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
// A new file gives freshen something to write.
writeTestFile(t, fs, filepath.Join(root, "added.txt"), "added")
opts = testOpts([]string{
testApp, "freshen", "-q", testFlagBase, root, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, []string{testFileTxt, "listing.mf", "added.txt"},
manifestPaths(t, fs, manifestPath))
}
// TestFreshenLeavesLeftoverTempFileOutOfListing freshens a manifest where
// an interrupted run left its temp file beside it: the leftover is not
// listed.
func TestFreshenLeavesLeftoverTempFileOutOfListing(t *testing.T) {
t.Parallel()
root := t.TempDir()
manifestPath := filepath.Join(root, "index.mf")
fs := afero.NewOsFs()
writeTestFile(t, fs, filepath.Join(root, testFileTxt), "hello")
opts := testOpts([]string{testApp, cmdGenerate, "-q", "-o", manifestPath, root}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
writeTestFile(t, fs, manifestTempPath(manifestPath), "part of a manifest")
// A new file gives freshen something to write.
writeTestFile(t, fs, filepath.Join(root, "added.txt"), "added")
opts = testOpts([]string{
testApp, "freshen", "-q", testFlagBase, root, manifestPath,
}, fs)
require.Equal(t, 0, runCLI(opts), "stderr: %s", testStderr(t, opts))
assert.ElementsMatch(t, []string{testFileTxt, "added.txt"},
manifestPaths(t, fs, manifestPath))
content, _ := afero.ReadFile(fs, "/testdir/file2.txt")
assert.Equal(t, "modified content2", string(content))
}
// TestFreshenRecordEntryMtimePresence pins the behavior of recordEntry
@@ -338,50 +169,21 @@ func TestFreshenRecordEntryMtimePresence(t *testing.T) {
func TestFreshenAddExistingRejectsMissingMtime(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
b := mfer.NewBuilder()
entry := &mfer.MFFilePath{
Path: "file1.txt",
Size: 8,
Mtime: nil,
Hashes: []*mfer.MFFileChecksum{
{MultiHash: hash},
{MultiHash: []byte{0x12, 0x20}},
},
}
err = addExistingToBuilder(b, entry)
err := addExistingToBuilder(b, entry)
require.ErrorIs(t, err, errEntryMissingMtime)
assert.Contains(t, err.Error(), "file1.txt")
}
// TestFreshenAddExistingRejectsShortHash pins that an existing manifest
// entry whose hash the builder refuses is reported as a problem with the
// manifest, with the command that fixes it.
func TestFreshenAddExistingRejectsShortHash(t *testing.T) {
t.Parallel()
sha1Hash, err := multihash.Encode(make([]byte, 20), multihash.SHA1)
require.NoError(t, err)
b := mfer.NewBuilder()
entry := &mfer.MFFilePath{
Path: "old.txt",
Size: 8,
Mtime: &mfer.Timestamp{Seconds: 1_700_000_000},
Hashes: []*mfer.MFFileChecksum{
{MultiHash: sha1Hash},
},
}
err = addExistingToBuilder(b, entry)
require.Error(t, err)
assert.Contains(t, err.Error(), "manifest entry old.txt")
assert.Contains(t, err.Error(), "mfer generate")
assert.Zero(t, b.FileCount())
}
// TestEntryMtime pins the presence semantics the callers depend on.
func TestEntryMtime(t *testing.T) {
t.Parallel()
internal/cli/gen.go
+1 -5
View File
@@ -89,15 +89,11 @@ func (mfa *CLIApp) collectInputPaths(args cli.Args) ([]string, error) {
// buildScannerOptions constructs scanner options from the CLI flags.
func (mfa *CLIApp) buildScannerOptions(ctx *cli.Context) *mfer.ScannerOptions {
output := ctx.String("output")
opts := &mfer.ScannerOptions{
IncludeDotfiles: ctx.Bool("include-dotfiles"),
FollowSymLinks: ctx.Bool("follow-symlinks"),
IncludeTimestamps: ctx.Bool("include-timestamps"),
Fs: mfa.Fs,
// Neither a manifest being replaced nor a temp file left by an
// interrupted run belongs in the new manifest.
ExcludePaths: []string{output, manifestTempPath(output)},
}
// Set seed for deterministic UUID if provided
@@ -221,7 +217,7 @@ func (mfa *CLIApp) generateManifestOperation(ctx *cli.Context) error {
}
// Create temp file for atomic write
tmpPath := manifestTempPath(outputPath)
tmpPath := outputPath + ".tmp"
outFile, err := mfa.Fs.Create(tmpPath)
if err != nil {
internal/cli/manifest_loader.go
+1 -1
View File
@@ -65,7 +65,7 @@ func (mfa *CLIApp) openManifestReader(pathOrURL string) (io.ReadCloser, error) {
}
// resolveManifestArg resolves the manifest path from CLI arguments.
// HTTP(S) URLs are returned as-is. Directories are searched for index.mf.
// HTTP(S) URLs are returned as-is. Directories are searched for index.mf/.index.mf.
// If no argument is given, the current directory is searched.
func (mfa *CLIApp) resolveManifestArg(ctx *cli.Context) (string, error) {
if ctx.Args().Len() > 0 {
internal/cli/mfer.go
+17 -54
View File
@@ -17,30 +17,15 @@ import (
const (
cmdGenerate = "generate"
cmdCheck = "check"
cmdFreshen = "freshen"
cmdExport = "export"
cmdFetch = "fetch"
cmdVersion = "version"
flagProgress = "progress"
flagTimeout = "timeout"
flagDest = "dest"
flagRequireSignature = "require-signature"
flagProgress = "progress"
manifestArgsUsage = "[manifest file]"
// defaultManifestName is the filename gen writes by default, the one
// looked for when a command is given a directory, and the one fetch
// appends to a directory URL.
defaultManifestName = "index.mf"
)
// manifestTempPath returns the temp file gen and freshen write a manifest
// to before renaming it to out.
func manifestTempPath(out string) string {
return out + ".tmp"
}
// errUnknownCommand indicates an unrecognized command argument.
var errUnknownCommand = errors.New("unknown command")
@@ -133,7 +118,7 @@ func commonFlags() []cli.Flag {
&cli.BoolFlag{
Name: "verbose",
Aliases: []string{"v"},
Usage: "Increase verbosity (-v for verbose, -v -v for debug)",
Usage: "Increase verbosity (-v for verbose, -vv for debug)",
Count: new(int),
},
&cli.BoolFlag{
@@ -144,23 +129,11 @@ func commonFlags() []cli.Flag {
}
}
// requireSignatureFlag returns the --require-signature flag taken by the
// check and fetch subcommands.
func requireSignatureFlag() *cli.StringFlag {
return &cli.StringFlag{
Name: flagRequireSignature,
Aliases: []string{"S"},
Usage: "Require manifest to be signed by the specified GPG key ID",
EnvVars: []string{"MFER_REQUIRE_SIGNATURE"},
}
}
func (mfa *CLIApp) generateCommand() *cli.Command {
return &cli.Command{
Name: cmdGenerate,
Aliases: []string{"gen"},
Usage: "Generate manifest file",
ArgsUsage: "[path ...]",
Name: cmdGenerate,
Aliases: []string{"gen"},
Usage: "Generate manifest file",
Action: func(c *cli.Context) error {
mfa.setVerbosity(c)
mfa.printBanner()
@@ -181,7 +154,7 @@ func (mfa *CLIApp) generateCommand() *cli.Command {
},
&cli.StringFlag{
Name: "output",
Value: defaultManifestName,
Value: "./.index.mf",
Aliases: []string{"o"},
Usage: "Specify output filename",
},
@@ -240,16 +213,21 @@ func (mfa *CLIApp) checkCommand() *cli.Command {
},
&cli.BoolFlag{
Name: "no-extra-files",
Usage: "Fail, instead of warning, if files in base directory are not in manifest",
Usage: "Fail if files exist in base directory that are not in manifest",
},
&cli.StringFlag{
Name: "require-signature",
Aliases: []string{"S"},
Usage: "Require manifest to be signed by the specified GPG key ID",
EnvVars: []string{"MFER_REQUIRE_SIGNATURE"},
},
requireSignatureFlag(),
),
}
}
func (mfa *CLIApp) freshenCommand() *cli.Command {
return &cli.Command{
Name: cmdFreshen,
Name: "freshen",
Usage: "Update manifest with changed, new, and removed files",
ArgsUsage: manifestArgsUsage,
Action: func(c *cli.Context) error {
@@ -346,30 +324,15 @@ func (mfa *CLIApp) listCommand() *cli.Command {
func (mfa *CLIApp) fetchCommand() *cli.Command {
return &cli.Command{
Name: cmdFetch,
Usage: "fetch manifest and referenced files",
ArgsUsage: "URL",
Name: cmdFetch,
Usage: "fetch manifest and referenced files",
Action: func(c *cli.Context) error {
mfa.setVerbosity(c)
mfa.printBanner()
return mfa.fetchManifestOperation(c)
},
Flags: append(commonFlags(),
&cli.DurationFlag{
Name: flagTimeout,
Value: httpTimeout,
Usage: "Time limit for each HTTP request, including the download " +
"of its body",
},
&cli.StringFlag{
Name: flagDest,
Aliases: []string{"d"},
Value: ".",
Usage: "Directory to download the files and the manifest into",
},
requireSignatureFlag(),
),
Flags: commonFlags(),
}
}
internal/log/log.go
+55 -149
View File
@@ -1,25 +1,19 @@
// Package log provides leveled logging on top of log/slog, and helpers that
// print progress lines which overwrite each other in place on a terminal.
//
// Until Init runs, log records go to slog.Default(), so a program that uses
// package mfer as a library gets them wherever it sends its own slog output.
// Init switches to the CLI's format on the stderr writer given to SetOutput.
// Progress lines are not log records: they are written straight to the stdout
// writer, never through slog.
// Package log provides leveled logging with progress output helpers
// on top of apex/log and pterm.
package log
import (
"context"
"fmt"
"io"
"log/slog"
"os"
"path/filepath"
"runtime"
"sync"
"github.com/apex/log"
acli "github.com/apex/log/handlers/cli"
"github.com/davecgh/go-spew/spew"
"golang.org/x/term"
"github.com/pterm/pterm"
)
// Level represents log severity levels.
@@ -64,98 +58,28 @@ func (l Level) String() string {
// helpers to the caller of the log package.
const callerSkip = 2
// Escape sequences for colored log lines.
const (
ansiReset = "\x1b[0m"
ansiBold = "\x1b[1m"
ansiRed = "\x1b[31m"
ansiYellow = "\x1b[33m"
ansiBlue = "\x1b[34m"
ansiWhite = "\x1b[37m"
)
//nolint:gochecknoglobals // package-level logger state by design
var (
// mu protects the variables below
// mu protects the output writers and level
mu sync.RWMutex
// stdout is the writer for progress output
stdout io.Writer = os.Stdout
// stderr is the writer for log output
stderr io.Writer = os.Stderr
// styled is false once DisableStyling has been called
styled = true
// logger is the CLI logger Init builds; nil until Init runs
logger *slog.Logger
// currentLevel is our log level (includes Verbose)
currentLevel = InfoLevel
)
// cliHandler is the slog.Handler the CLI logs through. Each record is one
// line: a symbol for its level right-aligned in four columns, then the
// message padded to 25 columns. In color, only the symbol is bold and in the
// level's color; the message is in the terminal's default color. Records are
// filtered by level before they are made, so the handler takes every record
// it is given.
type cliHandler struct {
mu sync.Mutex
w io.Writer
color bool
}
// Enabled reports true for every level.
func (h *cliHandler) Enabled(context.Context, slog.Level) bool {
return true
}
// Handle writes the record as one line.
func (h *cliHandler) Handle(_ context.Context, r slog.Record) error {
symbol, color := "•", ansiBlue
switch {
case r.Level >= slog.LevelError:
symbol, color = "⨯", ansiRed
case r.Level >= slog.LevelWarn:
color = ansiYellow
case r.Level < slog.LevelInfo:
color = ansiWhite
}
h.mu.Lock()
defer h.mu.Unlock()
if !h.color {
_, err := fmt.Fprintf(h.w, "%4s %-25s\n", symbol, r.Message)
return err
}
_, err := fmt.Fprintf(h.w, "%s%s%4s%s %-25s%s\n",
color, ansiBold, symbol, ansiReset, r.Message, ansiReset)
return err
}
// WithAttrs returns the handler unchanged: this package's helpers never
// attach attributes.
func (h *cliHandler) WithAttrs([]slog.Attr) slog.Handler {
return h
}
// WithGroup returns the handler unchanged: this package's helpers never open
// groups.
func (h *cliHandler) WithGroup(string) slog.Handler {
return h
}
// SetOutput configures the output writers for the log package.
// stdout is used for progress output, stderr is used for log messages
// from the next Init on.
// stdout is used for progress output, stderr is used for log messages.
func SetOutput(out, err io.Writer) {
mu.Lock()
defer mu.Unlock()
stdout = out
stderr = err
pterm.SetDefaultOutput(out)
}
// GetStdout returns the configured stdout writer.
@@ -174,29 +98,31 @@ func GetStderr() io.Writer {
return stderr
}
// DisableStyling turns off colors in log lines from the next Init on.
// DisableStyling turns off colors and styling for terminal output.
func DisableStyling() {
mu.Lock()
defer mu.Unlock()
pterm.DisableColor()
pterm.DisableStyling()
styled = false
pterm.Debug.Prefix.Text = ""
pterm.Info.Prefix.Text = ""
pterm.Success.Prefix.Text = ""
pterm.Warning.Prefix.Text = ""
pterm.Error.Prefix.Text = ""
pterm.Fatal.Prefix.Text = ""
}
// Init sends log records to the CLI handler on the stderr writer given to
// SetOutput. Log lines are colored when stdout is a terminal whose TERM is
// not dumb, unless DisableStyling has been called.
// Init initializes the logger with the CLI handler and default log level.
//
// It replaces the logger under the write lock, and log calls hold the read
// lock while they write, so once Init returns nothing writes through the
// logger it replaced.
// It reconfigures the process-global apex/log logger under the write lock so
// the global is never mutated while another goroutine holds the read lock to
// read it in emit. Without this, parallel callers (e.g. the test suite) race
// Init's SetLevel/SetHandler against concurrent log calls.
func Init() {
mu.Lock()
defer mu.Unlock()
color := styled && os.Getenv("TERM") != "dumb" &&
term.IsTerminal(int(os.Stdout.Fd()))
logger = slog.New(&cliHandler{w: stderr, color: color})
log.SetHandler(acli.New(stderr))
log.SetLevel(log.DebugLevel) // Let apex/log pass everything; we filter ourselves
}
// isEnabled returns true if messages at the given level should be logged.
@@ -207,87 +133,66 @@ func isEnabled(l Level) bool {
return l >= currentLevel
}
// logf logs a formatted message at level l if messages at l are enabled,
// holding the read lock while the record is written. slog has no verbose or
// fatal level: verbose messages are logged as info records, and fatal
// messages as error records.
func logf(l Level, format string, args ...any) {
// emit calls fn while holding the read lock if messages at level l are
// enabled. Holding the read lock across the apex/log call keeps the global
// logger from being read while Init reconfigures it under the write lock.
func emit(l Level, fn func()) {
mu.RLock()
defer mu.RUnlock()
if l < currentLevel {
return
}
lg := logger
if lg == nil {
lg = slog.Default()
}
msg := fmt.Sprintf(format, args...)
switch l {
case DebugLevel:
lg.Debug(msg)
case VerboseLevel, InfoLevel:
lg.Info(msg)
case WarnLevel:
lg.Warn(msg)
case ErrorLevel, FatalLevel:
lg.Error(msg)
if l >= currentLevel {
fn()
}
}
// Fatalf logs a formatted message at fatal level, then exits with status 1.
// Fatalf logs a formatted message at fatal level.
func Fatalf(format string, args ...any) {
logf(FatalLevel, format, args...)
os.Exit(1)
emit(FatalLevel, func() { log.Fatalf(format, args...) })
}
// Fatal logs a message at fatal level, then exits with status 1.
// Fatal logs a message at fatal level.
func Fatal(arg string) {
logf(FatalLevel, "%s", arg)
os.Exit(1)
emit(FatalLevel, func() { log.Fatal(arg) })
}
// Errorf logs a formatted message at error level.
func Errorf(format string, args ...any) {
logf(ErrorLevel, format, args...)
emit(ErrorLevel, func() { log.Errorf(format, args...) })
}
// Error logs a message at error level.
func Error(arg string) {
logf(ErrorLevel, "%s", arg)
emit(ErrorLevel, func() { log.Error(arg) })
}
// Warnf logs a formatted message at warn level.
func Warnf(format string, args ...any) {
logf(WarnLevel, format, args...)
emit(WarnLevel, func() { log.Warnf(format, args...) })
}
// Warn logs a message at warn level.
func Warn(arg string) {
logf(WarnLevel, "%s", arg)
emit(WarnLevel, func() { log.Warn(arg) })
}
// Infof logs a formatted message at info level.
func Infof(format string, args ...any) {
logf(InfoLevel, format, args...)
emit(InfoLevel, func() { log.Infof(format, args...) })
}
// Info logs a message at info level.
func Info(arg string) {
logf(InfoLevel, "%s", arg)
emit(InfoLevel, func() { log.Info(arg) })
}
// Verbosef logs a formatted message at verbose level.
func Verbosef(format string, args ...any) {
logf(VerboseLevel, format, args...)
emit(VerboseLevel, func() { log.Infof(format, args...) })
}
// Verbose logs a message at verbose level.
func Verbose(arg string) {
logf(VerboseLevel, "%s", arg)
emit(VerboseLevel, func() { log.Info(arg) })
}
// Debugf logs a formatted message at debug level with caller location.
@@ -306,12 +211,20 @@ func Debug(arg string) {
// DebugReal logs at debug level with caller info from the specified stack depth.
func DebugReal(arg string, cs int) {
mu.RLock()
defer mu.RUnlock()
if DebugLevel < currentLevel {
return
}
_, callerFile, callerLine, ok := runtime.Caller(cs)
if !ok {
return
}
logf(DebugLevel, "%s:%d: %s", filepath.Base(callerFile), callerLine, arg)
tag := fmt.Sprintf("%s:%d: ", filepath.Base(callerFile), callerLine)
log.Debug(tag + arg)
}
// Dump logs a spew dump of the arguments at debug level.
@@ -362,19 +275,12 @@ func GetLevel() Level {
// Progressf prints a progress message that overwrites the current line.
// Use ProgressDone() when progress is complete to move to the next line.
// Progress goes to the stdout writer whatever the log level.
func Progressf(format string, args ...any) {
mu.Lock()
defer mu.Unlock()
_, _ = fmt.Fprintf(stdout, "\r"+format, args...)
pterm.Printf("\r"+format, args...)
}
// ProgressDone clears the progress line when progress is complete.
func ProgressDone() {
mu.Lock()
defer mu.Unlock()
// Return to the start of the line and erase it
_, _ = fmt.Fprint(stdout, "\r\033[K")
// Clear the line with spaces and return to beginning
pterm.Print("\r\033[K")
}
internal/log/log_test.go
+3 -220
View File
@@ -1,229 +1,12 @@
//nolint:testpackage // white-box tests exercise unexported internals
package log
package log_test
import (
"bytes"
stdlog "log"
"log/slog"
"os"
"testing"
"github.com/stretchr/testify/assert"
"sneak.berlin/go/mfer/internal/log"
)
func TestBuild(t *testing.T) {
t.Parallel()
Init()
}
// capture points the package's writers at fresh buffers and sets its level,
// with colors off, then restores the default writers and level when the test
// ends. The state it changes is process-wide, so tests that use it do not
// run in parallel.
func capture(t *testing.T, l Level) (*bytes.Buffer, *bytes.Buffer) {
t.Helper()
var stdout, stderr bytes.Buffer
DisableStyling()
SetOutput(&stdout, &stderr)
SetLevel(l)
Init()
t.Cleanup(func() {
SetOutput(os.Stdout, os.Stderr)
SetLevel(InfoLevel)
Init()
})
return &stdout, &stderr
}
// TestLevelFiltering logs at every level, through both the plain and the
// formatting helpers, and checks that exactly the messages at or above the
// set level are written.
//
//nolint:paralleltest // changes the package's process-wide writers and level
func TestLevelFiltering(t *testing.T) {
messages := []struct {
level Level
text string
}{
{DebugLevel, "debug plain"},
{DebugLevel, "debug formatted"},
{VerboseLevel, "verbose plain"},
{VerboseLevel, "verbose formatted"},
{InfoLevel, "info plain"},
{InfoLevel, "info formatted"},
{WarnLevel, "warn plain"},
{WarnLevel, "warn formatted"},
{ErrorLevel, "error plain"},
{ErrorLevel, "error formatted"},
}
levels := []Level{DebugLevel, VerboseLevel, InfoLevel, WarnLevel, ErrorLevel}
for _, set := range levels {
t.Run(set.String(), func(t *testing.T) {
stdout, stderr := capture(t, set)
Debug("debug plain")
Debugf("debug %s", "formatted")
Verbose("verbose plain")
Verbosef("verbose %s", "formatted")
Info("info plain")
Infof("info %s", "formatted")
Warn("warn plain")
Warnf("warn %s", "formatted")
Error("error plain")
Errorf("error %s", "formatted")
for _, m := range messages {
if m.level >= set {
assert.Contains(t, stderr.String(), m.text)
} else {
assert.NotContains(t, stderr.String(), m.text)
}
}
assert.Empty(t, stdout.String())
})
}
}
// TestLineFormat checks the exact uncolored lines: a symbol per level
// right-aligned in four columns, then the message padded to 25 columns.
//
//nolint:paralleltest // changes the package's process-wide writers and level
func TestLineFormat(t *testing.T) {
_, stderr := capture(t, VerboseLevel)
Infof("scanning filesystem...")
Verbosef("+ %s (%s)", "a.txt", "1 B")
Warn("short")
Errorf("%s", "a message longer than twenty-five columns")
want := " • scanning filesystem... \n" +
" • + a.txt (1 B) \n" +
" • short \n" +
" ⨯ a message longer than twenty-five columns\n"
assert.Equal(t, want, stderr.String())
}
// TestDebugCallerTag checks that debug lines start with the file and line
// of the call that logged them.
//
//nolint:paralleltest // changes the package's process-wide writers and level
func TestDebugCallerTag(t *testing.T) {
_, stderr := capture(t, DebugLevel)
Debugf("enumerating path: %s", "/tmp")
assert.Regexp(t, `^ • log_test\.go:\d+: enumerating path: /tmp\s*\n$`,
stderr.String())
}
// TestColoredLine checks the colored lines: only the symbol is bold and in the
// level's color; the message is in the terminal's default color.
func TestColoredLine(t *testing.T) {
t.Parallel()
var buf bytes.Buffer
lg := slog.New(&cliHandler{w: &buf, color: true})
lg.Debug("debug")
lg.Info("info")
lg.Warn("warn")
lg.Error("error")
want := "\x1b[37m\x1b[1m •\x1b[0m debug \x1b[0m\n" +
"\x1b[34m\x1b[1m •\x1b[0m info \x1b[0m\n" +
"\x1b[33m\x1b[1m •\x1b[0m warn \x1b[0m\n" +
"\x1b[31m\x1b[1m ⨯\x1b[0m error \x1b[0m\n"
assert.Equal(t, want, buf.String())
}
// TestRecordLevels checks the slog level of the record each helper logs,
// through slog's text handler with the time left out. slog has no verbose
// level, so verbose messages are info records.
//
//nolint:paralleltest // changes the package's process-wide logger and level
func TestRecordLevels(t *testing.T) {
var buf bytes.Buffer
capture(t, DebugLevel)
opts := &slog.HandlerOptions{
Level: slog.LevelDebug,
ReplaceAttr: func(_ []string, a slog.Attr) slog.Attr {
if a.Key == slog.TimeKey {
return slog.Attr{}
}
return a
},
}
mu.Lock()
logger = slog.New(slog.NewTextHandler(&buf, opts))
mu.Unlock()
Debugf("debug")
Verbosef("verbose")
Infof("info")
Warnf("warn")
Errorf("error")
assert.Regexp(t, `^level=DEBUG msg="log_test\.go:\d+: debug"\n`+
"level=INFO msg=verbose\n"+
"level=INFO msg=info\n"+
"level=WARN msg=warn\n"+
"level=ERROR msg=error\n$", buf.String())
}
// TestProgress checks that progress lines go to the stdout writer whatever
// the log level, each starting with a carriage return so it overwrites the
// last, and that ProgressDone erases the line.
//
//nolint:paralleltest // changes the package's process-wide writers and level
func TestProgress(t *testing.T) {
stdout, stderr := capture(t, ErrorLevel)
Progressf("Scanning: %d files found", 1000)
Progressf("Scanning: %d files found", 2000)
ProgressDone()
assert.Equal(t,
"\rScanning: 1000 files found\rScanning: 2000 files found\r\x1b[K",
stdout.String())
assert.Empty(t, stderr.String())
}
// TestBeforeInit checks that records go to slog.Default() until Init runs,
// still filtered by the package's level. slog's default handler writes
// through the standard library's log package.
//
//nolint:paralleltest // changes the package's process-wide logger
func TestBeforeInit(t *testing.T) {
var buf bytes.Buffer
flags := stdlog.Flags()
stdlog.SetOutput(&buf)
stdlog.SetFlags(0)
mu.Lock()
logger = nil
mu.Unlock()
t.Cleanup(func() {
stdlog.SetOutput(os.Stderr)
stdlog.SetFlags(flags)
Init()
})
Infof("loaded manifest with %d files", 3)
Verbose("not shown at info level")
assert.Equal(t, "INFO loaded manifest with 3 files\n", buf.String())
log.Init()
}
mfer/builder.go
+4 -17
View File
@@ -35,8 +35,7 @@ var (
errPathDotDot = errors.New("contains '..' segment")
errSizeMismatch = errors.New("size mismatch")
errNegativeSize = errors.New("size cannot be negative")
errHashNotMultihash = errors.New("hash is not a valid multihash")
errHashTooShort = errors.New("hash digest is too short")
errEmptyHash = errors.New("hash cannot be nil or empty")
)
// ValidatePath checks that a file path conforms to manifest path invariants:
@@ -229,8 +228,7 @@ func (b *Builder) FileCount() int {
// AddFileWithHash adds a file entry with a pre-computed hash.
// This is useful when the hash is already known (e.g., from an existing manifest).
// Returns an error if path is invalid, size is negative, or hash is not a
// multihash with a digest of at least 32 bytes, as long as SHA-256's.
// Returns an error if path is empty, size is negative, or hash is nil/empty.
func (b *Builder) AddFileWithHash(
path RelFilePath,
size FileSize,
@@ -246,19 +244,8 @@ func (b *Builder) AddFileWithHash(
return errNegativeSize
}
decoded, err := multihash.Decode(hash)
if err != nil {
return fmt.Errorf("%w: %w", errHashNotMultihash, err)
}
// The reader's limit on decoding cost (maxDecodedGrowth) assumes every
// hash is at least as long as a SHA-256 multihash, so a manifest of
// shorter ones could fail to load.
if len(decoded.Digest) < sha256.Size {
return fmt.Errorf(
"%w: %d bytes, at least %d needed",
errHashTooShort, len(decoded.Digest), sha256.Size,
)
if len(hash) == 0 {
return errEmptyHash
}
entry := &MFFilePath{
mfer/builder_test.go
+27 -75
View File
@@ -4,15 +4,11 @@ package mfer
import (
"bytes"
"context"
"crypto/sha256"
"fmt"
"path/filepath"
"strings"
"testing"
"time"
"github.com/multiformats/go-multihash"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -46,10 +42,9 @@ func TestBuilderAddFileWithHash(t *testing.T) {
t.Parallel()
b := NewBuilder()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
hash := make([]byte, 34) // SHA256 multihash is 34 bytes
err = b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), hash)
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), hash)
require.NoError(t, err)
assert.Equal(t, 1, b.FileCount())
}
@@ -57,14 +52,12 @@ func TestBuilderAddFileWithHash(t *testing.T) {
func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel()
sha256Hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
t.Run("empty path", func(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("", 100, ModTime(time.Now()), sha256Hash)
hash := make([]byte, 34)
err := b.AddFileWithHash("", 100, ModTime(time.Now()), hash)
require.Error(t, err)
assert.Contains(t, err.Error(), "path")
})
@@ -73,58 +66,41 @@ func TestBuilderAddFileWithHashValidation(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), sha256Hash)
hash := make([]byte, 34)
err := b.AddFileWithHash("test.txt", -1, ModTime(time.Now()), hash)
require.Error(t, err)
assert.Contains(t, err.Error(), "size")
})
t.Run("nil hash", func(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "hash")
})
t.Run("empty hash", func(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), []byte{})
require.Error(t, err)
assert.Contains(t, err.Error(), "hash")
})
t.Run("valid inputs", func(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), sha256Hash)
hash := make([]byte, 34)
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), hash)
require.NoError(t, err)
assert.Equal(t, 1, b.FileCount())
})
}
func TestBuilderAddFileWithHashRejectsBadHashes(t *testing.T) {
t.Parallel()
sha1Hash, err := multihash.Encode(make([]byte, 20), multihash.SHA1)
require.NoError(t, err)
truncatedHash, err := multihash.Encode(make([]byte, sha256.Size-1), multihash.SHA2_256)
require.NoError(t, err)
tests := []struct {
name string
hash Multihash
want error
}{
{"nil hash", nil, errHashNotMultihash},
{"empty hash", []byte{}, errHashNotMultihash},
{"one-byte hash", []byte{0x12}, errHashNotMultihash},
// A SHA-256 code and 32-byte length, then only two bytes of digest.
{"malformed multihash", []byte{0x12, 0x20, 0x01, 0x02}, errHashNotMultihash},
// A valid multihash, but its 20-byte SHA-1 digest is too short.
{"SHA-1 multihash", sha1Hash, errHashTooShort},
// A valid multihash whose 31-byte digest is one byte short.
{"31-byte digest", truncatedHash, errHashTooShort},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
b := NewBuilder()
err := b.AddFileWithHash("test.txt", 100, ModTime(time.Now()), tt.hash)
require.ErrorIs(t, err, tt.want)
assert.Equal(t, 0, b.FileCount())
})
}
}
func TestBuilderBuild(t *testing.T) {
t.Parallel()
@@ -380,8 +356,7 @@ func TestBuilderBuildRoundTrip(t *testing.T) {
func TestBuilderBuildRoundTripLargeManifest(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
b := NewBuilder()
@@ -422,29 +397,6 @@ func TestNewManifestFromReaderTruncated(t *testing.T) {
assert.Error(t, err)
}
func TestNewManifestFromFileRequiresPath(t *testing.T) {
t.Parallel()
_, err := NewManifestFromFile(nil)
require.ErrorIs(t, err, errManifestPathEmpty)
_, err = NewManifestFromFile(&ManifestFromFileOptions{Fs: afero.NewMemMapFs()})
require.ErrorIs(t, err, errManifestPathEmpty)
}
func TestNewManifestFromFileNilFsUsesOsFs(t *testing.T) {
t.Parallel()
path := filepath.Join(t.TempDir(), "index.mf")
createTestManifest(t, afero.NewOsFs(), path, map[string][]byte{
testFileName: []byte("hello"),
})
m, err := NewManifestFromFile(&ManifestFromFileOptions{Path: path})
require.NoError(t, err)
assert.Len(t, m.Files(), 1)
}
func TestManifestString(t *testing.T) {
t.Parallel()
mfer/checker.go
+42 -83
View File
@@ -14,11 +14,7 @@ import (
"github.com/spf13/afero"
)
var (
errNoSigningPubKey = errors.New("manifest has no signing public key")
errManifestPathEmpty = errors.New("manifest path cannot be empty")
errBasePathEmpty = errors.New("base path cannot be empty")
)
var errNoSigningPubKey = errors.New("manifest has no signing public key")
// Result represents the outcome of checking a single file.
type Result struct {
@@ -77,51 +73,29 @@ type Checker struct {
fs afero.Fs
// manifestPaths is a set of paths in the manifest for quick lookup
manifestPaths map[RelFilePath]struct{}
// manifestInfo is the manifest file, which FindExtraFiles leaves out,
// matched with os.SameFile.
manifestInfo os.FileInfo
// manifestRelPath is the relative path of the manifest file from
// basePath (for exclusion)
manifestRelPath RelFilePath
// signature info from the manifest
signature []byte
signer []byte
signingPubKey []byte
}
// CheckerOptions configures a Checker.
type CheckerOptions struct {
// ManifestPath is the manifest file to check against (required).
ManifestPath string
// BasePath is the directory relative to which manifest paths are
// resolved (required).
BasePath string
// Fs is the filesystem to use, defaults to OsFs if nil.
Fs afero.Fs
}
// NewChecker creates a new Checker with the given options. It returns an
// error if opts is nil or either path is empty.
func NewChecker(opts *CheckerOptions) (*Checker, error) {
if opts == nil || opts.ManifestPath == "" {
return nil, errManifestPathEmpty
}
if opts.BasePath == "" {
return nil, errBasePathEmpty
}
fs := opts.Fs
// NewChecker creates a new Checker for the given manifest, base path, and filesystem.
// The basePath is the directory relative to which manifest paths are resolved.
// If fs is nil, the real filesystem (OsFs) is used.
func NewChecker(manifestPath string, basePath string, fs afero.Fs) (*Checker, error) {
if fs == nil {
fs = afero.NewOsFs()
}
m, err := NewManifestFromFile(&ManifestFromFileOptions{
Path: opts.ManifestPath,
Fs: fs,
})
m, err := NewManifestFromFile(fs, manifestPath)
if err != nil {
return nil, err
}
abs, err := filepath.Abs(opts.BasePath)
abs, err := filepath.Abs(basePath)
if err != nil {
return nil, err
}
@@ -133,20 +107,26 @@ func NewChecker(opts *CheckerOptions) (*Checker, error) {
manifestPaths[RelFilePath(f.GetPath())] = struct{}{}
}
manifestInfo, err := fs.Stat(opts.ManifestPath)
// Compute manifest's relative path from basePath for exclusion in FindExtraFiles
absManifest, err := filepath.Abs(manifestPath)
if err != nil {
return nil, err
}
manifestRel, err := filepath.Rel(abs, absManifest)
if err != nil {
manifestRel = ""
}
return &Checker{
basePath: AbsFilePath(abs),
files: files,
fs: fs,
manifestPaths: manifestPaths,
manifestInfo: manifestInfo,
signature: m.pbOuter.GetSignature(),
signer: m.pbOuter.GetSigner(),
signingPubKey: m.pbOuter.GetSigningPubKey(),
basePath: AbsFilePath(abs),
files: files,
fs: fs,
manifestPaths: manifestPaths,
manifestRelPath: RelFilePath(manifestRel),
signature: m.pbOuter.GetSignature(),
signer: m.pbOuter.GetSigner(),
signingPubKey: m.pbOuter.GetSigningPubKey(),
}, nil
}
@@ -267,27 +247,20 @@ func (c *Checker) Check(
return nil
}
// FindExtraFiles walks the filesystem and reports files not in the manifest,
// hidden files and directories included. The manifest file itself is not
// reported. Anything the search cannot read, such as a directory that cannot
// be listed, is reported with StatusError and the search goes on. Results are
// sent to the results channel. The channel is closed when done.
// FindExtraFiles walks the filesystem and reports files not in the manifest.
// Results are sent to the results channel. The channel is closed when done.
// Hidden files/directories (starting with .) are skipped, as they are excluded
// from manifests by default. The manifest file itself is also skipped.
func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) error {
if results != nil {
defer close(results)
}
// The search does not follow symlinks, so a base directory named
// through one is resolved first. If that fails, the base is searched as
// named and the search reports the problem.
root := string(c.basePath)
walkFn := func(walkPath string, info os.FileInfo, err error) error {
if err != nil {
return err
}
resolved, err := filepath.EvalSymlinks(root)
if err == nil {
root = resolved
}
walkFn := func(walkPath string, info os.FileInfo, walkErr error) error {
select {
case <-ctx.Done():
return ctx.Err()
@@ -295,22 +268,15 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
}
// Get relative path
rel, err := filepath.Rel(root, walkPath)
rel, err := filepath.Rel(string(c.basePath), walkPath)
if err != nil {
return err
}
relPath := RelFilePath(rel)
// Report what cannot be read, such as a directory that cannot be
// listed, and go on with the rest.
if walkErr != nil {
if results != nil {
results <- Result{
Path: relPath,
Status: StatusError,
Message: walkErr.Error(),
}
// Skip hidden files and directories (dotfiles)
if IsHiddenPath(filepath.ToSlash(rel)) {
if info.IsDir() {
return filepath.SkipDir
}
return nil
@@ -321,17 +287,10 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
return nil
}
// A symlink is compared by what it points to, so a manifest reached
// through one is not reported either.
if info.Mode()&os.ModeSymlink != 0 {
target, statErr := c.fs.Stat(walkPath)
if statErr == nil {
info = target
}
}
relPath := RelFilePath(rel)
// Skip the manifest file itself, however its path is spelled
if os.SameFile(info, c.manifestInfo) {
// Skip the manifest file itself
if relPath == c.manifestRelPath {
return nil
}
@@ -349,7 +308,7 @@ func (c *Checker) FindExtraFiles(ctx context.Context, results chan<- Result) err
return nil
}
return afero.Walk(c.fs, root, walkFn)
return afero.Walk(c.fs, string(c.basePath), walkFn)
}
func (c *Checker) checkFile(entry *MFFilePath, checkedBytes *FileSize) Result {
mfer/checker_test.go
+128 -271
View File
@@ -5,8 +5,6 @@ import (
"bytes"
"context"
"fmt"
"os"
"path/filepath"
"testing"
"time"
@@ -16,11 +14,9 @@ import (
)
const (
testFile1 = "file1.txt"
testFile2 = "file2.txt"
testExistsFile = "exists.txt"
testManifestPath = "/manifest.mf"
testDataDir = "/data"
testFile1 = "file1.txt"
testFile2 = "file2.txt"
testExistsFile = "exists.txt"
)
func TestStatusString(t *testing.T) {
@@ -70,13 +66,15 @@ func createTestManifest(
}
// createFilesOnDisk creates the given files on the filesystem under
// testDataDir.
// /data.
func createFilesOnDisk(t *testing.T, fs afero.Fs, files map[string][]byte) {
t.Helper()
basePath := "/data"
for path, content := range files {
fullPath := testDataDir + "/" + path
require.NoError(t, fs.MkdirAll(testDataDir, 0o755))
fullPath := basePath + "/" + path
require.NoError(t, fs.MkdirAll(basePath, 0o755))
require.NoError(t, afero.WriteFile(fs, fullPath, content, 0o644))
}
}
@@ -92,13 +90,9 @@ func TestNewChecker(t *testing.T) {
testFile1: []byte("hello"),
testFile2: []byte("world"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: "/",
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/", fs)
require.NoError(t, err)
assert.NotNil(t, chk)
assert.Equal(t, FileCount(2), chk.FileCount())
@@ -108,11 +102,7 @@ func TestNewChecker(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
_, err := NewChecker(&CheckerOptions{
ManifestPath: "/nonexistent.mf",
BasePath: "/",
Fs: fs,
})
_, err := NewChecker("/nonexistent.mf", "/", fs)
assert.Error(t, err)
})
@@ -121,73 +111,11 @@ func TestNewChecker(t *testing.T) {
fs := afero.NewMemMapFs()
require.NoError(t, afero.WriteFile(fs, "/bad.mf", []byte("not a manifest"), 0o644))
_, err := NewChecker(&CheckerOptions{
ManifestPath: "/bad.mf",
BasePath: "/",
Fs: fs,
})
_, err := NewChecker("/bad.mf", "/", fs)
assert.Error(t, err)
})
}
func TestNewCheckerRequiredPaths(t *testing.T) {
t.Parallel()
for _, tc := range []struct {
name string
opts *CheckerOptions
want string
is error
}{
{
name: "nil options",
opts: nil,
want: "manifest path cannot be empty",
is: errManifestPathEmpty,
},
{
name: "empty manifest path",
opts: &CheckerOptions{BasePath: testDataDir},
want: "manifest path cannot be empty",
is: errManifestPathEmpty,
},
{
name: "empty base path",
opts: &CheckerOptions{ManifestPath: testManifestPath},
want: "base path cannot be empty",
is: errBasePathEmpty,
},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
chk, err := NewChecker(tc.opts)
require.ErrorIs(t, err, tc.is)
require.EqualError(t, err, tc.want)
assert.Nil(t, chk)
})
}
}
func TestNewCheckerNilFsUsesOsFs(t *testing.T) {
t.Parallel()
dir := t.TempDir()
manifestPath := filepath.Join(dir, "index.mf")
content := []byte("hello")
createTestManifest(t, afero.NewOsFs(), manifestPath, map[string][]byte{
testFile1: content,
})
require.NoError(t, os.WriteFile(filepath.Join(dir, testFile1), content, 0o600))
chk, err := NewChecker(&CheckerOptions{ManifestPath: manifestPath, BasePath: dir})
require.NoError(t, err)
results := make(chan Result, 1)
require.NoError(t, chk.Check(context.Background(), results, nil))
assert.Equal(t, StatusOK, (<-results).Status)
}
func TestCheckerFileCountAndTotalBytes(t *testing.T) {
t.Parallel()
@@ -197,13 +125,9 @@ func TestCheckerFileCountAndTotalBytes(t *testing.T) {
"medium.txt": []byte("hello world"),
"large.txt": bytes.Repeat([]byte("x"), 1000),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: "/",
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/", fs)
require.NoError(t, err)
assert.Equal(t, FileCount(3), chk.FileCount())
@@ -218,14 +142,10 @@ func TestCheckAllFilesOK(t *testing.T) {
testFile1: []byte("content one"),
testFile2: []byte("content two"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -252,17 +172,13 @@ func TestCheckMissingFile(t *testing.T) {
testExistsFile: []byte("I exist"),
"missing.txt": []byte("I don't exist on disk"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
// Only create one file
createFilesOnDisk(t, fs, map[string][]byte{
testExistsFile: []byte("I exist"),
})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -295,17 +211,13 @@ func TestCheckSizeMismatch(t *testing.T) {
files := map[string][]byte{
testFileName: []byte("original content"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
// Create file with different size
createFilesOnDisk(t, fs, map[string][]byte{
testFileName: []byte("short"),
})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -325,7 +237,7 @@ func TestCheckHashMismatch(t *testing.T) {
files := map[string][]byte{
testFileName: originalContent,
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
// Create file with same size but different content
differentContent := []byte("different contnt") // same length (16 bytes) but different
require.Len(t, differentContent, len(originalContent), "test requires same length")
@@ -333,11 +245,7 @@ func TestCheckHashMismatch(t *testing.T) {
testFileName: differentContent,
})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -357,14 +265,10 @@ func TestCheckWithProgress(t *testing.T) {
testFile1: bytes.Repeat([]byte("a"), 100),
testFile2: bytes.Repeat([]byte("b"), 200),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -401,14 +305,10 @@ func TestCheckContextCancellation(t *testing.T) {
files[string(rune('a'+i%26))+".txt"] = bytes.Repeat([]byte("x"), 1000)
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -427,7 +327,7 @@ func TestFindExtraFiles(t *testing.T) {
manifestFiles := map[string][]byte{
testFile1: []byte("in manifest"),
}
createTestManifest(t, fs, testManifestPath, manifestFiles)
createTestManifest(t, fs, "/manifest.mf", manifestFiles)
// Disk has file1 and file2
createFilesOnDisk(t, fs, map[string][]byte{
@@ -435,11 +335,7 @@ func TestFindExtraFiles(t *testing.T) {
testFile2: []byte("extra file"),
})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -457,120 +353,46 @@ func TestFindExtraFiles(t *testing.T) {
assert.Equal(t, "not in manifest", extras[0].Message)
}
// TestFindExtraFilesReportsHiddenFilesButNotManifest keeps the manifest
// inside the checked tree: hidden files and directories are reported, the
// manifest is not. The manifest is recognized by file identity, which needs
// the real filesystem.
func TestFindExtraFilesReportsHiddenFilesButNotManifest(t *testing.T) {
func TestFindExtraFilesSkipsManifestAndDotfiles(t *testing.T) {
t.Parallel()
dir := t.TempDir()
manifestPath := filepath.Join(dir, "index.mf")
fs := afero.NewOsFs()
createTestManifest(t, fs, manifestPath, map[string][]byte{
fs := afero.NewMemMapFs()
manifestFiles := map[string][]byte{
testFile1: []byte("in manifest"),
}
createTestManifest(t, fs, "/data/.index.mf", manifestFiles)
createFilesOnDisk(t, fs, map[string][]byte{
testFile1: []byte("in manifest"),
})
// Create dotfile and manifest that should be skipped
require.NoError(t, afero.WriteFile(fs, "/data/.hidden", []byte("hidden"), 0o644))
require.NoError(t, afero.WriteFile(fs, "/data/.config/settings", []byte("cfg"), 0o644))
// Create a real extra file
require.NoError(t, fs.MkdirAll("/data", 0o755))
require.NoError(t, afero.WriteFile(fs, "/data/extra.txt", []byte("extra"), 0o644))
unlisted := []RelFilePath{"extra.txt", ".hidden", ".git/config"}
for _, p := range append([]RelFilePath{testFile1}, unlisted...) {
path := filepath.Join(dir, string(p))
require.NoError(t, fs.MkdirAll(filepath.Dir(path), 0o750))
require.NoError(t, afero.WriteFile(fs, path, []byte("x"), 0o600))
}
chk, err := NewChecker(&CheckerOptions{
ManifestPath: manifestPath,
BasePath: dir,
Fs: fs,
})
chk, err := NewChecker("/data/.index.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
var extras []RelFilePath
for r := range results {
extras = append(extras, r.Path)
}
assert.ElementsMatch(t, unlisted, extras)
}
// TestFindExtraFilesSkipsManifestReachedThroughSymlink checks a tree whose
// index.mf is a symlink to the manifest kept outside the tree: the symlink is
// not reported.
func TestFindExtraFilesSkipsManifestReachedThroughSymlink(t *testing.T) {
t.Parallel()
dir := t.TempDir()
tree := filepath.Join(dir, "tree")
manifestPath := filepath.Join(dir, "real.mf")
linkPath := filepath.Join(tree, "index.mf")
fs := afero.NewOsFs()
createTestManifest(t, fs, manifestPath, map[string][]byte{testFile1: []byte("x")})
require.NoError(t, fs.MkdirAll(tree, 0o750))
require.NoError(t,
afero.WriteFile(fs, filepath.Join(tree, testFile1), []byte("x"), 0o600))
require.NoError(t, os.Symlink(manifestPath, linkPath))
chk, err := NewChecker(&CheckerOptions{
ManifestPath: linkPath,
BasePath: tree,
Fs: fs,
})
err = chk.FindExtraFiles(context.Background(), results)
require.NoError(t, err)
results := make(chan Result, 10)
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
var extras []RelFilePath
var extras []Result
for r := range results {
extras = append(extras, r.Path)
extras = append(extras, r)
}
assert.Empty(t, extras)
}
// TestFindExtraFilesSearchesBaseNamedThroughSymlink names the checked tree
// through a symlink to it: the files in the tree are searched, and the
// symlink itself is not reported.
func TestFindExtraFilesSearchesBaseNamedThroughSymlink(t *testing.T) {
t.Parallel()
dir := t.TempDir()
tree := filepath.Join(dir, "tree")
link := filepath.Join(dir, "link")
manifestPath := filepath.Join(dir, "index.mf")
fs := afero.NewOsFs()
createTestManifest(t, fs, manifestPath, map[string][]byte{testFile1: []byte("x")})
require.NoError(t, fs.MkdirAll(tree, 0o750))
for _, name := range []string{testFile1, testFile2} {
require.NoError(t,
afero.WriteFile(fs, filepath.Join(tree, name), []byte("x"), 0o600))
// Should only report extra.txt, not .hidden, .config/settings, or .index.mf
for _, e := range extras {
t.Logf("extra: %s", e.Path)
}
require.NoError(t, os.Symlink(tree, link))
assert.Len(t, extras, 1)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: manifestPath,
BasePath: link,
Fs: fs,
})
require.NoError(t, err)
results := make(chan Result, 10)
require.NoError(t, chk.FindExtraFiles(context.Background(), results))
var extras []RelFilePath
for r := range results {
extras = append(extras, r.Path)
if len(extras) > 0 {
assert.Equal(t, RelFilePath("extra.txt"), extras[0].Path)
}
assert.Equal(t, []RelFilePath{testFile2}, extras)
}
func TestFindExtraFilesContextCancellation(t *testing.T) {
@@ -578,14 +400,10 @@ func TestFindExtraFilesContextCancellation(t *testing.T) {
fs := afero.NewMemMapFs()
files := map[string][]byte{testFileName: []byte("data")}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
ctx, cancel := context.WithCancel(context.Background())
@@ -601,14 +419,10 @@ func TestCheckNilChannels(t *testing.T) {
fs := afero.NewMemMapFs()
files := map[string][]byte{testFileName: []byte("data")}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
// Should not panic with nil channels
@@ -621,14 +435,10 @@ func TestFindExtraFilesNilChannel(t *testing.T) {
fs := afero.NewMemMapFs()
files := map[string][]byte{testFileName: []byte("data")}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
// Should not panic with nil channel
@@ -645,7 +455,7 @@ func TestCheckSubdirectories(t *testing.T) {
"dir1/dir2/file2.txt": []byte("content2"),
"dir1/dir2/dir3/deep.txt": []byte("deep content"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
// Create files with full directory structure
for path, content := range files {
@@ -655,11 +465,7 @@ func TestCheckSubdirectories(t *testing.T) {
require.NoError(t, afero.WriteFile(fs, fullPath, content, 0o644))
}
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -687,17 +493,13 @@ func TestCheckMissingFileDetectedWithoutFallback(t *testing.T) {
testExistsFile: []byte("here"),
"missing.txt": []byte("not on disk"),
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
// Only create one file on disk
createFilesOnDisk(t, fs, map[string][]byte{
testExistsFile: []byte("here"),
})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
@@ -717,18 +519,77 @@ func TestCheckMissingFileDetectedWithoutFallback(t *testing.T) {
assert.Equal(t, 0, statusCounts[StatusError], "no files should be ERROR")
}
func TestFindExtraFilesSkipsDotfiles(t *testing.T) {
t.Parallel()
// Regression test for #16: FindExtraFiles should not report dotfiles
// or the manifest file itself as extra files.
fs := afero.NewMemMapFs()
files := map[string][]byte{
testFile1: []byte("in manifest"),
}
createTestManifest(t, fs, "/data/.index.mf", files)
createFilesOnDisk(t, fs, files)
// Add dotfiles and manifest file on disk
require.NoError(t, afero.WriteFile(fs, "/data/.hidden", []byte("dotfile"), 0o644))
require.NoError(t, fs.MkdirAll("/data/.git", 0o755))
require.NoError(t,
afero.WriteFile(fs, "/data/.git/config", []byte("git config"), 0o644))
chk, err := NewChecker("/data/.index.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
err = chk.FindExtraFiles(context.Background(), results)
require.NoError(t, err)
var extras []Result
for r := range results {
extras = append(extras, r)
}
// Should report NO extra files — dotfiles and manifest should be skipped
assert.Empty(t, extras,
"FindExtraFiles should not report dotfiles or manifest file as extra; got: %v",
extras)
}
func TestFindExtraFilesSkipsManifestFile(t *testing.T) {
t.Parallel()
// The manifest file itself should never be reported as extra
fs := afero.NewMemMapFs()
files := map[string][]byte{
testFile1: []byte("content"),
}
createTestManifest(t, fs, "/data/index.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker("/data/index.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 10)
err = chk.FindExtraFiles(context.Background(), results)
require.NoError(t, err)
var extras []Result
for r := range results {
extras = append(extras, r)
}
assert.Empty(t, extras,
"manifest file should not be reported as extra; got: %v", extras)
}
func TestCheckEmptyManifest(t *testing.T) {
t.Parallel()
fs := afero.NewMemMapFs()
// Create manifest with no files
createTestManifest(t, fs, testManifestPath, map[string][]byte{})
createTestManifest(t, fs, "/manifest.mf", map[string][]byte{})
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
assert.Equal(t, FileCount(0), chk.FileCount())
@@ -760,14 +621,10 @@ func TestCheckProgressRateLimited(t *testing.T) {
files[name] = []byte("content")
}
createTestManifest(t, fs, testManifestPath, files)
createTestManifest(t, fs, "/manifest.mf", files)
createFilesOnDisk(t, fs, files)
chk, err := NewChecker(&CheckerOptions{
ManifestPath: testManifestPath,
BasePath: testDataDir,
Fs: fs,
})
chk, err := NewChecker("/manifest.mf", "/data", fs)
require.NoError(t, err)
results := make(chan Result, 200)
mfer/constants.go
-24
View File
@@ -17,28 +17,4 @@ const (
// uuidLength is the length in bytes of a binary UUID.
uuidLength = 16
// Numbers in mf.proto of MFFile.files and of the MFFilePath fields
// that decoding sets aside a fixed amount of memory for.
filesFieldNumber = 101
hashesFieldNumber = 3
mimeTypeFieldNumber = 301
mtimeFieldNumber = 302
ctimeFieldNumber = 303
// Bytes decoding sets aside for each file entry, hash, timestamp and
// MIME type, however short its encoding. checkDecodedSize refuses an
// inner message for which these add up to more than maxDecodedGrowth
// times its size.
decodedFileEntrySize = 160
decodedHashSize = 112
decodedTimestampSize = 64
decodedMIMETypeSize = 16
// Each file entry mfer writes holds a path of at least one byte, a
// multihash at least as long as SHA-256's 34 bytes (AddFileWithHash
// refuses shorter ones) and a modification time: at least 47 bytes,
// counted at 336. So its manifests add up to at most about 7.15 times
// their size, and this limit is about 12% above that.
maxDecodedGrowth = 8
)
mfer/deserialize.go
+6 -108
View File
@@ -11,7 +11,6 @@ import (
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/spf13/afero"
"google.golang.org/protobuf/encoding/protowire"
"google.golang.org/protobuf/proto"
"sneak.berlin/go/mfer/internal/bork"
"sneak.berlin/go/mfer/internal/log"
@@ -28,8 +27,6 @@ var (
errUUIDMismatch = errors.New("outer and inner UUID mismatch")
errInvalidFileFormat = errors.New("invalid file format")
errInvalidManifestPath = errors.New("manifest contains invalid path")
errDecodedTooLarge = errors.New(
"manifest would take too much memory to decode")
)
// validateUUID checks that the byte slice is a valid UUID (16 bytes, parseable).
@@ -157,85 +154,6 @@ func (m *manifest) decompressInner() ([]byte, error) {
return dat, nil
}
// checkDecodedSize refuses an encoded inner message whose file entries,
// hashes, timestamps and MIME types would take more than maxDecodedGrowth
// times its size to decode. Decoding sets aside a fixed amount for each,
// however short its encoding, so a message of empty ones would take about
// 50 times its size.
func checkDecodedSize(inner []byte) error {
limit := maxDecodedGrowth * int64(len(inner))
var decoded int64
add := func(size int64) error {
decoded += size
if decoded > limit {
return errDecodedTooLarge
}
return nil
}
return forEachBytesField(inner, func(num protowire.Number, entry []byte) error {
if num != filesFieldNumber {
return nil
}
err := add(decodedFileEntrySize)
if err != nil {
return err
}
return forEachBytesField(entry, func(num protowire.Number, _ []byte) error {
if num == hashesFieldNumber {
return add(decodedHashSize)
}
if num == mtimeFieldNumber || num == ctimeFieldNumber {
return add(decodedTimestampSize)
}
if num == mimeTypeFieldNumber {
return add(decodedMIMETypeSize)
}
return nil
})
})
}
// forEachBytesField calls fn with the number and value of each
// length-delimited field in the encoded message msg, and fails if msg is
// malformed.
func forEachBytesField(
msg []byte, fn func(num protowire.Number, value []byte) error,
) error {
for len(msg) > 0 {
num, wireType, tagLen := protowire.ConsumeTag(msg)
if tagLen < 0 {
return protowire.ParseError(tagLen)
}
valueLen := protowire.ConsumeFieldValue(num, wireType, msg[tagLen:])
if valueLen < 0 {
return protowire.ParseError(valueLen)
}
if wireType == protowire.BytesType {
value, _ := protowire.ConsumeBytes(msg[tagLen:])
err := fn(num, value)
if err != nil {
return err
}
}
msg = msg[tagLen+valueLen:]
}
return nil
}
func (m *manifest) deserializeInner() error {
err := m.validateOuterHeader()
if err != nil {
@@ -259,16 +177,10 @@ func (m *manifest) deserializeInner() error {
return bork.ErrFileTruncated
}
err = checkDecodedSize(dat)
if err != nil {
return fmt.Errorf("deserialize: unmarshal inner: %w", err)
}
// Deserialize inner message
m.pbInner = new(MFFile)
// Unknown fields would cost memory; mfer never writes a loaded manifest out.
err = proto.UnmarshalOptions{DiscardUnknown: true}.Unmarshal(dat, m.pbInner)
err = proto.Unmarshal(dat, m.pbInner)
if err != nil {
return fmt.Errorf("deserialize: unmarshal inner: %w", err)
}
@@ -337,8 +249,7 @@ func NewManifestFromReader(input io.Reader) (*manifest, error) {
// deserialize outer:
m.pbOuter = new(MFFileOuter)
// Unknown fields would cost memory; mfer never writes a loaded manifest out.
err = proto.UnmarshalOptions{DiscardUnknown: true}.Unmarshal(dat, m.pbOuter)
err = proto.Unmarshal(dat, m.pbOuter)
if err != nil {
return nil, err
}
@@ -352,29 +263,16 @@ func NewManifestFromReader(input io.Reader) (*manifest, error) {
return m, nil
}
// ManifestFromFileOptions configures NewManifestFromFile.
type ManifestFromFileOptions struct {
// Path is the manifest file to read (required).
Path string
// Fs is the filesystem to use, defaults to OsFs if nil.
Fs afero.Fs
}
// NewManifestFromFile reads a manifest from a file. It returns an error if
// opts is nil or its path is empty.
// NewManifestFromFile reads a manifest from a file path using the given filesystem.
// If fs is nil, the real filesystem (OsFs) is used.
//
//nolint:revive // unexported-return: exporting manifest is owner question 13
func NewManifestFromFile(opts *ManifestFromFileOptions) (*manifest, error) {
if opts == nil || opts.Path == "" {
return nil, errManifestPathEmpty
}
fs := opts.Fs
func NewManifestFromFile(fs afero.Fs, path string) (*manifest, error) {
if fs == nil {
fs = afero.NewOsFs()
}
f, err := fs.Open(opts.Path)
f, err := fs.Open(path)
if err != nil {
return nil, err
}
mfer/deserialize_fuzz_test.go
+5 -12
View File
@@ -54,15 +54,9 @@ func FuzzNewManifestFromReader(f *testing.F) {
}
// It also keeps a few copies of its input. Buffers grow by
// copying, so reaching those sizes allocates up to about six times
// them in total. Decoding the decompressed data takes up to
// maxDecodedGrowth times its size for file entries, hashes,
// timestamps and MIME types, and drops fields it does not know.
// The strings and bytes it copies out of it, such as many one-byte
// values in one hash, take up to about five times more under the
// race detector, which pads every small copy to 16 bytes, and about
// half that without it. Twenty times the input and the
// decompressed data leaves room for all of that.
// copying, so reaching those sizes allocates a few times them in
// total: sixteen times the input and the decompressed data leaves
// room for that.
//
// The decoder also sets aside a new buffer of one to two times the
// window for each frame that asks for a larger window than the
@@ -77,9 +71,8 @@ func FuzzNewManifestFromReader(f *testing.F) {
// fails if the decoder accepts windows of twice zstdWindowSize; the
// seed whose two frames together exceed MaxDecompressedSize fails
// if the decoder decodes them in full instead of stopping at the
// declared size; the seeds of empty file entries and of a file
// entry of empty hashes fail if the parser decodes them.
limit := 20*(uint64(len(data))+decompressed) + 24*zstdWindowSize
// declared size.
limit := 16*(uint64(len(data))+decompressed) + 24*zstdWindowSize
allocated := after.TotalAlloc - before.TotalAlloc
if allocated > limit {
mfer/deserialize_path_test.go
+1 -114
View File
@@ -6,14 +6,10 @@ import (
"context"
"crypto/sha256"
"fmt"
"strconv"
"strings"
"testing"
"time"
"github.com/google/uuid"
"github.com/klauspost/compress/zstd"
"github.com/multiformats/go-multihash"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/protobuf/encoding/protowire"
@@ -118,119 +114,10 @@ func TestDeserializeRejectsInvalidEntryPaths(t *testing.T) {
}
}
// Entries of a path, an empty hash, an empty MIME type and empty modification
// and change times are counted at 416 bytes each (160 + 112 + 16 + 64 + 64)
// and take 16 bytes plus the path to encode. A 35-character path makes that
// 51 bytes, about 8.2 times: refused, and leaving any one of the five
// uncounted, even the MIME type, brings it under 8. A 37-character path makes
// it 53 bytes, about 7.8 times: loaded.
func TestDeserializeRefusesEntriesThatDecodeTooLarge(t *testing.T) {
t.Parallel()
tests := []struct {
pathLen int
refused bool
}{
{35, true},
{37, false},
}
for _, tt := range tests {
t.Run(strconv.Itoa(tt.pathLen), func(t *testing.T) {
t.Parallel()
entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
entry = protowire.AppendString(entry, strings.Repeat("a", tt.pathLen))
entry = protowire.AppendTag(entry, 3, protowire.BytesType) // MFFilePath.hashes
entry = protowire.AppendBytes(entry, nil)
entry = protowire.AppendTag(entry, 301, protowire.BytesType) // MFFilePath.mimeType
entry = protowire.AppendBytes(entry, nil)
entry = protowire.AppendTag(entry, 302, protowire.BytesType) // MFFilePath.mtime
entry = protowire.AppendBytes(entry, nil)
entry = protowire.AppendTag(entry, 303, protowire.BytesType) // MFFilePath.ctime
entry = protowire.AppendBytes(entry, nil)
id := uuid.New()
inner := protowire.AppendTag(nil, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
for range 1000 {
inner = protowire.AppendTag(inner, 101, protowire.BytesType) // MFFile.files
inner = protowire.AppendBytes(inner, entry)
}
_, err := NewManifestFromReader(bytes.NewReader(wrapInner(t, id, inner)))
if tt.refused {
require.ErrorIs(t, err, errDecodedTooLarge)
} else {
require.NoError(t, err)
}
})
}
}
// Fields the decoder does not know are dropped, in the outer message, the inner
// message and a file entry, so that they take no memory once loaded.
func TestDeserializeDropsUnknownFields(t *testing.T) {
t.Parallel()
unknown := protowire.AppendTag(nil, 99, protowire.BytesType) // in no message
unknown = protowire.AppendBytes(unknown, []byte("not known"))
entry := protowire.AppendTag(nil, 1, protowire.BytesType) // MFFilePath.path
entry = protowire.AppendString(entry, "a")
entry = append(entry, unknown...)
id := uuid.New()
inner := protowire.AppendTag(nil, 101, protowire.BytesType) // MFFile.files
inner = protowire.AppendBytes(inner, entry)
inner = protowire.AppendTag(inner, 102, protowire.BytesType) // MFFile.uuid
inner = protowire.AppendBytes(inner, id[:])
inner = append(inner, unknown...)
data := wrapInner(t, id, inner)
data = append(data, unknown...) // the outer message ends the file
m, err := NewManifestFromReader(bytes.NewReader(data))
require.NoError(t, err)
require.Len(t, m.Files(), 1)
assert.Empty(t, m.pbOuter.ProtoReflect().GetUnknown())
assert.Empty(t, m.pbInner.ProtoReflect().GetUnknown())
assert.Empty(t, m.Files()[0].ProtoReflect().GetUnknown())
}
// Many empty files with names of at most three characters and modification
// times at the epoch make about the densest manifest mfer writes: it takes
// about 7 times its size to decode, and still loads. A signature would not
// change the inner message, so none is added.
func TestDeserializeLoadsDensestManifest(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
b := NewBuilder()
b.SetIncludeTimestamps(true)
const files = 10000
for i := range files {
name := RelFilePath(strconv.FormatInt(int64(i), 36))
require.NoError(t, b.AddFileWithHash(name, 0, ModTime(time.Unix(0, 0)), hash))
}
var buf bytes.Buffer
require.NoError(t, b.Build(context.Background(), &buf))
m, err := NewManifestFromReader(&buf)
require.NoError(t, err)
assert.Len(t, m.Files(), files)
}
func TestDeserializeValidManifestRoundTrips(t *testing.T) {
t.Parallel()
hash, err := multihash.Encode(make([]byte, sha256.Size), multihash.SHA2_256)
require.NoError(t, err)
hash := make([]byte, 34) // multihash: 2-byte prefix + 32-byte SHA-256
b := NewBuilder()
require.NoError(t, b.AddFileWithHash("dir/file.txt", 123, ModTime{}, hash))
mfer/mf.pb.go
+1 -1
View File
@@ -587,7 +587,7 @@ const file_mf_proto_rawDesc = "" +
"\fVERSION_NONE\x10\x00\x12\x0f\n" +
"\vVERSION_ONE\x10\x01B\f\n" +
"\n" +
"_createdAtB\x1bZ\x19sneak.berlin/go/mfer/mferb\x06proto3"
"_createdAtB\x1dZ\x1bgit.eeqj.de/sneak/mfer/mferb\x06proto3"
var (
file_mf_proto_rawDescOnce sync.Once
mfer/mf.proto
+1 -1
View File
@@ -1,6 +1,6 @@
syntax = "proto3";
option go_package = "sneak.berlin/go/mfer/mfer";
option go_package = "git.eeqj.de/sneak/mfer/mfer";
message Timestamp {
int64 seconds = 1;
mfer/mf.proto.sha256
-1
View File
@@ -1 +0,0 @@
fa6fceaba5553c8667c631994535fe5c307c8adb19f3ad289babf79a0f438650 mf.proto
mfer/mf_test.go
-29
View File
@@ -1,29 +0,0 @@
package mfer_test
import (
"crypto/sha256"
"encoding/hex"
"os"
"strings"
"testing"
"github.com/stretchr/testify/require"
)
// mf.pb.go is generated from mf.proto and committed. `make generate`
// records the hash of the mf.proto it generated from in mf.proto.sha256.
func TestGeneratedCodeMatchesProto(t *testing.T) {
t.Parallel()
proto, err := os.ReadFile("mf.proto")
require.NoError(t, err)
recorded, err := os.ReadFile("mf.proto.sha256")
require.NoError(t, err)
recordedHash, _, _ := strings.Cut(string(recorded), " ")
sum := sha256.Sum256(proto)
require.Equal(t, recordedHash, hex.EncodeToString(sum[:]),
"mfer/mf.proto has changed since mfer/mf.pb.go was generated "+
"from it: run `make generate` and commit the result")
}
mfer/scanner.go
+1 -22
View File
@@ -4,7 +4,6 @@ import (
"context"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
@@ -58,8 +57,6 @@ type ScannerOptions struct {
SigningOptions *SigningOptions
// Seed, if set, derives a deterministic UUID from this seed.
Seed string
// ExcludePaths lists files to leave out of the listing, matched with os.SameFile.
ExcludePaths []string
}
// FileEntry represents a file that has been enumerated.
@@ -78,7 +75,6 @@ type Scanner struct {
totalBytes FileSize // cached sum of all file sizes
options *ScannerOptions
fs afero.Fs
excluded []fs.FileInfo // the files named in ExcludePaths that exist
}
// NewScanner creates a new Scanner with default options.
@@ -97,22 +93,11 @@ func NewScannerWithOptions(opts *ScannerOptions) *Scanner {
fs = afero.NewOsFs()
}
s := &Scanner{
return &Scanner{
files: make([]*FileEntry, 0),
options: opts,
fs: fs,
}
// A path that cannot be stat'd, normally because no file is there,
// needs no leaving out.
for _, p := range opts.ExcludePaths {
info, err := s.fs.Stat(p)
if err == nil {
s.excluded = append(s.excluded, info)
}
}
return s
}
// EnumerateFile adds a single file to the scanner, calling stat() to get metadata.
@@ -450,12 +435,6 @@ func (s *Scanner) enumerateFileWithInfo(
info = realInfo
}
for _, excluded := range s.excluded {
if os.SameFile(info, excluded) {
return nil
}
}
entry := &FileEntry{
Path: RelFilePath(cleanPath),
AbsPath: AbsFilePath(absPath),
mfer/testdata/fuzz/FuzzNewManifestFromReader/empty-file-entries
Vendored
-2
View File
@@ -1,2 +0,0 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06\xff\xff\xff\x03\xc2\x06 {\x16\xbdu\xa0\xa2\x11\xfcH\xef*\x1b7\r\x99\xefb\x04\x02g\n\xa9\xf3B5\xe5p\x96\x8c\x8c\xac\x0e\xca\x06\x10\x03Q\xb2\xd0\x19`F\xc1\xb1\xc0Z\xf4x\xf4g^\xba\f\xa1\x06(\xb5/\xfd\x04h\x04\x01\x00d\x01\xb2\x06\x10\x03Q\xb2\xd0\x19`F\xc1\xb1\xc0Z\xf4x\xf4g^\xaa\x06\x00\x01T\x13\x024\xce\xff\rL\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15M\x00\x00\x00\x01T\x00\x044\xfc\xff\x153\xea\a\xb4")
mfer/testdata/fuzz/FuzzNewManifestFromReader/file-entry-of-empty-hashes
Vendored
-2
View File
@@ -1,2 +0,0 @@
go test fuzz v1
[]byte("ZNAVSRFG\xa8\x06\x01\xb0\x06\x01\xb8\x06\xff\xff\xff\x03\xc2\x06 .\xcd\x11|0\xfcP\xe5\x1b\xe3\xc6Ӡ\xcdڤx\xcd\x169t\x1a9~ǽB\xc9\xe8G`\x05\xca\x06\x10\x11*!\x0e\x95EF\xb8\xbd\x9f\xde\x12MF\r\x99\xba\f\xa6\x06(\xb5/\xfd\x04h,\x01\x00\xb4\x01\xb2\x06\x10\x11*!\x0e\x95EF\xb8\xbd\x9f\xde\x12MF\r\x99\xaa\x06\xe6\xff\xff\x03\x1a\x00\x01T\x14\x024\x8b\xff\x17L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15L\x00\x00\x00\x01T\x00\x044\xfd\xff\x15M\x00\x00\x00\x01T\x00\x044\xfc\xff\x15\x02\xd1.\xe3")
package.json
+1
View File
@@ -1,5 +1,6 @@
{
"name": "mfer",
"version": "0.1.0",
"private": true,
"description": "Development tooling for the mfer repository: prettier, used by script/fmt and script/fmt-check to format and verify Markdown and JSON.",
"license": "WTFPL",
script/bootstrap
+3 -6
View File
@@ -132,18 +132,15 @@ main() {
# ---- JS / docs repos ----
# This is a Go repo, but node and yarn are required anyway: prettier
# formats the Markdown and JSON, and script/fmt-check verifies it.
# The version is pinned by package.json/yarn.lock: yarn checks every
# package it fetches against its yarn.lock integrity hash, and
# --frozen-lockfile fails instead of rewriting a yarn.lock that no
# longer matches package.json.
# The version is pinned by package.json/yarn.lock, whose integrity
# hashes --frozen-lockfile enforces.
ensure_node
ensure_yarn
install_js_deps
# ---- Go repos ----
if missing go; then pkg_install go golang go go; fi
# No golangci-lint: script/lint runs it in Docker only. No gofumpt:
# script/gofumpt builds the version tools/go.mod requires.
# No golangci-lint: script/lint runs it in Docker only.
go mod download
# ---- Python repos ----
script/build
-20
View File
@@ -1,20 +0,0 @@
#!/bin/sh
# script/build: build the mfer binary into bin/mfer, stamped with the
# revision `mfer version` prints, derived as script/docker derives it.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
go build -tags urfave_cli_no_docs \
-ldflags "-X main.Gitrev=$version" \
-o bin/mfer ./cmd/mfer
}
main "$@"
script/fmt
+13 -3
View File
@@ -5,11 +5,21 @@ set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
# committed, so this is normally a no-op).
ensure_pb() {
if [ ! -f mfer/mf.pb.go ] ||
[ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then
(cd mfer && go generate .)
fi
}
main() {
cd "$ROOT"
# Go, then Markdown and JSON, through the same scripts script/fmt-check
# uses, so both see the same files and the same tool versions.
"$SCRIPT_DIR/gofumpt" --write
ensure_pb
gofumpt -l -w mfer internal cmd
# Markdown and JSON, over the same file set script/fmt-check verifies.
"$SCRIPT_DIR/prettier" --write
}
script/fmt-check
+2 -2
View File
@@ -1,13 +1,13 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing: Go via script/gofumpt,
# script/fmt, but fails instead of writing: Go via script/fmt-check-go,
# Markdown and JSON via script/prettier.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
main() {
"$SCRIPT_DIR/gofumpt" --check
"$SCRIPT_DIR/fmt-check-go"
"$SCRIPT_DIR/prettier" --check
}
script/fmt-check-go
Executable
+29
View File
@@ -0,0 +1,29 @@
#!/bin/sh
# script/fmt-check-go: check Go formatting (read-only). Split out from
# script/fmt-check so the Docker lint stage, whose image has no node and
# therefore no prettier, can run the Go half on its own.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
# committed, so this is normally a no-op).
ensure_pb() {
if [ ! -f mfer/mf.pb.go ] ||
[ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then
(cd mfer && go generate .)
fi
}
main() {
cd "$ROOT"
ensure_pb
if [ -n "$(gofmt -l .)" ]; then
echo "gofmt: files need formatting:" >&2
gofmt -l . >&2
exit 1
fi
}
main "$@"
script/generate
-55
View File
@@ -1,55 +0,0 @@
#!/bin/sh
# script/generate: regenerate mfer/mf.pb.go from mfer/mf.proto, and record
# the hash of that mf.proto in mfer/mf.proto.sha256. Nothing else
# regenerates mf.pb.go: it is committed, so building and checking need no
# protoc. A test fails while mf.proto no longer matches the recorded hash.
#
# Needs exactly the protoc and protoc-gen-go versions named in the header of
# the committed mf.pb.go (README.md says how to install them). Another
# version writes a different mf.pb.go, so the script refuses to run.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# protoc 33.4 names itself v6.33.4 in the mf.pb.go header.
PROTOC_VERSION="33.4"
PROTOC_GEN_GO_VERSION="v1.36.11"
# require_version <command> <its exact --version output>
require_version() {
actual="$("$1" --version 2>/dev/null || true)"
if [ "$actual" != "$2" ]; then
echo "generate: needs $2 on PATH, found: ${actual:-none}" >&2
echo " README.md says how to install it." >&2
exit 1
fi
}
# sha256 <file>: print "<hash> <file>", with sha256sum, or with shasum
# where there is no sha256sum.
sha256() {
if command -v sha256sum >/dev/null 2>&1; then
sha256sum "$1"
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 "$1"
else
echo "generate: needs sha256sum or shasum on PATH" >&2
exit 1
fi
}
main() {
cd "$ROOT/mfer"
# `go install` puts protoc-gen-go in $(go env GOPATH)/bin, which is
# often not on PATH.
PATH="$PATH:$(go env GOPATH)/bin"
require_version protoc "libprotoc $PROTOC_VERSION"
require_version protoc-gen-go "protoc-gen-go $PROTOC_GEN_GO_VERSION"
# Hashed before regenerating, so a missing hash tool stops the script
# before it changes anything. Regenerating leaves mf.proto as it is.
proto_hash="$(sha256 mf.proto)"
go generate .
echo "$proto_hash" >mf.proto.sha256
}
main "$@"
script/gofumpt
-43
View File
@@ -1,43 +0,0 @@
#!/bin/sh
# script/gofumpt: run gofumpt over this repo's Go files.
#
# Takes exactly one mode argument, --write or --check, and runs the same
# gofumpt version over the same files in both modes. script/fmt and
# script/fmt-check both go through here, and so does the Docker lint
# stage, so what gets formatted and what gets verified cannot drift
# apart.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# gofumpt v0.12.0, 2026-10-04, required by tools/go.mod. `go tool` run in
# tools/ builds it from source checked against the hashes in tools/go.sum,
# so neither a developer machine nor the lint image needs it installed.
usage() {
echo "usage: script/gofumpt --write|--check" >&2
exit 2
}
main() {
[ "$#" -eq 1 ] || usage
cd "$ROOT/tools"
# Every Go file in the repo, from $ROOT down. gofumpt holds generated
# files, such as mfer/mf.pb.go, to gofmt's rules only.
case "$1" in
--write) go tool gofumpt -l -w "$ROOT" ;;
--check)
# Own line: a failing command inside `[ -n "$(...)" ]` does
# not trip `set -e`, so a gofumpt that never ran would pass.
unformatted="$(go tool gofumpt -l "$ROOT")"
if [ -n "$unformatted" ]; then
echo "gofumpt: files need formatting (run make fmt):" >&2
echo "$unformatted" >&2
exit 1
fi
;;
*) usage ;;
esac
}
main "$@"
script/prettier
+24 -16
View File
@@ -18,9 +18,24 @@ usage() {
exit 2
}
# Only the prettier yarn installed from yarn.lock, never one on PATH: a
# different version formats differently.
PRETTIER="$ROOT/node_modules/.bin/prettier"
# Prefer the version pinned by package.json/yarn.lock so that CI and
# developer machines format identically. Fall back to a prettier on PATH,
# but say so, because a different version formats differently.
find_prettier() {
if [ -x "$ROOT/node_modules/.bin/prettier" ]; then
printf '%s\n' "$ROOT/node_modules/.bin/prettier"
return 0
fi
if command -v prettier >/dev/null 2>&1; then
echo "prettier: node_modules/.bin/prettier is absent; using the" \
"prettier on PATH, which may be a different version than the" \
"one pinned in package.json. Run script/bootstrap to install" \
"the pinned version." >&2
command -v prettier
return 0
fi
return 1
}
main() {
[ "$#" -eq 1 ] || usage
@@ -31,17 +46,10 @@ main() {
cd "$ROOT"
# node_modules keeps the old prettier after package.json moves to a new
# one, until script/bootstrap runs again, so compare the two. A prettier
# that runs also means node is on PATH, which reading package.json needs.
if ! installed="$("$PRETTIER" --version 2>/dev/null)"; then
echo "prettier: not installed; run script/bootstrap" >&2
exit 1
fi
pinned="$(node -p 'require("./package.json").devDependencies.prettier')"
if [ "$installed" != "$pinned" ]; then
echo "prettier: package.json pins $pinned but $installed is" \
"installed; run script/bootstrap" >&2
if ! prettier_bin="$(find_prettier)"; then
echo "prettier: not found." >&2
echo " Install it with: script/bootstrap" >&2
echo " (installs the version pinned in package.json/yarn.lock)" >&2
exit 1
fi
@@ -54,8 +62,8 @@ main() {
# patterns always match at least one tracked file (README.md,
# package.json), so an empty match means the glob broke, and prettier
# erroring out is exactly what we want rather than a vacuous pass.
"$PRETTIER" "$mode" "**/*.md"
"$PRETTIER" "$mode" "**/*.json"
"$prettier_bin" "$mode" "**/*.md"
"$prettier_bin" "$mode" "**/*.json"
}
main "$@"
script/test
+11
View File
@@ -4,8 +4,19 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Regenerate mfer/mf.pb.go from mfer/mf.proto if it is missing or stale
# (mirrors the old Makefile prerequisite; the generated file is
# committed, so this is normally a no-op).
ensure_pb() {
if [ ! -f mfer/mf.pb.go ] ||
[ -n "$(find mfer/mf.proto -newer mfer/mf.pb.go 2>/dev/null)" ]; then
(cd mfer && go generate .)
fi
}
main() {
cd "$ROOT"
ensure_pb
go test -timeout 30s -race -cover ./... ||
{
echo "--- Rerunning with -v for details ---"
tools/go.mod
-15
View File
@@ -1,15 +0,0 @@
// The developer tools this repo runs with `go tool`, kept out of the mfer
// module so they add nothing to what mfer's users download. `go tool` builds
// exactly the source whose hashes go.sum here records.
module sneak.berlin/go/mfer/tools
go 1.26.0
tool mvdan.cc/gofumpt
require (
golang.org/x/mod v0.40.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/tools v0.49.0 // indirect
mvdan.cc/gofumpt v0.12.0 // indirect
)
tools/go.sum
-20
View File
@@ -1,20 +0,0 @@
github.com/go-quicktest/qt v1.102.0 h1:HSQxCeh5YZH3EL3W39ixjtyaEhcWSXQHtHnMBzSs474=
github.com/go-quicktest/qt v1.102.0/go.mod h1:p4lGIVX+8Wa6ZPNDvqcxq36XpUDLh42FLetFU7odllI=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g=
github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs=
golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs=
golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/tools v0.49.0 h1:3NI7VXzL9+1WZD52Dx2ttoPwD5DWrFGpl9mFZDlmisI=
golang.org/x/tools v0.49.0/go.mod h1:SJNXV9DBKT0UbdttsQjbfJlAE/q+y36++zo3uL3N0Oo=
mvdan.cc/gofumpt v0.12.0 h1:1Lbudkz2kpM9Cjz2pL4M19u7q+GaEhCTNf7N9mfpcho=
mvdan.cc/gofumpt v0.12.0/go.mod h1:SmBHHrljiZu/uoypeKup3rFzP6eoC9UwCp2iH5E3jZA=