2 Commits
Author SHA1 Message Date
sneak 08139beab2 fetch refuses a manifest that lists another file's temp name (closes #151)
check / check (push) Failing after 2s
fetch downloads each file to a temp name beside it and first removes
whatever is there. A manifest listing both a.txt and .a.txt.tmp had
fetch delete the second while fetching the first, then exit 0 with a
tree check rejects.

The refusal of a manifest that lists the saved manifest's own name or
temp name now covers this too: a listed file, or a directory a listed
file is in, may not sit at any name fetch writes besides the listed
files themselves. Temp names come from tempPathFor, names are compared
ignoring case as before, and the refusal still happens before the
destination is created or any file requested.

Model: opus-5-5
2026-10-04 17:05:02 +00:00
clawbot d3394bd2a2 Makefile: thin shims only, add make build (closes #73)
check / check (push) Failing after 2s
The Makefile is now one shim per script/ entrypoint, in the order of
the canonical model Makefile, plus build, generate and fuzz. The new
script/build writes bin/mfer with the urfave_cli_no_docs tag and the
git describe revision that script/docker stamps. script/generate now
adds the Go bin directory to PATH itself, and the Docker lint stage
calls script/gofumpt --check directly. Removed: the tarball-caching
rules and their ignore entries, godoc, ci, fixme, clean, run, default,
fmt-check-go, fmt-check-md, and bin/gitrev.sh, which only the
Makefile called.

Model: opus-5-5
2026-10-04 18:38:13 +02:00
11 changed files with 172 additions and 137 deletions

No files matched your search

+2 -3
View File
@@ -59,6 +59,5 @@
**/.vscode **/.vscode
**/*.sublime-* **/*.sublime-*
# This repo's own host-built archives (Makefile). # This repo's own host-built binary (make build).
*.tmp /bin/mfer
*.dockerimage
-5
View File
@@ -1,11 +1,6 @@
/bin/mfer /bin/mfer
/tmp /tmp
/node_modules/ /node_modules/
*.tmp
*.dockerimage
/vendor
vendor.tzst
modcache.tzst
# Generated manifest files # Generated manifest files
/index.mf /index.mf
+1 -1
View File
@@ -11,7 +11,7 @@ COPY . .
# Go half of fmt-check only: this image has no node, so no prettier. The # Go half of fmt-check only: this image has no node, so no prettier. The
# markdown half runs in the mdfmt stage below. The image has no gofumpt # markdown half runs in the mdfmt stage below. The image has no gofumpt
# either; script/gofumpt builds the version it pins with `go run`. # either; script/gofumpt builds the version it pins with `go run`.
RUN make fmt-check-go RUN script/gofumpt --check
# The linter directly, not `make lint`: script/lint builds this stage, and # The linter directly, not `make lint`: script/lint builds this stage, and
# there is no docker inside this build. # there is no docker inside this build.
RUN golangci-lint run --config .golangci.yml ./... RUN golangci-lint run --config .golangci.yml ./...
+21 -76
View File
@@ -1,20 +1,8 @@
export DOCKER_BUILDKIT := 1 .PHONY: bootstrap setup test lint fmt fmt-check check docker hooks build generate fuzz
export PROGRESS_NO_TRUNC := 1
GOPATH := $(shell go env GOPATH)
export PATH := $(PATH):$(GOPATH)/bin
SOURCEFILES := mfer/*.go mfer/*.proto internal/*/*.go cmd/*/*.go go.mod go.sum
ARCH := $(shell uname -m)
GITREV_BUILD := $(shell bash $(PWD)/bin/gitrev.sh 2>/dev/null || echo unknown)
APPNAME := mfer
VERSION := 0.1.0
export DOCKER_IMAGE_CACHE_DIR := $(HOME)/Library/Caches/Docker/$(APPNAME)-$(ARCH)
GOLDFLAGS += -X main.Version=$(VERSION)
GOLDFLAGS += -X main.Gitrev=$(GITREV_BUILD)
GOFLAGS := -ldflags "$(GOLDFLAGS)"
.PHONY: bootstrap setup docker default run ci test fuzz check lint fmt fmt-check fmt-check-go fmt-check-md hooks fixme generate # Makefile targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section
default: fmt test # of README.md).
bootstrap: bootstrap:
@script/bootstrap @script/bootstrap
@@ -22,75 +10,32 @@ bootstrap:
setup: setup:
@script/setup @script/setup
run: ./bin/mfer
./$<
./$< gen
ci: test
test: test:
@script/test @script/test
fuzz:
@script/fuzz
fixme:
@grep -nir fixme . | grep -v Makefile
check:
@script/check
fmt-check:
@script/fmt-check
# Halves of fmt-check, for environments that have only one toolchain:
# the Docker lint stage has Go but no node, the markdown stage the reverse.
fmt-check-go:
@script/gofumpt --check
fmt-check-md:
@script/prettier --check
hooks:
@script/install-precommit
generate:
@script/generate
bin/mfer: $(SOURCEFILES)
cd cmd/mfer && go build -tags urfave_cli_no_docs -o ../../bin/mfer $(GOFLAGS) .
clean:
rm -rfv bin/mfer cmd/mfer/mfer *.dockerimage
fmt:
@script/fmt
lint: lint:
@script/lint @script/lint
fmt:
@script/fmt
fmt-check:
@script/fmt-check
check:
@script/check
docker: docker:
@script/docker @script/docker
sneak-mfer.$(ARCH).tzst.dockerimage: $(SOURCEFILES) vendor.tzst modcache.tzst hooks:
docker build --progress plain --build-arg GITREV=$(GITREV_BUILD) -t sneak/mfer . @script/install-precommit
docker save sneak/mfer | pv | zstdmt -19 > $@
du -sh $@
godoc: build:
open http://127.0.0.1:6060 @script/build
godoc -http=:6060
vendor.tzst: go.mod go.sum generate:
go mod tidy @script/generate
go mod vendor
cd vendor && tar -c . | pv | zstdmt -19 > $(PWD)/$@.tmp
rm -rf vendor
mv $@.tmp $@
modcache.tzst: go.mod go.sum fuzz:
go mod tidy @script/fuzz
cd $(HOME)/go/pkg && chmod -R u+rw . && rm -rf mod sumdb
go mod download -x
cd $(shell go env GOMODCACHE) && tar -c . | pv | zstdmt -19 > $(PWD)/$@.tmp
mv $@.tmp $@
+14 -8
View File
@@ -29,7 +29,7 @@ is committed, so no `protoc` toolchain is required:
```sh ```sh
git clone https://git.eeqj.de/sneak/mfer.git git clone https://git.eeqj.de/sneak/mfer.git
cd mfer cd mfer
go build -o bin/mfer ./cmd/mfer make build
``` ```
Generate a manifest for a directory tree, verify it later, and fetch a published Generate a manifest for a directory tree, verify it later, and fetch a published
@@ -76,6 +76,9 @@ provide:
such as `script/docker` such as `script/docker`
- `script/test` — run the test suite (`go test`); one test fails when - `script/test` — run the test suite (`go test`); one test fails when
`mfer/mf.proto` no longer matches the hash `script/generate` recorded `mfer/mf.proto` no longer matches the hash `script/generate` recorded
- `script/build` (`make build`) — build the `mfer` binary into `bin/mfer`,
stamped with the revision `mfer version` prints: the output of
`git describe --tags --always --dirty`, as `script/docker` passes it
- `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from - `script/generate` (`make generate`) — regenerate `mfer/mf.pb.go` from
`mfer/mf.proto` and record the hash of that `mfer/mf.proto` in `mfer/mf.proto` and record the hash of that `mfer/mf.proto` in
`mfer/mf.proto.sha256`; the only thing that regenerates the committed `mfer/mf.proto.sha256`; the only thing that regenerates the committed
@@ -85,8 +88,8 @@ provide:
[its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4) [its release](https://github.com/protocolbuffers/protobuf/releases/tag/v33.4)
and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11 and put its `bin/protoc` on `PATH`) and `protoc-gen-go` v1.36.11
(`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which (`go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11`, which
installs it in `$(go env GOPATH)/bin`; `make generate` adds that directory to installs it in `$(go env GOPATH)/bin`; `script/generate` adds that directory
`PATH`) to `PATH`)
- `script/fuzz` — fuzz the manifest parser for one minute; run by hand - `script/fuzz` — fuzz the manifest parser for one minute; run by hand
(`make fuzz`), never by CI, while `script/test` runs its committed seed corpus (`make fuzz`), never by CI, while `script/test` runs its committed seed corpus
as ordinary tests as ordinary tests
@@ -98,8 +101,8 @@ provide:
- `script/gofumpt` — run `gofumpt` over every Go file in the repository in the - `script/gofumpt` — run `gofumpt` over every Go file in the repository in the
given mode, `--write` or `--check`, at the one version it pins (built on given mode, `--write` or `--check`, at the one version it pins (built on
demand by `go run`, so nothing installs it); `script/fmt`, `script/fmt-check` demand by `go run`, so nothing installs it); `script/fmt`, `script/fmt-check`
and the Docker lint stage (`make fmt-check-go`) all go through it, so they and the Docker lint stage all go through it, so they cannot disagree about Go
cannot disagree about Go formatting formatting
- `script/prettier` — run prettier over the repository's canonical file set - `script/prettier` — run prettier over the repository's canonical file set
(Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or (Markdown and JSON, minus `.prettierignore`) in the given mode, `--write` or
`--check`; the single definition of that file set, so `script/fmt` and `--check`; the single definition of that file set, so `script/fmt` and
@@ -265,9 +268,12 @@ are now tracked only in the [issues](https://git.eeqj.de/sneak/mfer/issues).
cryptographic integrity of downloaded files. A file already there with the cryptographic integrity of downloaded files. A file already there with the
size and hash the manifest lists is skipped. Once every file is in place, size and hash the manifest lists is skipped. Once every file is in place,
the manifest is saved there as `index.mf`, so `mfer check` can verify the the manifest is saved there as `index.mf`, so `mfer check` can verify the
tree later. A manifest that lists `index.mf` (in any letter case) or tree later. Each file is downloaded to a temp file beside it, such as
`.index.mf.tmp` at the top of the tree is refused before any file is `.a.txt.tmp` for `a.txt`, then moved into place. A manifest is refused
downloaded, since saving the manifest would replace it. before any file is downloaded if it lists a file where fetch writes
another: at the temp file of a listed file, or at `index.mf` or
`.index.mf.tmp` at the top of the tree. Names are compared in any letter
case.
- `mfer fetch --require-signature <fingerprint> https://example.com/stuff/` - `mfer fetch --require-signature <fingerprint> https://example.com/stuff/`
- as above, but first refuses a manifest not signed by the key with that - as above, but first refuses a manifest not signed by the key with that
fingerprint, as `mfer check --require-signature` does, before downloading fingerprint, as `mfer check --require-signature` does, before downloading
-12
View File
@@ -1,12 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
main() {
if [[ -n "${GITREV:-}" ]]; then
echo "$GITREV"
else
git describe --tags --always --dirty=-dirty
fi
}
main "$@"
+36 -18
View File
@@ -91,10 +91,10 @@ var (
// errHashMismatch indicates a downloaded file whose hash matches no // errHashMismatch indicates a downloaded file whose hash matches no
// manifest hash. // manifest hash.
errHashMismatch = errors.New("hash mismatch") errHashMismatch = errors.New("hash mismatch")
// errManifestNameListed indicates a manifest that lists a file where // errNameClash indicates a manifest that lists a file where fetch
// fetch saves the manifest. // writes another file.
errManifestNameListed = errors.New( errNameClash = errors.New(
"manifest lists a file where fetch saves the manifest") "manifest lists a file where fetch writes another file")
) )
// DownloadProgress reports the progress of a single file download. // DownloadProgress reports the progress of a single file download.
@@ -412,7 +412,7 @@ func (mfa *CLIApp) fetchManifestOperation(ctx *cli.Context) error {
// fetchManifest downloads the manifest at manifestURL and parses it, // fetchManifest downloads the manifest at manifestURL and parses it,
// enforcing --require-signature if it is given and refusing a manifest // enforcing --require-signature if it is given and refusing a manifest
// that lists a file where it will be saved. It returns the manifest as // that lists a file where fetch writes another. It returns the manifest as
// downloaded, to be saved once the files are in place, and the files it // downloaded, to be saved once the files are in place, and the files it
// lists. // lists.
func fetchManifest( func fetchManifest(
@@ -452,7 +452,7 @@ func fetchManifest(
files := manifest.Files() files := manifest.Files()
err = checkManifestNameUnlisted(files) err = checkNoNameClash(files)
if err != nil { if err != nil {
return nil, nil, err return nil, nil, err
} }
@@ -462,19 +462,37 @@ func fetchManifest(
return manifestData, files, nil return manifestData, files, nil
} }
// checkManifestNameUnlisted returns an error if files lists a file or // checkNoNameClash returns an error if files lists a file, or a directory
// directory at the top of the tree under the name fetch saves the // a file is in, under a name where fetch writes another file: the temp
// manifest as, or under that name's temp file. Saving the manifest would // file it downloads a listed file to, or, at the top of the tree, the
// replace or remove it, or fail once every file was downloaded, leaving a // saved manifest or its temp file. fetch would remove or replace what is
// tree check rejects. Names are compared ignoring case, since on a // listed there, or fail partway, leaving a tree check rejects. Names are
// case-insensitive filesystem INDEX.MF and index.mf are one file. // compared ignoring case, since on a case-insensitive filesystem INDEX.MF
func checkManifestNameUnlisted(files []*mfer.MFFilePath) error { // and index.mf are one file.
for _, f := range files { func checkNoNameClash(files []*mfer.MFFilePath) error {
top, _, _ := strings.Cut(filepath.Clean(f.GetPath()), string(filepath.Separator)) sep := string(filepath.Separator)
if strings.EqualFold(top, defaultManifestName) || // written maps each name fetch writes, other than the listed files
strings.EqualFold(top, tempPathFor(defaultManifestName)) { // themselves, in lower case, to the file it writes there.
return fmt.Errorf("%w: %s", errManifestNameListed, f.GetPath()) written := map[string]string{
defaultManifestName: "the saved manifest",
tempPathFor(defaultManifestName): "the saved manifest's temp file",
}
for _, f := range files {
tmpPath := tempPathFor(filepath.Clean(f.GetPath()))
written[strings.ToLower(tmpPath)] = "the temp file for " + f.GetPath()
}
for _, f := range files {
// Look up each directory on the file's path, then the file itself.
parts := strings.Split(strings.ToLower(filepath.Clean(f.GetPath())), sep)
for i := range parts {
what, ok := written[strings.Join(parts[:i+1], sep)]
if ok {
return fmt.Errorf("%w: %s (%s)", errNameClash, f.GetPath(), what)
}
} }
} }
+73 -12
View File
@@ -1176,23 +1176,84 @@ func TestFetchRefusesListedManifestName(t *testing.T) {
t.Run(listed, func(t *testing.T) { t.Run(listed, func(t *testing.T) {
t.Parallel() t.Parallel()
// Built directly rather than scanned, since a scan lists no files := map[string][]byte{listed: []byte("listed")}
// hidden files and never a path starting with "./".
content := []byte("listed")
builder := mfer.NewBuilder()
_, err := builder.AddFile(mfer.RelFilePath(listed), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
require.NoError(t, err)
var manifest bytes.Buffer assertFetchRefused(t, builtManifest(t, files), files,
require.NoError(t, builder.Build(context.Background(), &manifest)) "manifest lists a file where fetch writes another file: "+listed)
assertFetchRefused(t, manifest.Bytes(), map[string][]byte{listed: content},
"manifest lists a file where fetch saves the manifest: "+listed)
}) })
} }
} }
// TestFetchRefusesListedTempName fetches manifests that list a.txt and
// .a.txt.tmp, the temp file fetch downloads a.txt to, at the top of the
// tree and in a directory. Downloading a.txt would remove .a.txt.tmp, so
// fetch must refuse the manifest before it creates the destination or
// requests any file. A manifest that lists only one of the two is fetched
// in full.
func TestFetchRefusesListedTempName(t *testing.T) {
t.Parallel()
for _, dir := range []string{"", "sub/"} {
file, tmp := dir+"a.txt", dir+".a.txt.tmp"
both := map[string][]byte{
file: []byte("a file"),
tmp: []byte("a file at its temp name"),
}
t.Run(file+" and "+tmp, func(t *testing.T) {
t.Parallel()
assertFetchRefused(t, builtManifest(t, both), both,
"manifest lists a file where fetch writes another file: "+
tmp+" (the temp file for "+file+")")
})
for listed, content := range both {
t.Run("only "+listed, func(t *testing.T) {
t.Parallel()
files := map[string][]byte{listed: content}
manifest := builtManifest(t, files)
server := httptest.NewServer(fetchTestHandler(manifest, files))
defer server.Close()
dest := t.TempDir()
opts := testOpts([]string{
testApp, cmdFetch, "-q", "--" + flagDest, dest, server.URL,
}, afero.NewOsFs())
require.Equal(t, 0, runCLI(opts), testStderr(t, opts))
want := maps.Clone(files)
want[defaultManifestName] = manifest
assert.Equal(t, want, filesUnder(t, dest))
})
}
}
}
// builtManifest returns a manifest of files, built directly rather than
// scanned, since a scan lists no hidden files and never a path starting
// with "./".
func builtManifest(t *testing.T, files map[string][]byte) []byte {
t.Helper()
builder := mfer.NewBuilder()
for p, content := range files {
_, err := builder.AddFile(mfer.RelFilePath(p), mfer.FileSize(len(content)),
mfer.ModTime(time.Now()), bytes.NewReader(content), nil)
require.NoError(t, err)
}
var manifest bytes.Buffer
require.NoError(t, builder.Build(context.Background(), &manifest))
return manifest.Bytes()
}
// assertFetchRefused serves manifest, a manifest of files, and fetches it // assertFetchRefused serves manifest, a manifest of files, and fetches it
// with flags into a directory that does not exist yet. fetch must fail // with flags into a directory that does not exist yet. fetch must fail
// with message after requesting only the manifest, and must not create // with message after requesting only the manifest, and must not create
Executable
+20
View File
@@ -0,0 +1,20 @@
#!/bin/sh
# script/build: build the mfer binary into bin/mfer, stamped with the
# revision `mfer version` prints, derived as script/docker derives it.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
go build -tags urfave_cli_no_docs \
-ldflags "-X main.Gitrev=$version" \
-o bin/mfer ./cmd/mfer
}
main "$@"
+3
View File
@@ -40,6 +40,9 @@ sha256() {
main() { main() {
cd "$ROOT/mfer" cd "$ROOT/mfer"
# `go install` puts protoc-gen-go in $(go env GOPATH)/bin, which is
# often not on PATH.
PATH="$PATH:$(go env GOPATH)/bin"
require_version protoc "libprotoc $PROTOC_VERSION" require_version protoc "libprotoc $PROTOC_VERSION"
require_version protoc-gen-go "protoc-gen-go $PROTOC_GEN_GO_VERSION" require_version protoc-gen-go "protoc-gen-go $PROTOC_GEN_GO_VERSION"
# Hashed before regenerating, so a missing hash tool stops the script # Hashed before regenerating, so a missing hash tool stops the script
+2 -2
View File
@@ -4,8 +4,8 @@
# Takes exactly one mode argument, --write or --check, and runs the same # Takes exactly one mode argument, --write or --check, and runs the same
# gofumpt version over the same files in both modes. script/fmt and # gofumpt version over the same files in both modes. script/fmt and
# script/fmt-check both go through here, and so does the Docker lint # script/fmt-check both go through here, and so does the Docker lint
# stage (make fmt-check-go), so what gets formatted and what gets # stage, so what gets formatted and what gets verified cannot drift
# verified cannot drift apart. # apart.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"