Nothing limits how much fetch and check read from a server for a manifest or a file #168

New Issue

2026-10-07T10:38:41+02:00

clawbot commented

2026-10-07 10:38:41 +02:00

Problem

fetch reads the whole manifest into memory with io.ReadAll (fetchManifest in internal/cli/fetch.go). check <URL> copies it to a temp file, and mfer.NewManifestFromReader then reads all of it with io.ReadAll (mfer/deserialize.go). A server can send gigabytes within the request time limit (10 minutes by default for fetch, 30 seconds for check) and use up memory or disk, although a valid manifest can never be much larger than its 256 MB decompressed limit.
saveResponse in internal/cli/fetch.go copies a file's body with no limit, so a server sending more than the listed size fills the disk until the request times out; the size is checked only afterwards.

Definition of done

The library refuses a manifest larger than a stated maximum without reading past it. The maximum follows from MaxDecompressedSize (zstd's worst-case growth of the inner message, plus room for the signature and the key), is an exported constant next to MaxDecompressedSize, and the Compression section of docs/FORMAT.md states it.
fetch, and check given a URL, stop reading the manifest once it passes that maximum, with an error that names the limit.
fetch stops reading a file one byte past its listed size and reports the size mismatch, removing the temp file.
Tests: a test server sending a manifest body that never ends, and one sending a file body longer than listed; both fail promptly with the size error and leave no temp file.
make check passes.

Model: opus-5-5

## Problem - `fetch` reads the whole manifest into memory with `io.ReadAll` (`fetchManifest` in `internal/cli/fetch.go`). `check <URL>` copies it to a temp file, and `mfer.NewManifestFromReader` then reads all of it with `io.ReadAll` (`mfer/deserialize.go`). A server can send gigabytes within the request time limit (10 minutes by default for `fetch`, 30 seconds for `check`) and use up memory or disk, although a valid manifest can never be much larger than its 256 MB decompressed limit. - `saveResponse` in `internal/cli/fetch.go` copies a file's body with no limit, so a server sending more than the listed size fills the disk until the request times out; the size is checked only afterwards. ## Definition of done - The library refuses a manifest larger than a stated maximum without reading past it. The maximum follows from `MaxDecompressedSize` (zstd's worst-case growth of the inner message, plus room for the signature and the key), is an exported constant next to `MaxDecompressedSize`, and the Compression section of `docs/FORMAT.md` states it. - `fetch`, and `check` given a URL, stop reading the manifest once it passes that maximum, with an error that names the limit. - `fetch` stops reading a file one byte past its listed size and reports the size mismatch, removing the temp file. - Tests: a test server sending a manifest body that never ends, and one sending a file body longer than listed; both fail promptly with the size error and leave no temp file. - `make check` passes. Model: opus-5-5

clawbot referenced this issue

2026-10-07 10:38:50 +02:00

Review toward 1.0: identify and file all next steps #99

clawbot referenced this issue from a commit

2026-10-07 11:50:01 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168)

clawbot referenced this issue from a commit

2026-10-07 11:59:52 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168)

clawbot referenced a pull request that will close this issue

2026-10-07 12:22:40 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168) #172

clawbot commented

2026-10-07 12:22:49 +02:00

Built in #172. The library refuses a manifest larger than MaxManifestSize (258 MiB), fetch and check stop downloading a manifest at that point, and fetch stops reading a file one byte past its listed size. The new manifest test streams the full 258 MiB, which makes the race-enabled test run noticeably heavier; the PR says by how much.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/mfer/pulls/172. The library refuses a manifest larger than `MaxManifestSize` (258 MiB), `fetch` and `check` stop downloading a manifest at that point, and `fetch` stops reading a file one byte past its listed size. The new manifest test streams the full 258 MiB, which makes the race-enabled test run noticeably heavier; the PR says by how much. Model: opus-5-5

clawbot referenced this issue from a commit

2026-10-07 13:55:31 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168)

clawbot referenced this issue from a commit

2026-10-07 14:07:41 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168)

clawbot closed this issue

2026-10-07 14:25:46 +02:00

clawbot referenced this issue from a commit

2026-10-07 14:25:47 +02:00

Limit how much fetch and check read for a manifest or a file (closes #168)

Sign in to join this conversation.