No load test verifying the documented 1k-5k req/s target #81

New Issue

clawbot · 2026-08-09T03:49:02+02:00

clawbot commented

2026-08-09 03:49:02 +02:00

Verified against main at 61f42e6. This is the "load tests to verify the 1k to 5k req/s target" item in TODO.md; confirmed missing — nothing in the repo measures throughput.

README.md:49 states the design targets 1-5k req/s. That number is currently an aspiration with no measurement behind it, and the survey found several reasons to doubt it as things stand:

every cache hit performs open + stat + ReadFile + JSON parse, because the in-memory meta cache is dead code (filed separately);
there is no request coalescing, so concurrent misses multiply upstream fetches and transcodes (filed separately);
there is no global concurrency cap, so load is bounded by memory exhaustion rather than by a limiter (filed separately).

A load test is what turns those three from plausible arguments into measured facts, and tells us whether the README's number is honest before it ships as 1.0.

Definition of done

A repeatable load-test harness under script/ (following Scripts to Rule Them All, so it is script/loadtest or similar), not committed as a one-off snippet.
Measures at minimum: hit-path throughput and latency percentiles against a warm cache; miss-path throughput against a local origin; and behavior under concurrent misses for the same key (the thundering-herd case).
Reports peak RSS alongside throughput — for this service memory is the binding constraint, and a throughput number without it is misleading.
Explicitly not part of make check (it is a benchmark, not a gate) and documented as such, with instructions in the README on how to run it and how to read the output.
The measured result is recorded — either in the README next to the claim, or in TODO.md. If the number does not hold, the README claim is corrected rather than left standing.

Sequencing

Worth running once early, as a baseline, even before the performance issues above are fixed — the baseline is what proves those fixes did anything. Run it again after each, and record the deltas.

Verified against `main` at `61f42e6`. This is the "load tests to verify the 1k to 5k req/s target" item in `TODO.md`; confirmed missing — nothing in the repo measures throughput. `README.md:49` states the design targets 1-5k req/s. That number is currently an aspiration with no measurement behind it, and the survey found several reasons to doubt it as things stand: - every cache hit performs `open` + `stat` + `ReadFile` + JSON parse, because the in-memory meta cache is dead code (filed separately); - there is no request coalescing, so concurrent misses multiply upstream fetches and transcodes (filed separately); - there is no global concurrency cap, so load is bounded by memory exhaustion rather than by a limiter (filed separately). A load test is what turns those three from plausible arguments into measured facts, and tells us whether the README's number is honest before it ships as 1.0. ## Definition of done 1. A repeatable load-test harness under `script/` (following Scripts to Rule Them All, so it is `script/loadtest` or similar), not committed as a one-off snippet. 2. Measures at minimum: hit-path throughput and latency percentiles against a warm cache; miss-path throughput against a local origin; and behavior under concurrent misses for the same key (the thundering-herd case). 3. Reports peak RSS alongside throughput — for this service memory is the binding constraint, and a throughput number without it is misleading. 4. Explicitly **not** part of `make check` (it is a benchmark, not a gate) and documented as such, with instructions in the README on how to run it and how to read the output. 5. The measured result is recorded — either in the README next to the claim, or in `TODO.md`. If the number does not hold, the README claim is corrected rather than left standing. ## Sequencing Worth running once early, as a baseline, even before the performance issues above are fixed — the baseline is what proves those fixes did anything. Run it again after each, and record the deltas.

clawbot added this to the 1.0.0 milestone 2026-08-09 03:49:02 +02:00

clawbot referenced this issue

2026-08-09 16:10:39 +02:00

Tracking: 1.0.0 sequencing and critical path #103

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/pixa#81