sneak/pixa
1
0
Fork 0
Code Issues 11 Pull Requests 4 Actions 11 Packages Projects Releases Wiki Activity

No load test verifying the documented 1k-5k req/s target #81

New Issue
Closed
opened 2026-08-09 03:49:02 +02:00 by clawbot · 2 comments
clawbot commented 2026-08-09 03:49:02 +02:00
Collaborator
Copy Link
Copy Source

Verified against main at 61f42e6. This is the "load tests to verify the 1k to 5k req/s target" item in TODO.md; confirmed missing — nothing in the repo measures throughput.

README.md:49 states the design targets 1-5k req/s. That number is currently an aspiration with no measurement behind it, and the survey found several reasons to doubt it as things stand:

  • every cache hit performs open + stat + ReadFile + JSON parse, because the in-memory meta cache is dead code (filed separately);
  • there is no request coalescing, so concurrent misses multiply upstream fetches and transcodes (filed separately);
  • there is no global concurrency cap, so load is bounded by memory exhaustion rather than by a limiter (filed separately).

A load test is what turns those three from plausible arguments into measured facts, and tells us whether the README's number is honest before it ships as 1.0.

Definition of done

  1. A repeatable load-test harness under script/ (following Scripts to Rule Them All, so it is script/loadtest or similar), not committed as a one-off snippet.
  2. Measures at minimum: hit-path throughput and latency percentiles against a warm cache; miss-path throughput against a local origin; and behavior under concurrent misses for the same key (the thundering-herd case).
  3. Reports peak RSS alongside throughput — for this service memory is the binding constraint, and a throughput number without it is misleading.
  4. Explicitly not part of make check (it is a benchmark, not a gate) and documented as such, with instructions in the README on how to run it and how to read the output.
  5. The measured result is recorded — either in the README next to the claim, or in TODO.md. If the number does not hold, the README claim is corrected rather than left standing.

Sequencing

Worth running once early, as a baseline, even before the performance issues above are fixed — the baseline is what proves those fixes did anything. Run it again after each, and record the deltas.

Verified against `main` at `61f42e6`. This is the "load tests to verify the 1k to 5k req/s target" item in `TODO.md`; confirmed missing — nothing in the repo measures throughput. `README.md:49` states the design targets 1-5k req/s. That number is currently an aspiration with no measurement behind it, and the survey found several reasons to doubt it as things stand: - every cache hit performs `open` + `stat` + `ReadFile` + JSON parse, because the in-memory meta cache is dead code (filed separately); - there is no request coalescing, so concurrent misses multiply upstream fetches and transcodes (filed separately); - there is no global concurrency cap, so load is bounded by memory exhaustion rather than by a limiter (filed separately). A load test is what turns those three from plausible arguments into measured facts, and tells us whether the README's number is honest before it ships as 1.0. ## Definition of done 1. A repeatable load-test harness under `script/` (following Scripts to Rule Them All, so it is `script/loadtest` or similar), not committed as a one-off snippet. 2. Measures at minimum: hit-path throughput and latency percentiles against a warm cache; miss-path throughput against a local origin; and behavior under concurrent misses for the same key (the thundering-herd case). 3. Reports peak RSS alongside throughput — for this service memory is the binding constraint, and a throughput number without it is misleading. 4. Explicitly **not** part of `make check` (it is a benchmark, not a gate) and documented as such, with instructions in the README on how to run it and how to read the output. 5. The measured result is recorded — either in the README next to the claim, or in `TODO.md`. If the number does not hold, the README claim is corrected rather than left standing. ## Sequencing Worth running once early, as a baseline, even before the performance issues above are fixed — the baseline is what proves those fixes did anything. Run it again after each, and record the deltas.
clawbot added this to the 1.0.0 milestone 2026-08-09 03:49:02 +02:00
clawbot commented 2026-10-04 22:25:39 +02:00
Author
Collaborator
Copy Link
Copy Source

Plan, checked against next at 8314099. Two of the three doubts the issue lists are gone (content types are kept in memory since #70; concurrent misses share one fetch since #65; processing and fetches are bounded since #64), but nothing measures throughput yet.

  • script/loadtest, a Scripts to Rule Them All entrypoint, not part of make check. It builds and starts pixad against a temporary state directory and a small local origin (a tiny Go program or the test fixtures served by one), drives it with a widely used load tool pinned by hash (for example vegeta, installed with go install at a pinned commit as sneak/prompts now asks for host Go tools), and reports, for each scenario, requests per second, latency percentiles (p50, p95, p99) and pixad's peak resident memory: warm-cache hits, misses against the local origin, and many concurrent misses for one image.
  • Its duration and concurrency are arguments with small defaults. This host is shared and must not be loaded hard: verify the script with short, low-concurrency runs only, and record that small baseline in TODO.md as what it is (shared host, low concurrency), not as the answer to the 1-5k claim.
  • README.md says how to run it and how to read its output, and keeps "toward a target of 1-5k r/s" worded as a target that has not been measured at scale, with a pointer to the script.

Model: opus-5-5

Plan, checked against `next` at `8314099`. Two of the three doubts the issue lists are gone (content types are kept in memory since https://git.eeqj.de/sneak/pixa/issues/70; concurrent misses share one fetch since https://git.eeqj.de/sneak/pixa/issues/65; processing and fetches are bounded since https://git.eeqj.de/sneak/pixa/issues/64), but nothing measures throughput yet. - `script/loadtest`, a Scripts to Rule Them All entrypoint, not part of `make check`. It builds and starts `pixad` against a temporary state directory and a small local origin (a tiny Go program or the test fixtures served by one), drives it with a widely used load tool pinned by hash (for example `vegeta`, installed with `go install` at a pinned commit as `sneak/prompts` now asks for host Go tools), and reports, for each scenario, requests per second, latency percentiles (p50, p95, p99) and `pixad`'s peak resident memory: warm-cache hits, misses against the local origin, and many concurrent misses for one image. - Its duration and concurrency are arguments with small defaults. This host is shared and must not be loaded hard: verify the script with short, low-concurrency runs only, and record that small baseline in `TODO.md` as what it is (shared host, low concurrency), not as the answer to the 1-5k claim. - `README.md` says how to run it and how to read its output, and keeps "toward a target of 1-5k r/s" worded as a target that has not been measured at scale, with a pointer to the script. Model: opus-5-5
clawbot commented 2026-10-04 23:07:04 +02:00
Author
Collaborator
Copy Link
Copy Source

Built in #209: script/loadtest (make loadtest) runs the hit, miss and herd scenarios against a new pixad and a small upstream host, cmd/loadtest-origin, and prints throughput, latency percentiles, pixad's peak resident memory and the requests that reached the upstream host. README.md says how to run it and read its output and keeps 1-5k r/s as a target not yet measured; TODO.md records a small baseline from this shared machine, which says nothing about the target.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/pixa/pulls/209: `script/loadtest` (`make loadtest`) runs the `hit`, `miss` and `herd` scenarios against a new pixad and a small upstream host, `cmd/loadtest-origin`, and prints throughput, latency percentiles, pixad's peak resident memory and the requests that reached the upstream host. `README.md` says how to run it and read its output and keeps 1-5k r/s as a target not yet measured; `TODO.md` records a small baseline from this shared machine, which says nothing about the target. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
needs-checks
needs-rebase
needs-review
needs-rework
No labels
Milestone
No items
No Milestone 1.0.0
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/pixa#81