Add script/loadtest to measure throughput, latency and memory (closes #81)

script/loadtest [duration [clients]], or make loadtest, is a benchmark
that script/check does not run. It builds the image and vegeta from a
pinned commit, then for each scenario starts a new pixad container and
a new cmd/loadtest-origin container, an upstream host that answers every
path with one generated JPEG: a cached image (hit), a new source image
per request (miss), and each new source image asked for by all clients
at once (herd). It prints vegeta's report, pixad's peak resident memory
and the requests the origin got. The containers share a network in
203.0.113.0/24, as pixad refuses private and local upstream addresses.
README.md says how to run and read it; TODO.md records a small baseline.

Model: opus-5-5
This commit is contained in:
2026-10-04 22:22:57 +00:00
parent 0a78165b67
commit 3969146a7f
5 changed files with 328 additions and 6 deletions
+59 -4
View File
@@ -146,10 +146,11 @@ name.
Multiple source paths may reference the same content blob; the
database tracks references rather than using filesystem refcounting.
Toward a target of 1-5k r/s, pixa keeps in memory the content types of
the 10,000 transformed images most recently cached or served, so a
cache hit on one of them reads only the image file from disk and not
the metadata file stored beside it.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, pixa keeps in memory the content types of the 10,000
transformed images most recently cached or served, so a cache hit on one of them
reads only the image file from disk and not the metadata file stored beside it.
### Routes
@@ -576,6 +577,8 @@ them. We provide:
- `script/check` — run test, lint, and fmt-check
- `script/docker` — build the Docker image tagged via `script/projectname`
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
- `script/loadtest` — measure pixad's throughput, latency and peak memory; a
benchmark, not part of `script/check` (see Load Test)
- `script/cibuild` — CI entrypoint: `docker build .` with a new
`CHECK_EPOCH` on every run, so the Dockerfile's checks run instead of
coming from the build cache, and a green run implies a green repo
@@ -584,6 +587,58 @@ them. We provide:
- `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`
## Load Test
`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how
much memory it uses. It is a benchmark, not a check: `script/check` does not run
it. It needs Docker and Go.
```bash
script/loadtest # 10 seconds per scenario, 4 clients
script/loadtest 30s 32 # 30 seconds per scenario, 32 clients
```
It builds the image with `script/docker` and the load tool,
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
starts a new pixad container and a new origin container, `cmd/loadtest-origin`:
an upstream host that answers every path with the same generated 1600x1200 JPEG.
vegeta then sends requests from the given number of clients, each sending its
next request as soon as its last one is answered, all for an image resized to
400x300 WebP:
- `hit`: the same image every time, put in the cache first;
- `miss`: a new source image every time, so pixad fetches and converts each one;
- `herd`: each new source image once per client in a row, so that all clients
ask for it at the same time and share one fetch and one conversion (see
Routes).
pixad refuses upstream hosts with private or local addresses, so the containers
share a Docker network in `203.0.113.0/24`, a range set aside for documentation.
A second run on the same Docker host while one is going fails, as it cannot
create that network.
For each scenario the script prints vegeta's report and two lines of its own:
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
per second, and how many were answered successfully per second; the last is
the number to compare with the target under Storage;
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
to the end of its answer; `50`, `95` and `99` are the 50th, 95th and 99th
percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other
numbers are not for the scenario described, such as `503` when pixad was busy;
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since
its container started, in kB; for `hit` it includes the request that put the
image in the cache;
- `requests to the origin`: the fetches pixad made: one for `hit`, one per
request for `miss`, and one per image for `herd`, that is the requests sent
divided by the number of clients.
The numbers depend on the machine and on whatever else runs on it. The first
measurement, made on a shared machine with few clients, is in `TODO.md`; it says
nothing about the target.
## TODO
See [TODO.md](TODO.md) for the full prioritized task list.