Add script/loadtest to measure throughput, latency and memory (closes #81)
check / check (push) Failing after 2s
check / check (push) Failing after 2s
script/loadtest [duration [clients]] (make loadtest, defaults 10s and 4) measures pixad in three scenarios: hit (one cached image), miss (a new source image every request) and herd (each new source image asked for by all clients at once). For each it prints vegeta's report (requests per second, p50/p95/p99 latency, status codes), pixad's peak resident memory and how many requests reached the origin. It is a benchmark, not run by script/check. The origin it uses, internal/loadtestorigin behind cmd/loadtest-origin, answers every path with one generated JPEG. Bad arguments are refused before the build. README.md says how to run and read it and keeps 1-5k r/s as a target not yet measured at scale. Model: opus-5-5
This commit was merged in pull request #209.
This commit is contained in:
@@ -146,10 +146,11 @@ name.
|
||||
|
||||
Multiple source paths may reference the same content blob; the
|
||||
database tracks references rather than using filesystem refcounting.
|
||||
Toward a target of 1-5k r/s, pixa keeps in memory the content types of
|
||||
the 10,000 transformed images most recently cached or served, so a
|
||||
cache hit on one of them reads only the image file from disk and not
|
||||
the metadata file stored beside it.
|
||||
|
||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||
transformed images most recently cached or served, so a cache hit on one of them
|
||||
reads only the image file from disk and not the metadata file stored beside it.
|
||||
|
||||
### Routes
|
||||
|
||||
@@ -576,6 +577,8 @@ them. We provide:
|
||||
- `script/check` — run test, lint, and fmt-check
|
||||
- `script/docker` — build the Docker image tagged via `script/projectname`
|
||||
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
|
||||
- `script/loadtest` — measure pixad's throughput, latency and peak memory; a
|
||||
benchmark, not part of `script/check` (see Load Test)
|
||||
- `script/cibuild` — CI entrypoint: `docker build .` with a new
|
||||
`CHECK_EPOCH` on every run, so the Dockerfile's checks run instead of
|
||||
coming from the build cache, and a green run implies a green repo
|
||||
@@ -584,6 +587,58 @@ them. We provide:
|
||||
- `script/install-precommit` — install the git pre-commit hook that
|
||||
runs `script/precommit`
|
||||
|
||||
## Load Test
|
||||
|
||||
`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how
|
||||
much memory it uses. It is a benchmark, not a check: `script/check` does not run
|
||||
it. It needs Docker and Go.
|
||||
|
||||
```bash
|
||||
script/loadtest # 10 seconds per scenario, 4 clients
|
||||
script/loadtest 30s 32 # 30 seconds per scenario, 32 clients
|
||||
```
|
||||
|
||||
It builds the image with `script/docker` and the load tool,
|
||||
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
|
||||
starts a new pixad container and a new origin container, `cmd/loadtest-origin`:
|
||||
an upstream host that answers every path with the same generated 1600x1200 JPEG.
|
||||
vegeta then sends requests from the given number of clients, each sending its
|
||||
next request as soon as its last one is answered, all for an image resized to
|
||||
400x300 WebP:
|
||||
|
||||
- `hit`: the same image every time, put in the cache first;
|
||||
- `miss`: a new source image every time, so pixad fetches and converts each one;
|
||||
- `herd`: each new source image once per client in a row, so that all clients
|
||||
ask for it at the same time and share one fetch and one conversion (see
|
||||
Routes).
|
||||
|
||||
pixad refuses upstream hosts with private or local addresses, so the containers
|
||||
share a Docker network in `203.0.113.0/24`, a range set aside for documentation.
|
||||
A second run on the same Docker host while one is going fails, as it cannot
|
||||
create that network.
|
||||
|
||||
For each scenario the script prints vegeta's report and two lines of its own:
|
||||
|
||||
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
|
||||
per second, and how many were answered successfully per second; the last is
|
||||
the number to compare with the target under Storage;
|
||||
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
|
||||
to the end of its answer; `50`, `95` and `99` are the 50th, 95th and 99th
|
||||
percentiles;
|
||||
- `Status Codes` and `Error Set`: anything other than `200` means the other
|
||||
numbers are not for the scenario described, such as `503` when pixad was busy;
|
||||
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
|
||||
- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since
|
||||
its container started, in kB; for `hit` it includes the request that put the
|
||||
image in the cache;
|
||||
- `requests to the origin`: the fetches pixad made: one for `hit`, one per
|
||||
request for `miss`, and one per image for `herd`, that is the requests sent
|
||||
divided by the number of clients.
|
||||
|
||||
The numbers depend on the machine and on whatever else runs on it. The first
|
||||
measurement, made on a shared machine with few clients, is in `TODO.md`; it says
|
||||
nothing about the target.
|
||||
|
||||
## TODO
|
||||
|
||||
See [TODO.md](TODO.md) for the full prioritized task list.
|
||||
|
||||
Reference in New Issue
Block a user