1 Commits
Author SHA1 Message Date
clawbot 44f3192a03 Rewrite README.md sentence by sentence (closes #327)
check / check (push) Canceled after 0s
Rewrites README.md from scratch against the code on next, for
#327: what pixa is and how to use
it, deployment, design, the settings reference, then development and
testing. The `### Configuration` heading and its one-item-per-setting
list stay as TestReadmeNamesEverySetting reads them.

Rework after review: all 38 findings of
#345 (comment) are
addressed, the decode and encode times are restored, and 13 further
sentences were fixed in a full pass.

Model: fable-5-1
2026-10-27 02:29:34 +00:00
+278 -211
View File
@@ -30,11 +30,12 @@ make docker
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
```
`make build` compiles pixa on your machine. That needs the libraries
`script/bootstrap --cgo` installs (see Deployment).
`make build` compiles pixa on your machine into `bin/pixad`. `pixad` is the
server program. Compiling needs the libraries `script/bootstrap --cgo` installs
(see Deployment).
The signing key is pixa's one secret. It must be 32 characters or longer. The
placeholder in the example file is refused at startup.
The signing key is pixa's one required secret. It must be 32 characters or
longer. The placeholder in the example file is refused at startup.
Once pixa runs, it serves images through URLs of this form:
@@ -69,8 +70,6 @@ variable wins over the same setting in the file.
pixa listens on plain HTTP. It runs behind a reverse proxy: a server in front of
it that receives the public requests and passes them on. The reverse proxy
terminates TLS, the encryption behind `https`.
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy gets its
own TLS certificate and does everything in this list without further settings.
The reverse proxy must:
@@ -80,12 +79,16 @@ The reverse proxy must:
form from its pages unless `Origin` or `Referer` names the host in `Host`. It
builds encrypted URLs from `Host`. It checks `Referer` against
`referer_blocklist`.
- set `X-Forwarded-For` to the client's address, with `trusted_proxies` naming
the address its requests reach pixa from. The login limit and
- set `X-Forwarded-For` to the client's address. `trusted_proxies` must hold the
proxy's address as pixa sees it. The login limit and
`image_requests_per_minute` then count each client by its own address.
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
is the longest pixa takes to fetch, convert and send an image.
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy does
everything in this list without further settings, and gets its own TLS
certificate.
The reverse proxy may also refuse `/metrics`, as the example does. Only a
scraper that reaches pixa directly can then read it. pixa itself asks for
`metrics.username` and `metrics.password` there.
@@ -114,7 +117,7 @@ database and the disk cache:
the limit for a while. Leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
each start from the free space on the volume (see Configuration). Anything
else that fills or frees space on the volume moves it.
else that fills or frees space on the volume moves that default.
A load balancer's health check can request `/.well-known/healthcheck.json`. It
answers 200 whenever pixa is running, in maintenance mode too (see
@@ -122,28 +125,28 @@ answers 200 whenever pixa is running, in maintenance mode too (see
On `SIGTERM` or `SIGINT`, pixa stops accepting connections. The requests in
progress and the images being processed get 5 seconds to finish. A request not
finished by then is cut off. pixa then writes the last-used times of cached
images that still wait to be written (see Storage), and exits. It exits with
status 1 when:
finished by then is cut off. pixa then saves the last-used times it has not yet
written (see Storage), and exits. It exits with status 1 when:
- images were still being processed after those 5 seconds;
- some of those times could not be written;
- some of those times could not be saved;
- another part of pixa failed to stop.
Otherwise it exits with 0. `docker stop` waits 10 seconds before it kills the
container.
Outside Docker, pixa needs libvips and libheif at run time. libvips is the image
library pixa converts images with. pixa does not start unless libvips has its
JPEG XL support. On Alpine that is the `vips-jxl` package. The nix and brew
packages of libvips include it. So do the apt packages from Debian 12 and Ubuntu
24.04 on. Compiling pixa also needs the development files of libvips and
libheif, `pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of
these. The `Dockerfile` installs its own packages. Plain `script/bootstrap`
installs git, make and Go; `script/setup` and `script/cibuild` run it. It also
installs Node, Yarn and prettier, which format the markdown. It installs none of
the C libraries: the checks compile pixa in Docker. Docker itself must already
be installed.
Outside Docker, pixa needs these at run time:
- libvips, the image library pixa converts images with, including its JPEG XL
support. pixa does not start without that support. On Alpine it is the
`vips-jxl` package. The nix and brew packages of libvips include it. So do the
apt packages from Debian 12 and Ubuntu 24.04 on.
- libheif, which libvips needs for AVIF.
- the system's CA certificates, which HTTPS fetches need.
Compiling pixa needs the development files of libvips and libheif, `pkg-config`
and a C compiler. `script/bootstrap --cgo` installs them, and libvips and
libheif with them. The `Dockerfile` installs its own packages.
## Running under upaas
@@ -161,19 +164,20 @@ run under. Its app for pixa needs:
comma-separated; default none. Every image on a listed host is served to
anyone, so list only hosts you control, where nobody else can put images.
- `PIXA_CACHE_MAX_BYTES`: the disk cache limit in bytes; `0` turns the disk
cache off. Set it. The volume is a directory on the host's filesystem,
shared with the host and every other app. The default is 75% of the sum of
that filesystem's free space and what the cache holds, and at least 500
MiB. So the cache can grow to three quarters or more of the space they all
have left, and to all of it when that is 500 MiB or less.
cache off. Set it. The volume is a directory on the host's filesystem. The
host and every other app use that filesystem too. The default is 75% of
the sum of that filesystem's free space and what the cache already holds.
So by default the cache may grow to three quarters of the space the host
and the other apps have left. The default is also at least 500 MiB. So
when that space is 500 MiB or less, the cache may take all of it.
- the rest are in the list under Configuration.
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once. On top of that, the fetched source images
waiting for processing can take `upstream_connections` times
`upstream_max_response_size`. 4.2 GiB is a measured value. It is the most
resident memory grew while pixa processed one 8192x8192 source, the largest it
accepts, saving it at full size to each output format in turn. Memory grew
most for a 16-bit JPEG XL of random pixels saved as WebP. At the defaults that
`upstream_max_response_size`. 4.2 GiB is a measured value. It is the largest
rise in resident memory measured while pixa saved one 8192x8192 source, the
largest it accepts, to each output format at full size. The rise was largest
for a 16-bit JPEG XL of random pixels saved as WebP. At the defaults the total
is half the CPUs pixa sees times 4.2 GiB, plus 64 times 50 MiB. On a host
where pixa sees 48 CPUs, that is 24 times 4.2 GiB plus 3.1 GiB, about 104 GiB.
Set `PIXA_MAX_CONCURRENT_PROCESSING` and `PIXA_UPSTREAM_CONNECTIONS` so that
@@ -197,17 +201,18 @@ run under. Its app for pixa needs:
- **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
after a deploy and marks the deploy failed unless it is `healthy`. The probe
takes the port from `PORT` (default `8080`). A port changed only in a mounted
config file is not seen by it, so change the port with `PORT`.
takes the port from `PORT` (default `8080`). It does not see a port changed
only in a mounted config file, so change the port with `PORT`.
## Rationale
A web application that shows images needs each one at several sizes and in
several formats. pixa makes them on request from one original, and keeps what it
made. The application stores one image per picture and hands out URLs. pixa is
one binary. At run time it needs only the image libraries named under
Deployment. Signed URLs with an expiry keep strangers from using it to resize
anything they like. Allowlisted hosts give open access where that is wanted.
one binary. At run time it needs only the image libraries and the CA
certificates named under Deployment. Signed URLs with an expiry keep strangers
from using it to resize anything they like. Allowlisted hosts give open access
where that is wanted.
## Design
@@ -249,18 +254,15 @@ These terms are the names the code uses:
### Routes
pixa answers these routes. Any other path answers 404. A path in this list asked
with a method the list does not give answers 405. A browser's CORS preflight
request to any path under `/v1/` answers 200, in maintenance mode too. CORS is
the browser rule that lets a page on one site read responses from another. The
preflight is the `OPTIONS` request the browser sends first.
pixa answers these routes. Any other path answers 404. A listed path answers 405
to any method the list does not give for it.
- `GET /` — the login page, or the generator page with a login session (see
Encrypted URLs). Needs nothing. Answers 200.
- `POST /` — log in with the signing key typed into the login page. Needs the
login page's form (below). Answers 303 to `/`, with a login session cookie,
for the right key. Answers 200 with the login page and an error for a wrong
key. Answers 429 over the login limit (see Limits).
login page's form (below). For the right key it answers 303 to `/` and sets
the login session cookie. For a wrong key it answers 200 with the login page
and an error. Answers 429 over the login limit (see Limits).
- `POST /generate` — make an encrypted URL from the generator page's form. Needs
a login session and the generator page's form. Without a login session it
answers 303 to `/`. Answers 200 with the page showing the URL. Answers 400
@@ -342,28 +344,36 @@ preflight is the `OPTIONS` request the browser sends first.
Answers 401 without them. Answers 404 when they are not set, as the route then
does not exist.
A browser's CORS preflight request to any path under `/v1/` answers 200, in
maintenance mode too. CORS is the browser rule that lets a page on one site read
responses from another. The preflight is the `OPTIONS` request the browser sends
first.
Every response carries an `X-Request-Id` header with the request's ID. A client
can quote it when reporting a problem. The ID is the request's own
`X-Request-Id` header, which a reverse proxy may send, when that is 1 to 64
letters, digits, `-`, `_` or `.`. Otherwise it is a random one pixa makes, which
tells nothing about the machine or the other requests. pixa's log line for the
request carries the same ID as `request_id`. So do the lines it logs when it
fetches, converts and serves an image. The fetch sends it to the upstream host
as `X-Request-Id`.
can quote it when reporting a problem. A reverse proxy may send an
`X-Request-Id` header with the request. pixa keeps that ID when it is 1 to 64
letters, digits, `-`, `_` or `.`. Otherwise pixa makes a random ID, which tells
nothing about the machine or the other requests. pixa's log line for the request
carries the same ID as `request_id`. So do the lines it logs when it fetches,
converts and serves an image. The fetch sends it to the upstream host as
`X-Request-Id`.
The `POST` routes accept only a form that pixa's own page served. The page puts
a token in the form and sets a cookie to match. A request without both is
refused with 403. So another site cannot submit the form from a visitor's
browser. The login and generator pages are meant to be opened over HTTPS. While
`debug` is off, a form sent from a page opened over plain HTTP is refused
with 403. While `debug` is on, a form sent from a page opened over HTTPS is
refused instead. Plain HTTP is for development on the browser's own machine. The
login session cookie is always marked `Secure`. Over plain HTTP a browser keeps
such a cookie only for its own machine (`localhost`), if at all. A form is also
refused with 403 when the page's host is not the `Host` header pixa receives. So
a reverse proxy in front of pixa must pass that header on unchanged. A form body
over 1 MiB is refused with 413. The image routes answer the errors listed for
them with a JSON body holding `error`, `status` and `timestamp`.
browser. A form is also refused with 403 when the page's host is not the `Host`
header pixa receives. So a reverse proxy in front of pixa must pass that header
on unchanged. A form body over 1 MiB is refused with 413.
The login and generator pages are meant to be opened over HTTPS. While `debug`
is off, a form sent from a page opened over plain HTTP is refused with 403.
While `debug` is on, a form sent from a page opened over HTTPS is refused
instead. Plain HTTP is for development on the browser's own machine. The login
session cookie is always marked `Secure`. Over plain HTTP a browser keeps such a
cookie only for its own machine (`localhost`), if at all.
The image routes answer the errors listed for them with a JSON body holding
`error`, `status` and `timestamp`.
### Image URLs
@@ -462,11 +472,11 @@ it would come out above 8192, the request is refused with 400. The fit modes:
at `100x0` is 100x1 by its proportions, but comes out at 8192x1. A side that
would come out above 8192 is refused with 400.
An AVIF image of more than 16,777,216 pixels is refused with 400 before anything
is decoded. That is the number of pixels in 4096x4096 or 8192x2048. A larger
AVIF encode could take more than 40 seconds (see Limits). The limit includes
`orig` of an AVIF source, and `auto` when it chooses AVIF. So a client whose
`Accept` names `image/avif` but not `image/jxl` is refused such a size.
An AVIF output of more than 16,777,216 pixels is refused with 400 before
anything is decoded. That is the number of pixels in 4096x4096 or 8192x2048. A
larger AVIF encode could take more than 40 seconds (see Limits). The limit
includes `orig` of an AVIF source, and `auto` when it chooses AVIF. So a client
whose `Accept` names `image/avif` but not `image/jxl` is refused such a size.
**Source formats.** The source may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The
upstream's `Content-Type` must name its format, and the image's first bytes must
@@ -485,8 +495,9 @@ from its `Accept` header, in this order:
1. JPEG XL, when the header names `image/jxl`;
2. AVIF, when it names `image/avif`;
3. WebP, when it names `image/webp`;
4. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
allows it, or when there is no `Accept` header or it is empty.
4. JPEG, when the most specific of `image/jpeg`, `image/*` and `*/*` that the
header names has a `q` above 0, or when there is no `Accept` header or it is
empty.
An entry with `q=0` refuses its format. Other `q` values do not change the
order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them
@@ -589,9 +600,8 @@ change what it asks for.
1. Open `/` in a browser over HTTPS and log in with the signing key. The login
session lasts 30 days, or until someone logs out with the generator page's
Logout button.
2. On the generator page, give the source image's URL, the width and height, the
format, quality and fit, and how long the URL lasts. Then submit the form
(`POST /generate`).
2. On the generator page, fill in the form. Its fields are listed below. Then
submit it (`POST /generate`).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on. While `debug` is on,
the URL starts with `http` instead. The name after the token is ignored. It
@@ -600,12 +610,20 @@ change what it asks for.
The form's fields:
- The format is JPEG XL unless another is chosen. A form sent with an empty
format, or none, also makes a JPEG XL URL.
- Width and height both empty or `0` keep the source's size. When only one of
them is empty or `0`, that side is worked out from the source's proportions
- The source image's URL. Its scheme is not kept (below).
- The width and height. Both empty or `0` keep the source's size. When only one
of them is empty or `0`, that side is worked out from the source's proportions
when the image is made (see Image URLs).
- How long the URL lasts is chosen from 1 minute to 1 year, or never.
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. It
is JPEG XL unless another is chosen. A form sent with an empty format, or
none, also makes a JPEG XL URL.
- The quality, a whole number from 1 to 100. The page offers five values, and
`85` unless another is chosen.
- The fit, one of the five fit modes (see Image URLs). It is `cover` unless
another is chosen.
- How long the URL lasts, in seconds, or `0` for never. The page offers never, 1
minute, 1 hour, 1 day, 1 week, 30 days and 1 year, and never unless another is
chosen.
The page refuses a form with 400 when:
@@ -613,25 +631,25 @@ The page refuses a form with 400 when:
- the source URL's path holds a `..` segment, as `/v1/image/` refuses one;
- a width, height, quality or expiry is not a number, or is out of range;
- the fit is not one of the five fit modes;
- the source URL would be fetched written another way (below).
- pixa would fetch the source URL in another form (below).
pixa fetches a source URL only as it was written. The page refuses one that pixa
would fetch another way, and shows the URL pixa would fetch. A URL with a user
name and password, which pixa does not send, is one example. A path holding
`%2F`, which pixa would fetch as `/`, or `(`, which it would fetch as `%28`, is
another (see Image URLs).
The page refuses a source URL that pixa would fetch in another form, and shows
the URL pixa would fetch. A URL with a user name and password is one example:
pixa does not send them. A URL ending in `?` with no query is another. A path
holding `%2F` is another: pixa would fetch it as `/`. So is a path holding `(`:
pixa would fetch it as `%28` (see Image URLs).
The image is made only when the URL is requested. A side that would come out
above 8192, or an AVIF output of more than 16,777,216 pixels, is refused with
400 then (see Image URLs).
Every login is made with the one signing key, so logging out ends every login
session, in every browser, a copy of the session cookie included. pixa keeps a
random value in its database. It makes the value when the database has none, and
Every login is made with the one signing key. So logging out ends every login
session, in every browser, and a copied session cookie too. pixa keeps a random
value in its database. It makes the value when the database has none, and
replaces it at each logout. A login session carries the value current at its
login. It is refused once the value has changed, also after a restart. The check
compares no time, so a login after a logout works whatever the host's clock
says. The 30 days are still measured by the host's clock.
login. Once the value has changed, the session is refused, after a restart too.
The check compares no time, so a login after a logout works whatever the host's
clock says. The 30 days are still measured by the host's clock.
The token holds the source's host, path and query, and the size, format,
quality, fit and expiry. It is encrypted with a key derived from `signing_key`.
@@ -698,7 +716,8 @@ The rest of the rules:
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
writes an EXIF block of its own, which pixa cannot stop. The block holds the
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
Nothing in it comes from the source.
Nothing in it comes from the source. The Docker image has libvips 8.16, so its
JPEG XL output carries this block.
### Limits
@@ -712,34 +731,40 @@ client goes away. The shared fetch sends the first request's ID upstream. The
lines logged for the fetch and the conversion carry that ID.
**Timeouts.** `downstream_timeout` (default `60s`) bounds the whole answer to
one request. The upstream fetch counts toward it. So do the waits for an
upstream connection and for a processing slot, up to 10 seconds each. Keep it
longer than `upstream_fetch_timeout` plus 20 seconds.
one request. The upstream fetch counts toward it. So do the waits for a
connection slot and for a processing slot. The wait for a shared
`upstream_connections` slot ends after 10 seconds, and so does the wait for a
processing slot. The wait for an `upstream_connections_per_host` slot ends only
with `downstream_timeout`. Keep `downstream_timeout` longer than
`upstream_fetch_timeout` plus 20 seconds.
A conversion whose request's `downstream_timeout` has ended stops waiting for a
processing slot. Once it has one, it gives the slot back without beginning to
encode. For a shared conversion, the first request's deadline counts. A request
waiting for a shared conversion that stops this way is answered 503
`server busy, try again later`, though its own `downstream_timeout` may not have
encode. For a shared conversion, the first request's deadline counts. When a
shared conversion stops this way, each request waiting for it is answered 503
`server busy, try again later`, even if its own `downstream_timeout` has not
ended. govips, the library pixa drives libvips through, gives no way to stop an
encode once it has begun. So an encode keeps its slot until it finishes, which
can be after `downstream_timeout`.
**Encode times.** The limits on image size bound how long an encode can take.
**Encode times.** The limits on image size bound how long a conversion can take.
They are 8192 on each side of the source and of the output, and 16,777,216
pixels for an AVIF output. The WebP and JPEG XL encoders run below libvips'
default effort. AVIF runs at the lowest effort govips can set. With one libvips
thread, as pixa runs it, 8192x8192 random pixels with an alpha channel take up
to about:
pixels for an AVIF output. Decoding an 8192x8192 source takes up to about 11
seconds. The WebP and JPEG XL encoders run below libvips' default effort. AVIF
runs at the lowest effort govips can set. With one libvips thread, as pixa runs
it, encoding 8192x8192 random pixels takes up to about:
- 23 seconds as WebP, and 16 without an alpha channel;
- 27 seconds as JPEG XL, and 13 without an alpha channel.
- 1 second as JPEG;
- 15 seconds as PNG;
- 16 seconds as GIF;
- 23 seconds as WebP with an alpha channel, and 16 without;
- 27 seconds as JPEG XL with an alpha channel, and 13 without.
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels with an alpha
channel takes up to about 34 seconds. At 8192x8192 and quality 85 it would take
about 96. These times hold at any quality and at 8 or 16 bits per sample. No
encode within the limits takes more than about 40 seconds of the default
`downstream_timeout` of 60.
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
85 it would take about 96 seconds. These times hold at any quality and at 8 or
16 bits per sample. No encode within the limits takes more than about 40 seconds
of the default `downstream_timeout` of 60.
**Processing and connections.** `max_concurrent_processing` images are decoded
and encoded at once. A request that finds every slot in use waits up to 10
@@ -749,24 +774,26 @@ seconds for one. If none frees up, it is answered 503
together, and `upstream_connections_per_host` those to one host. A fetch holds
its connection until its image has been processed. A fetch that finds every
shared connection in use waits up to 10 seconds the same way, then answers 503.
A fetch that finds every connection to its host in use waits for one until
`downstream_timeout` ends.
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
per client address. An IPv6 client is counted by its `/64`. An attempt over the
limit is refused with 429 and a `Retry-After` header. One client may send
`image_requests_per_minute` requests per minute (default 600) to `/v1/image/`
and `/v1/e/` together, `GET` and `HEAD`, cache hits included. A request over the
limit is refused with 429 and a `Retry-After` header. A browser's CORS preflight
is not counted. `0` turns the limit off.
per client address. An attempt over the limit is refused with 429 and a
`Retry-After` header. One client may send `image_requests_per_minute` requests
per minute (default 600) to `/v1/image/` and `/v1/e/` together, `GET` and
`HEAD`, cache hits included. A request over the limit is refused with 429 and a
`Retry-After` header. A browser's CORS preflight is not counted. `0` turns the
limit off. Both limits count an IPv6 client by its `/64`.
**The client address.** Behind a reverse proxy, the client address comes from
`X-Forwarded-For` only when the address pixa sees the request come from is in
`trusted_proxies`. Otherwise all users behind the proxy count as one client.
That address is not always the proxy's own. A proxy on the Docker host that
connects to pixa over `127.0.0.1` is seen as the gateway of the container's
Docker network, such as `172.17.0.1` on the default bridge. A proxy that
connects through another of the host's addresses is seen with that address. To
be sure, read it as `remoteIP` in pixa's request log while it is not in
`trusted_proxies` (see `trusted_proxies` under Configuration). The default
**The client address.** Behind a reverse proxy, pixa takes the client address
from `X-Forwarded-For` only when the request arrives from an address in
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The
address the request arrives from is not always the proxy's own. A proxy on the
Docker host that connects to pixa over `127.0.0.1` is seen as the gateway of the
container's Docker network, such as `172.17.0.1` on the default bridge. A proxy
that connects through another of the host's addresses is seen with that address.
To be sure, read the address as `remoteIP` in pixa's request log while it is not
in `trusted_proxies` (see `trusted_proxies` under Configuration). The default
`trusted_proxies` is the RFC 1918 ranges, the private address ranges. With that
default, a client with a private address can choose the address it is counted
by, through its own `X-Forwarded-For`, because its own address is trusted too.
@@ -789,7 +816,8 @@ with 413.
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name. The database is one SQLite file. SQLite is the database library pixa
embeds; nothing else needs to run.
embeds; nothing else needs to run. In WAL mode, the default, SQLite keeps
`state.sqlite3-wal` and `state.sqlite3-shm` beside it while pixa runs.
The image cache keeps these tables in the database:
@@ -815,7 +843,7 @@ Each source and variant file is written under a temporary name and synced to
disk before it is renamed into place. So a crash never leaves part of a file
under its own name.
When pixa starts, and every five minutes after, it checks the cached files
When pixa starts, and then at the spacing below, it checks the cached files
against the database. It removes each source or variant whose file is empty, or
not the size its row records. It logs a warning naming the file. A later request
for that image fetches the source or makes the variant again. The check also
@@ -853,20 +881,26 @@ also removes the expired rows of the negative cache every five minutes, at most
Every setting can be given as an environment variable, in a YAML config file
(`--config`), or both. A variable present in the environment wins over the file,
even when it is empty. The file wins over the built-in default. The one
exception is a variable set in the file's `env:` section. pixa sets that
variable while loading the file, so it wins over the environment pixa started
with. It also wins over the file's key for the same setting. A variable's value
is parsed as the same text in the file would be. A setting that holds a list
takes comma-separated entries, with the spaces around each trimmed. An empty
variable is an empty list. A value that does not parse or is invalid aborts
startup, naming the setting and its variable. A key in the file set to null,
`key:` with nothing after it included, aborts startup too: omit the key to use
the default. pixa refuses to start when a variable starts with `PIXA_` but is
not in the list below, such as a misspelled one or `PIXA_PORT`. The error names
the variable, as it names an unknown key in the file. pixa also accepts
`PIXA_CONFIG_PATH`, which names the config file, like `--config`. The variables
set by the file's `env:` section are checked the same way.
even when it is empty. The file wins over the built-in default. A variable's
value is read as the same text in quotes in the file would be. A setting that
holds a list takes comma-separated entries, with the spaces around each trimmed.
An empty variable is an empty list.
The file's `env:` section is the one exception to that order. pixa sets each
variable named there while it loads the file. So that variable wins over the
environment pixa started with, and over the file's key for the same setting.
pixa refuses to start when a value does not parse or is invalid. The error names
the setting. It also refuses to start when a key in the file is set to null,
`key:` with nothing after it included. Omit the key to use the default. It
refuses to start when the file has a key that is not in the list below, and the
error names the key. It refuses to start when a variable starts with `PIXA_` but
is not in the list below, such as a misspelled one or `PIXA_PORT`. The error
names the variable. The variables set by the file's `env:` section are checked
the same way.
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`.
pixa reads at most one config file. It reads the file given with `--config` (or
`-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
@@ -936,14 +970,27 @@ Each item below is one setting, named by its config key:
`Referer`, for example with `Referrer-Policy: no-referrer`, is not stopped.
The login and generator pages are not covered.
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
ranges pixa never fetches from, on top of the built-in ones. The built-in
ranges are loopback, private, link-local, carrier-grade NAT, benchmarking,
NAT64 and the like, and always apply. An invalid CIDR aborts startup. An IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds. So an IPv6 range such as `::/0` refuses neither it nor a
plain IPv4 address. An entry written in that form is read as the IPv4 range it
covers: `::ffff:10.0.0.0/104` refuses what `10.0.0.0/8` refuses. Such an entry
shorter than `/96` covers more than IPv4 addresses and aborts startup.
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
aborts startup. An IPv6 address in the IPv4-mapped form (`::ffff:a.b.c.d`) is
matched only as the IPv4 address it holds. So an IPv6 range such as `::/0`
refuses neither it nor a plain IPv4 address. An entry written in that form is
read as the IPv4 range it covers: `::ffff:10.0.0.0/104` refuses what
`10.0.0.0/8` refuses. Such an entry shorter than `/96` covers more than IPv4
addresses and aborts startup. The built-in ranges always apply. They are:
- `0.0.0.0/8`, this network, `0.0.0.0` included;
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
- `100.64.0.0/10`, carrier-grade NAT;
- `127.0.0.0/8`, loopback;
- `169.254.0.0/16`, link-local;
- `192.0.0.0/24`, IETF protocol assignments;
- `198.18.0.0/15`, benchmarking;
- `224.0.0.0/4`, multicast;
- `::`, the unspecified address;
- `::1`, loopback;
- `64:ff9b::/96` and `64:ff9b:1::/48`, NAT64;
- `fc00::/7`, the unique local (private) range;
- `fe80::/10`, link-local;
- `ff00::/8`, multicast.
- `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
private ranges: `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`. CIDR
ranges of the reverse proxies in front of pixa. `X-Forwarded-For` is believed
@@ -964,16 +1011,18 @@ Each item below is one setting, named by its config key:
is for testing only.
- `upstream_connections_per_host`: variable
`PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
one upstream host at once. It must be at least 1.
one upstream host at once. It must be at least 1. A fetch that finds every
connection to its host in use waits for one until `downstream_timeout` ends
(see Limits).
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
The most connections to all upstream hosts together at once, on top of the
limit per host. It must be at least 1. A fetch holds its connection until its
image has been processed (see Limits).
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
default half the CPUs pixa can use, rounded down, and at least 1. The most
images decoded and encoded at once. The number of CPUs follows a container's
CPU limit. A value that is not a whole number of at least 1 aborts startup.
Each image can take a lot of memory (see Running under upaas).
default half the CPUs Go uses (`GOMAXPROCS`), rounded down, and at least 1.
The most images decoded and encoded at once. `GOMAXPROCS` follows a
container's CPU limit. A value that is not a whole number of at least 1 aborts
startup. Each image can take a lot of memory (see Running under upaas).
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
default `600`. How many requests one client may send per minute to
`/v1/image/` and `/v1/e/` together (see Limits). `0` turns the limit off. A
@@ -996,11 +1045,12 @@ Each item below is one setting, named by its config key:
digits, hyphens and dots, with a letter in its last part) or an IP address
(IPv6 in brackets, in its shortest form). It may have a port from 1 to 65535
that has no leading zero and is not the scheme's default. Any other value
aborts startup, another scheme such as a browser extension's included.
aborts startup, including an origin with another scheme, such as a browser
extension's.
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
username for `/metrics`. pixa serves `/metrics`, and measures requests for it,
only when the username and the password are both set. Setting one without the
other aborts startup.
username for `/metrics`. pixa serves `/metrics`, and measures every HTTP
request, only when the username and the password are both set. Setting one
without the other aborts startup.
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
password for `/metrics`. Set it together with the username.
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. Sentry is an error
@@ -1009,12 +1059,12 @@ Each item below is one setting, named by its config key:
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
plain-HTTP local development (see Routes).
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
`true`, `/v1/image/` and `/v1/e/` answer every request with 503, a
`Retry-After` header of 300 seconds and a JSON error body. The health check
still answers 200 and reports `"maintenance_mode": true`. It stays 200 because
the image's Docker `HEALTHCHECK` requests it, and upaas marks a deploy failed
when its container is unhealthy. The login and generator pages and `/metrics`
keep working.
`true`, `/v1/image/` and `/v1/e/` answer every `GET` and `HEAD` request with
503, a `Retry-After` header of 300 seconds and a JSON error body. The health
check still answers 200 and reports `"maintenance_mode": true`. It stays 200
because the image's Docker `HEALTHCHECK` requests it, and upaas marks a deploy
failed when its container is unhealthy. The login and generator pages and
`/metrics` keep working.
`configs/config.example.yml` names every setting with its default.
@@ -1022,7 +1072,7 @@ Each item below is one setting, named by its config key:
- **Dependency injection**: Uber fx, a library that builds the program's parts
in dependency order and starts and stops them. `internal/app` puts every part
of pixad in one fx graph. `internal/providers` builds the image cache, the
of `pixad` in one fx graph. `internal/providers` builds the image cache, the
image proxy and the other parts that serve images and logins. Each setting's
default is declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above:
@@ -1042,7 +1092,7 @@ Each item below is one setting, named by its config key:
- `session`: the login session
- **HTTP router**: go-chi
- **Image processing**: govips, a Go wrapper for libvips
- **Database**: SQLite via modernc.org/sqlite
- **Database**: SQLite via `modernc.org/sqlite`
- **Static assets**: embedded in the binary via `//go:embed`
- **Metrics**: Prometheus, at `/metrics`. Prometheus is a monitoring system that
reads metrics from that route. pixa serves `/metrics` and measures HTTP
@@ -1074,21 +1124,22 @@ Each item below is one setting, named by its config key:
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for the development
workflow, and the Makefile targets are thin shims that call them. We provide:
workflow, and most Makefile targets are thin shims that call them. We provide:
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier, and
download the Go modules. It can be run again at any time. With `--cgo`, it
installs the C compiler and the libvips (with its JPEG XL support) and libheif
libraries instead of Node, Yarn and prettier. Compiling and testing pixa need
those.
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
libheif instead of Node, Yarn and prettier. Compiling pixa outside Docker, as
`make build` and `script/bench` do, needs those. Docker itself must already be
installed.
- `script/setup` — make a fresh clone ready for development: bootstrap, then
install-precommit.
- `script/projectname` — output the project name (`pixa`).
- `script/test` — run the test suite: build the `test` phase of the
`Dockerfile`, writing no image.
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
writing no image. The linter never runs on the host.
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
- `script/lint` — run `golangci-lint`: build the `lint` phase of the
`Dockerfile`, writing no image. The linter never runs on the host.
- `script/fmt` — format the Go code with `gofmt` and the markdown with prettier
(writes).
- `script/fmt-check` — check the same formatting (read-only), on the host.
- `script/check` — run test, lint, and fmt-check.
@@ -1097,8 +1148,8 @@ workflow, and the Makefile targets are thin shims that call them. We provide:
and `test` phases, so this runs them too.
- `script/docker-smoke` — build the image, start it, and wait for it to be
healthy.
- `script/loadtest` — measure pixad's throughput, latency and peak memory. It is
a benchmark, not part of `script/check` (see Load Test).
- `script/loadtest` — measure `pixad`'s throughput, latency and peak memory. It
is a benchmark, not part of `script/check` (see Load Test).
- `script/bench` — run the Go benchmark that serves cache hits in parallel
through the image handler, with Go on the host after `script/bootstrap --cgo`.
It is a benchmark, not part of `script/check` (see Load Test).
@@ -1110,28 +1161,42 @@ workflow, and the Makefile targets are thin shims that call them. We provide:
`script/precommit`.
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
build argument, which every stage of the `Dockerfile` declares. So every `RUN`
step after that line, the lint and test steps included, runs on every build
instead of coming from the build cache. Above the line are the steps that
install system packages and download the Go modules. Docker keeps their layers
until the base image, the step itself, `go.mod` or `go.sum` changes. Go's build
cache is kept from build to build: the steps that compile mount it, so a package
whose inputs are unchanged is not compiled again. Both `go test` runs pass
build argument. Every stage of the `Dockerfile` declares it with an
`ARG CACHEBUST` line. So every `RUN` step after that line, the lint and test
steps included, runs on every build instead of coming from the build cache. The
steps above that line install system packages and download the Go modules.
Docker keeps their layers until the base image, the step itself, `go.mod` or
`go.sum` changes.
Go's build cache is kept from build to build: the steps that compile mount it,
so a package whose inputs are unchanged is not compiled again. The test step
runs `go test`, and reruns it with `-v` after a failure. Both runs pass
`-count=1`, so no stored test result is replayed. The lint step also mounts
golangci-lint's cache. That cache reuses a package's findings only while its
files, its imports, `go.mod`, `.golangci.yml` and the golangci-lint version are
unchanged. The `test` phase runs on the Debian Go image, with Debian's libvips
and libheif. The image pixa ships is built and runs on Alpine. These scripts are
the standard copies from `sneak/prompts`, kept identical to them:
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
`script/setup` and `script/install-precommit`. `script/fmt` and
`script/fmt-check` are the standard copies with pixa's `gofmt` step kept before
prettier. prettier formats the markdown only, not the HTML templates, as it
cannot parse a Go template action inside a tag.
`golangci-lint`'s cache. That cache reuses a package's findings only while its
files, its imports, `go.mod`, `.golangci.yml` and the `golangci-lint` version
are unchanged.
The `test` phase runs on the Debian Go image, with Debian's libvips and libheif.
The image pixa ships is built and runs on Alpine.
These scripts are the standard copies from `sneak/prompts`, kept identical to
them:
- `script/check`
- `script/cibuild`
- `script/docker`
- `script/lint`
- `script/test`
- `script/setup`
- `script/install-precommit`
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
step kept before prettier. prettier formats the markdown only, not the HTML
templates, as it cannot parse a Go template action inside a tag.
## Load Test
`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how
`script/loadtest` (or `make loadtest`) measures how fast `pixad` answers and how
much memory it uses. It is a benchmark, not a check: `script/check` does not run
it. It needs Docker and Go.
@@ -1140,23 +1205,24 @@ script/loadtest # 10 seconds per scenario, 4 clients
script/loadtest 30s 32 # 30 seconds per scenario, 32 clients
```
It builds the image with `script/docker` and the load tool,
It builds the image with `script/docker`. It also builds the load tool,
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
starts a new pixad container and a new origin container, `cmd/loadtest-origin`.
The origin is an upstream host that answers every path with the same generated
1600x1200 JPEG. vegeta then sends requests from the given number of clients.
Each client sends its next request as soon as its last one is answered. Every
request asks for an image resized to 400x300 WebP:
starts a new `pixad` container and a new origin container,
`cmd/loadtest-origin`. The origin is an upstream host that answers every path
with the same generated 1600x1200 JPEG. vegeta then sends requests from the
given number of clients. Each client sends its next request as soon as its last
one is answered. Every request asks for an image resized to 400x300 WebP:
- `hit`: the same image every time, put in the cache first;
- `miss`: a new source image every time, so pixad fetches and converts each one;
- `herd`: each new source image once per client in a row. All clients then ask
for it at the same time and share one fetch and one conversion (see Limits).
- `miss`: a new source image every time, so `pixad` fetches and converts each
one;
- `herd`: every client asks for each new source image at the same moment. The
clients then share one fetch and one conversion (see Limits).
pixad refuses upstream hosts with private or local addresses, so the containers
share a Docker network in `203.0.113.0/24`, a range set aside for documentation.
A second run on the same Docker host while one is going fails, as it cannot
create that network.
`pixad` refuses upstream hosts with private or local addresses, so the
containers share a Docker network in `203.0.113.0/24`, a range set aside for
documentation. A second run on the same Docker host while one is going fails, as
it cannot create that network.
For each scenario the script prints vegeta's report and two lines of its own:
@@ -1166,19 +1232,20 @@ For each scenario the script prints vegeta's report and two lines of its own:
to the end of its answer. `50`, `95` and `99` are the 50th, 95th and 99th
percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other
numbers are not for the scenario described, such as `503` when pixad was busy;
numbers are not for the scenario described, such as `503` when `pixad` was
busy;
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since
its container started, in kB; for `hit` it includes the request that put the
image in the cache;
- `requests to the origin`: the fetches pixad made. That is one for `hit`, one
- `pixad peak memory (VmHWM)`: the peak resident memory of the `pixad` process
since its container started, in kB; for `hit` it includes the request that put
the image in the cache;
- `requests to the origin`: the fetches `pixad` made. That is one for `hit`, one
per request for `miss`, and one per image for `herd`: the requests sent
divided by the number of clients.
The numbers depend on the machine and on whatever else runs on it.
`script/bench` (or `make bench`) measures cache hits alone, without the network
and with no pixad process. A Go benchmark in `internal/handlers` serves one
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
cached image through the `/v1/image/` handler from as many goroutines at once as
the machine has CPUs. It reports the hits served per second as `hits/s`. Each
run starts with an empty cache and an empty database, a SQLite file of its own,