pixa/README.md

# pixa

pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
fetches images from other servers, resizes and converts them, and serves the
results.

A web page asks pixa for an image at a size and in a format. pixa fetches the
original image, makes the requested image from it, and serves that. It keeps
both on disk. Later requests for the same image are served from disk, without a
fetch or a conversion.

In this document, the upstream host is the server that holds an original image.

## Getting Started

```bash
# clone and build
git clone https://git.eeqj.de/sneak/pixa.git
cd pixa
make build

# make a config file from the example and put a real signing key in it
cp configs/config.example.yml config.yml
openssl rand -base64 32   # prints a new signing key
$EDITOR config.yml        # replace the signing_key placeholder with it
./bin/pixad --config config.yml

# or build and run the Docker image
make docker
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
```

`make build` compiles pixa on your machine into `bin/pixad`. `pixad` is the
server program. Compiling needs the libraries `script/bootstrap --cgo` installs
(see Deployment).

The signing key is pixa's one required secret. It must be 32 characters or
longer. The placeholder in the example file is refused at startup.

Once pixa runs, it serves images through URLs of this form:

```
/v1/image/<host>/<path>/<size>.<format>
```

For example, this URL serves `https://cdn.example.com/photos/cat.jpg` at 800x600
as WebP:

```
/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp
```

A URL for a host in `allowlist_hosts` needs nothing more. A URL for any other
host needs a signature (see Signed URLs). Someone who knows the signing key can
also make an encrypted URL on pixa's own web page (see Encrypted URLs). An
encrypted URL works for any host and hides the original image's URL.

The example config allowlists no host. Every image on an allowlisted host is
served to anyone, without a signature, at any size and in any format. So
allowlist only hosts you control, where nobody else can put images.

A container takes its settings from environment variables (see Configuration).
Only `PIXA_SIGNING_KEY` is required. Without it, the container exits at startup
and names the variable. Every other setting has a built-in default. A config
file mounted at `/etc/pixa/config.yml` is read when present. An environment
variable wins over the same setting in the file.

## Deployment

pixa listens on plain HTTP. It runs behind a reverse proxy: a server in front of
it that receives the public requests and passes them on. The reverse proxy
terminates TLS, the encryption behind `https`.

The reverse proxy must:

- terminate TLS. The login and generator pages work only over HTTPS (see
  Routes).
- pass the `Host`, `Origin` and `Referer` headers on unchanged. pixa refuses a
  form from its pages unless `Origin` or `Referer` names the host in `Host`. It
  builds encrypted URLs from `Host`. It checks `Referer` against
  `referer_blocklist`.
- set `X-Forwarded-For` to the client's address. `trusted_proxies` must hold the
  proxy's address as pixa sees it. The login limit and
  `image_requests_per_minute` then count each client by its own address.
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
  is the longest pixa takes to fetch, convert and send an image.

[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server.
Caddy does everything in this list without further settings, and gets its own
TLS certificate.

The reverse proxy may also refuse `/metrics`, as the example does. Only a
scraper that reaches pixa directly can then read it. pixa itself asks for
`metrics.username` and `metrics.password` there.

pixa does the rest itself:

- it checks signatures and encrypted URLs;
- it applies the allowlist;
- it refuses upstream hosts with private or local addresses;
- it limits login attempts and each client's image requests;
- it limits the size of upstream responses and the dimensions of images;
- it sends the security headers, `Strict-Transport-Security` included, with
  every response.

The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
database and the disk cache:

- It needs a persistent volume. Without one, every restart starts with an empty
  cache. Losing the directory also ends every login session (see Encrypted
  URLs).
- In the container, the startup script gives the directory to the user pixa runs
  as, uid 65532, and sets its mode to `750`. Outside the container, the user
  running pixa must be able to write it.
- `cache_max_bytes` limits the cached images. The limit does not count the
  database, or files still being written. Eviction runs in the background, so
  the cache can pass the limit for a while. Leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
  each start from the free space on the volume (see Configuration). Anything
  else that fills or frees space on the volume moves that default.

A load balancer's health check can request `/.well-known/healthcheck.json`. It
answers 200 whenever pixa is running, in maintenance mode too (see
`maintenance_mode`).

On `SIGTERM` or `SIGINT`, pixa stops accepting connections. The requests in
progress and the images being processed get 5 seconds to finish. A request not
finished by then is cut off. pixa then saves the last-used times it has not yet
written (see Storage), and exits. It exits with status 1 when:

- images were still being processed after those 5 seconds;
- some of those times could not be saved;
- another part of pixa failed to stop.

Otherwise it exits with 0. `docker stop` waits 10 seconds before it kills the
container.

Outside Docker, pixa needs these at run time:

- libvips, the image library pixa converts images with, including its JPEG XL
  support. pixa does not start without that support. On Alpine it is the
  `vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do
  the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on.
- libheif, which libvips needs for AVIF.
- the system's CA certificates, which HTTPS fetches need.

Compiling pixa needs the development files of libvips and libheif, `pkg-config`
and a C compiler. `script/bootstrap --cgo` installs them, and libvips and
libheif with them. The `Dockerfile` installs its own packages.

## Running under upaas

[upaas](https://git.eeqj.de/sneak/upaas) is the deployment tool pixa is meant to
run under. Its app for pixa needs:

- **Port:** pixa listens on container port `8080`.
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its database and
  cache.
- **Environment variables:**
    - `PIXA_SIGNING_KEY` (required): the secret for signed URLs, encrypted URLs
      and login. It is 32 characters or more, for example from
      `openssl rand -base64 32`.
    - `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
      comma-separated; default none. Every image on a listed host is served to
      anyone, so list only hosts you control, where nobody else can put images.
    - `PIXA_CACHE_MAX_BYTES`: the disk cache limit in bytes; `0` turns the disk
      cache off. Set it. The volume is a directory on the host's filesystem. The
      host and every other app use that filesystem too. The default is 75% of
      the sum of that filesystem's free space and what the cache already holds.
      So by default the cache may grow to three quarters of the space the host
      and the other apps have left. The default is also at least 500 MiB. So
      when that space is 500 MiB or less, the cache may take all of it.
    - the rest are in the list under Configuration.
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
  the images it processes at once. On top of that, the fetched source images
  waiting for processing can take `upstream_connections` times
  `upstream_max_response_size`. 4.2 GiB is a measured value. It is the largest
  rise in resident memory measured while pixa saved one 8192x8192 source, the
  largest it accepts, to each output format at full size. The rise was largest
  for a 16-bit JPEG XL of random pixels saved as WebP. At the defaults the total
  is half the CPUs pixa sees times 4.2 GiB, plus 64 times 50 MiB. On a host
  where pixa sees 48 CPUs, that is 24 times 4.2 GiB plus 3.1 GiB, about 104 GiB.
  Set `PIXA_MAX_CONCURRENT_PROCESSING` and `PIXA_UPSTREAM_CONNECTIONS` so that
  this fits the memory the container may use.
- **Temporary files:** while pixa processes a large source, libvips writes the
  decoded pixels to a file in `/tmp`. The file is removed from the directory
  once it is written, so a listing of `/tmp` does not show it. It keeps its
  space until the image is done: on disk, or in memory where `/tmp` is a tmpfs.
  For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory
  as well, and the 4.2 GiB above includes that memory. For an 8192x8192 source
  the file was measured at:
    - 192 MiB for an 8-bit RGB JPEG;
    - 512 MiB for a 16-bit RGBA PNG or JPEG XL;
    - 1 GiB for a 32-bit floating-point RGBA JPEG XL.

    So count up to 1 GiB per image processed at once: 24 GiB at the defaults on
    a host where pixa sees 48 CPUs. This `/tmp` is in the container's own
    filesystem, not on the volume at `/var/lib/pixa`. `cache_max_bytes` does not
    count these files, and the 4.2 GiB above does not include them.

- **Health check:** the image's `HEALTHCHECK` requests
  `/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
  after a deploy and marks the deploy failed unless it is `healthy`. The probe
  takes the port from `PORT` (default `8080`). It does not see a port changed
  only in a mounted config file, so change the port with `PORT`.

## Rationale

A web application that shows images needs each one at several sizes and in
several formats. pixa makes them on request from one original, and keeps what it
made. The application stores one image per picture and hands out URLs. pixa is
one binary. At run time it needs only the image libraries and the CA
certificates named under Deployment. Signed URLs with an expiry keep strangers
from using it to resize anything they like. Allowlisted hosts give open access
where that is wanted.

## Design

### Concepts

These terms are the names the code uses:

- **Source**: the image at one upstream URL, named by its host, path and query.
- **Source key**: the SHA-256 hash of a source's host, path and query. SHA-256
  is a hash function: it turns any text into a fixed-length fingerprint. The
  source key names the source's file and its row in the database.
- **Transform**: the output size, format, quality and fit that an image request
  asks for. With the format `auto`, pixa chooses the output format from the
  `Accept` header.
- **Image request**: a source and a transform. Each image route reads one from
  its URL.
- **Variant**: the image pixa serves for an image request, made from its source
  with its transform.
- **Variant key**: the SHA-256 hash of a variant's source, size, format, quality
  and fit. It names the variant's file and its row in the database, and gives
  its `ETag`.
- **Signed URL**: a `/v1/image/` URL whose `sig` parameter signs its image
  request and its expiry `exp` with `signing_key`.
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
  expiry, encrypted with a key derived from `signing_key`.
- **Login session**: an encrypted cookie that keeps a browser logged in to the
  generator page. It lasts 30 days, or until a logout (see Encrypted URLs).
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
  after a fetch failed in one of these ways:
    - the upstream answered with an error status;
    - the body's first bytes were not those of the format its `Content-Type`
      names.
- **Image cache**: the part of pixa that keeps the sources, the variants and the
  negative cache (see Storage).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
  sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
  when it can, and otherwise makes it from the source.

### Routes

pixa answers these routes. Any other path answers 404. A listed path answers 405
to any method the list does not give for it.

- `GET /` — the login page, or the generator page with a login session (see
  Encrypted URLs). Needs nothing. Answers 200.
- `POST /` — log in with the signing key typed into the login page. Needs the
  login page's form (below). For the right key it answers 303 to `/` and sets
  the login session cookie. For a wrong key it answers 200 with the login page
  and an error. Answers 429 over the login limit (see Limits).
- `POST /generate` — make an encrypted URL from the generator page's form. Needs
  a login session and the generator page's form. Without a login session it
  answers 303 to `/`. Answers 200 with the page showing the URL. Answers 400
  with the page naming a field that is not valid. Answers 500 when the URL
  cannot be made.
- `POST /logout` — log out with the generator page's Logout button. That ends
  every login session, in every browser (see Encrypted URLs), and removes the
  cookie. Needs a login session and the generator page's form. Without a login
  session it only removes the cookie. Answers 303 to `/`. Answers 500 when the
  logout cannot be saved in the database.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
  `/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
  and converted (see Image URLs). Needs a signature, unless the host is
  allowlisted (see Source Hosts). Answers:
    - 200 with the image.
    - 304 when `If-None-Match` matches the image's `ETag`.
    - 400 for a URL or parameter that is not valid.
    - 400 when the format is `auto` and the `Accept` header is not valid.
    - 406 when the format is `auto` and `Accept` allows none of the formats it
      chooses from.
    - 401 for a missing or wrong signature.
    - 401 for a missing `exp`.
    - 401 for an `exp` in the past.
    - 403 when the request's `Referer` names a host in `referer_blocklist`. This
      is checked before the signature, the cache and the upstream fetch.
    - 403 when the upstream host, or a host it redirects to, is `localhost` or
      ends in `.localhost` or `.local`.
    - 403 when the upstream host, or a host it redirects to, has an address in a
      blocked network (see `blocked_networks`).
    - 429 over `image_requests_per_minute` (see Limits).
    - 499, with no body, when the client goes away before its image is ready,
      unless its request is the one making the image (see Limits).
    - 415 when the upstream answers with a `Content-Type` that is not one of the
      source formats (see Image URLs).
    - 415 when the upstream sends a body whose first bytes are not those of the
      format its `Content-Type` names.
    - 415 when the upstream sends an image whose first bytes libvips reads as
      another format.
    - 502 when the upstream answers with an error status.
    - 502 for 5 minutes after that for the same source URL, when pixa has
      neither the image nor its source in its cache.
    - 502 likewise for 5 minutes after a body whose first bytes are not those of
      its `Content-Type`.
    - 503 when pixa is busy (see Limits).
    - 503 in maintenance mode.
    - 504 when the upstream has not sent its response headers within
      `upstream_fetch_timeout`.
    - 500 when that time runs out while the image itself is still arriving.
    - 500 for any other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
  (see Encrypted URLs). Needs nothing but the URL. Answers:
    - 200 with the image.
    - 304 when `If-None-Match` matches the image's `ETag`.
    - 400 for a token that does not decrypt.
    - 400 for a token that asks for a size or fit that is not valid.
    - 410 once the token has expired.
    - 400 when the token's format is `auto` and the `Accept` header is not
      valid.
    - These answers, for the same causes as on `/v1/image/`:
        - 406
        - 403
        - 415
        - 429
        - 499, with no body
        - 502
        - 503
        - 504
        - 500
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs
  nothing. Answers 200.
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
  `uptime_seconds`, `uptime_human`, `version`, `appname` and `maintenance_mode`.
  Needs nothing. Answers 200, always.
- `GET /static/<file>` — the stylesheet and script the login and generator pages
  load. Needs nothing. Answers 200, or 404 for a file that does not exist. It
  answers any other method as it answers `GET`.
- `GET /metrics` — Prometheus metrics (see Architecture). Needs HTTP basic
  authentication with `metrics.username` and `metrics.password`. Answers 200.
  Answers 401 without them. Answers 404 when they are not set, as the route then
  does not exist.

A browser's CORS preflight request to any path under `/v1/` answers 200, in
maintenance mode too. CORS is the browser rule that lets a page on one site read
responses from another. The preflight is the `OPTIONS` request the browser sends
first.

Every response carries an `X-Request-Id` header with the request's ID. A client
can quote it when reporting a problem. A reverse proxy may send an
`X-Request-Id` header with the request. pixa keeps that ID when it is 1 to 64
letters, digits, `-`, `_` or `.`. Otherwise pixa makes a random ID, which tells
nothing about the machine or the other requests. pixa's log line for the request
carries the same ID as `request_id`. So do the lines it logs when it fetches,
converts and serves an image. The fetch sends it to the upstream host as
`X-Request-Id`.

The `POST` routes accept only a form that pixa's own page served. The page puts
a token in the form and sets a cookie to match. A request without both is
refused with 403. So another site cannot submit the form from a visitor's
browser. A form is also refused with 403 when the page's host is not the `Host`
header pixa receives. So a reverse proxy in front of pixa must pass that header
on unchanged. A form body over 1 MiB is refused with 413.

The login and generator pages are meant to be opened over HTTPS. While `debug`
is off, a form sent from a page opened over plain HTTP is refused with 403.
While `debug` is on, a form sent from a page opened over HTTPS is refused
instead. Plain HTTP is for development on the browser's own machine. The login
session cookie is always marked `Secure`. Over plain HTTP a browser keeps such a
cookie only for its own machine (`localhost`), if at all.

The image routes answer the errors listed for them with a JSON body holding
`error`, `status` and `timestamp`.

### Image URLs

An image URL has one of these forms. The second has no format:

```
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
/v1/image/<host>/<path>/<size>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
```

- `<host>`: the upstream host, with a port when the source URL has one (see
  Source Hosts).
- `<path>`: the source URL's path, and its query after `%3F` (below).
- `<size>`: `orig` or `<width>x<height>`, for example `800x600`. A side of `0`
  is worked out from the other side and the source's proportions (below). `orig`
  is `0x0`: the source's own size.
- `<format>`: `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`, `avif`,
  `jxl` (JPEG XL), `gif`, or `auto` (below). `orig` is the source's own format.
  A URL with no format is served as JPEG XL, the default, as with `jxl`.
- `sig` and `exp`: the signature and its expiry, needed unless the host is
  allowlisted (see Signed URLs).
- `q`: the output quality, a whole number from 1 to 100. It is `85` when the URL
  has no `q`.
- `fit`: how the image is fitted to `<size>`: `cover`, `contain`, `fill`,
  `inside` or `outside` (below). It is `cover` when the URL has no `fit`.

`q` and `fit` are part of what is cached, so each value of either is a separate
cached image. A request is refused with 400 when its query string cannot be
decoded, or when it gives any parameter more than once. A request with a `q`,
`fit` or `exp` that is empty or not valid is refused with 400 too.

pixa fetches sources over HTTPS only, and only from hosts with valid
certificates. With `allow_http` set, it fetches every source over plain HTTP
instead. That is for testing only.

**The source path and query.** A source URL's query follows its path after
`%3F`, the encoded `?`:

```
/v1/image/<host>/<path>%3F<query>/<size>.<format>
```

pixa decodes the percent-escapes in the image URL's path once. It then reads the
host up to the first `/`, and the size and format after the last `/`. Between
them it reads the source path up to the first `?`, and the source query after
it. So these must be percent-encoded, as `%3F` and `%25`:

- the `?` before the query;
- any `?` in the query;
- every `%` in the source path or query.

Any other character may be written as itself or percent-encoded. `/`, `;`, `,`,
`=` and `&` work either way. Neither the source path nor the source query may
hold a `#`. An image URL that writes one as `%23` is refused with 400. For
example, the image
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 as WebP is:

```
/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp
```

Signed URLs shows this URL with its signature.

pixa keeps the source path decoded and fetches it percent-encoded again. Every
character other than these is sent as a percent-escape: letters, digits, `-`,
`.`, `_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@`. So:

- `my%20cat.jpg` in the image URL is the source path `/my cat.jpg`, fetched as
  `/my%20cat.jpg`;
- `my%2520cat.jpg` is the source path `/my%20cat.jpg`, fetched as
  `/my%2520cat.jpg`;
- `100%25.png` is fetched as `/100%25.png`.

The source query is sent as the source URL writes it, its percent-escapes
included. Two kinds of character are percent-encoded first: those a browser
percent-encodes in a query, and a `%` that does not begin a percent-escape. The
first kind is a space, `"`, `'`, `<`, `>`, control characters and characters
outside ASCII. pixa signs and caches the query in that form. So
`img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are the same
source query, `name=my%20cat`.

**Sizes.** The largest width or height is 8192, for the source and for the
output. A size with one side `0` has that side worked out from the other side
and the source's proportions. The side is rounded down, but is at least 1. When
it would come out above 8192, the request is refused with 400. The fit modes:

- `cover`: scale the image to cover the size, keeping its proportions, and cut
  out the centre. The output is exactly the size.
- `contain`: scale the image to fit inside the size, keeping its proportions.
  The output can be smaller than the size on one side.
- `fill`: stretch the image to the size, without keeping its proportions.
- `inside`: as `contain`, but never enlarge the image.
- `outside`: scale the image to cover the size, keeping its proportions, without
  cutting. The output can be larger than the size on one side. An 8192x1 source
  at `100x0` is 100x1 by its proportions, but comes out at 8192x1. A side that
  would come out above 8192 is refused with 400.

An AVIF output of more than 16,777,216 pixels is refused with 400 before
anything is decoded. That is the number of pixels in 4096x4096 or 8192x2048. A
larger AVIF encode could take more than 40 seconds (see Limits). The limit
includes `orig` of an AVIF source, and `auto` when it chooses AVIF. So a client
whose `Accept` names `image/avif` but not `image/jxl` is refused such a size.

**Source formats.** The source may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The
upstream's `Content-Type` must name its format, and the image's first bytes must
match it. An upstream answering with any other `Content-Type`, SVG included, is
refused with 415. So is a body whose first bytes are not those of the format its
`Content-Type` names: a JPEG sent as `image/png`, text sent as an image, or an
empty body. pixa then does not fetch that source URL again for 5 minutes, as
after an upstream error status. Meanwhile it answers 502 for an image that would
need that fetch. An image whose first bytes libvips reads as another format is
refused with 415 too. An SVG whose first bytes pass the check for AVIF is one
example.

**The format `auto`.** With `auto`, pixa chooses the format for each request
from its `Accept` header, in this order:

1. JPEG XL, when the header names `image/jxl`;
2. AVIF, when it names `image/avif`;
3. WebP, when it names `image/webp`;
4. JPEG, when the most specific of `image/jpeg`, `image/*` and `*/*` that the
   header names has a `q` above 0, or when there is no `Accept` header or it is
   empty.

An entry with `q=0` refuses its format. Other `q` values do not change the
order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them
also send `image/*` and `*/*`. pixa never sends a format the client refused.
When the header allows none of the four, the answer is 406. A header that does
not parse, or has a `q` that is not a number from 0 to 1, is refused with 400.
The signature, or the token of an encrypted URL, covers `auto` itself, so one
URL serves every client. Each format chosen is cached as a separate image. Every
answer that depends on `Accept` carries `Vary: Accept`: the image, a 304, and
the 400 and 406 above. So a shared cache keeps the formats apart too.

**Response headers.** An image is served with
`Cache-Control: public, max-age=<seconds>, immutable`. When the URL has an
expiry (an `exp`, or the expiry of an encrypted URL), `max-age` is the whole
seconds left until then, at most one year. So no browser or proxy cache keeps
the image after pixa would refuse the URL. A URL with no expiry gets one year.
`immutable` only stops a client revalidating while its copy is fresh. The
response also carries `X-Pixa-Cache`: `HIT` for an image served from the cache,
and `MISS` for one made for this request. The `ETag` is the first 16 characters
of the variant key, in quotes.

### Signed URLs

A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
time (seconds since 1 January 1970) when it expires, and `sig`, its signature.
The signature is an HMAC-SHA256: a hash of the request's details, keyed with
`signing_key`, that only someone who knows the key can make. Signatures match
exactly: every part of the signed data must be exactly what was signed. There is
no suffix, wildcard or partial matching.

The signed data is one line of nine parts, joined with `:`:

```
HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit")
```

- `host`: the source host as the URL writes it up to the first `/`, in lower
  case. An IPv6 address keeps its brackets, and a port is kept after its `:`.
  Examples: `cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`.
- `path`: the source path with its percent-escapes decoded, as under Image URLs,
  then with each `%` written as `%25` and each `:` as `%3A`. So
  `/photos/my%20cat.jpg` in the URL is signed as `/photos/my cat.jpg`, and
  `/photos/a:b.jpg` as `/photos/a%3Ab.jpg`.
- `query`: the source query as under Image URLs, with `%` and `:` written as in
  `path`. It is empty when there is no query. Neither part then holds a `:`, so
  path `/a:b` with no query and path `/a` with query `b:` are signed
  differently.
- `width` and `height`: the requested size in pixels. A side of `0` is signed as
  `0`, and `orig` as `0` and `0`.
- `format`: the output format as listed under Image URLs, with `original` signed
  as `orig`, `jpg` as `jpeg`, and no format as `jxl`. So a URL with no format
  has the signature of the same URL ending in `.jxl`. `auto` is signed as
  `auto`, not as the format chosen for the request.
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature
  expires. A request whose `exp` is not a whole number is refused with 400. An
  empty `exp=` is one example.
- `quality`: the URL's `q`, or `85` when the URL has no `q`.
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.

The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
compares it exactly. A signature encoded without padding is refused with 401.
Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding.

**Example.** With the signing key `example-signing-key-for-documentation`,
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
1704067200, with the default quality and fit:

1. Build the signed data:
   `cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
2. Compute its HMAC-SHA256 with the signing key.
3. Encode the result as base64url, keeping the `=` padding:
   `-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=`
4. The URL:
   `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=&exp=1704067200`

For the same image at quality 40 with fit `contain`, the signed data ends in
`:40:contain`. The signature is `5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=`,
and the URL is
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.

For a source URL with a query,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
800x600 WebP with the same key, expiry, quality and fit:

1. The signed data, with the query's `:` written as `%3A` and its `%` as `%25`:
   `cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
2. The signature: `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
3. The URL, with the `?` before the query written as `%3F` and the query's `%`
   as `%25`:
   `/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`

### Encrypted URLs

An encrypted URL is an image URL made on pixa's own web page by someone who
knows the signing key. It works for any upstream host, allowlisted or not,
without a signature. Whoever gets it can neither read the source URL from it nor
change what it asks for.

1. Open `/` in a browser over HTTPS and log in with the signing key. The login
   session lasts 30 days, or until someone logs out with the generator page's
   Logout button.
2. On the generator page, fill in the form. Its fields are listed below. Then
   submit it (`POST /generate`).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
   it expires. `<host>` is the host the page was opened on. While `debug` is on,
   the URL starts with `http` instead. The name after the token is ignored. It
   only gives the URL a file extension: `jpg` for `orig` and `auto`, and `jxl`
   for a form with no format.

The form's fields:

- The source image's URL. Its scheme is not kept (below).
- The width and height. Both empty or `0` keep the source's size. When only one
  of them is empty or `0`, that side is worked out from the source's proportions
  when the image is made (see Image URLs).
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`.
  `jxl` is selected unless another is chosen. A form sent with an empty format,
  or none, also makes a JPEG XL URL.
- The quality, a whole number from 1 to 100. The page offers five values: `25`,
  `50`, `70`, `85` and `100`. `85` is selected unless another is chosen.
- The fit, one of the five fit modes (see Image URLs). `cover` is selected
  unless another is chosen.
- How long the URL lasts, in seconds, or `0` for never. The page offers seven
  choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is
  selected unless another is chosen.

The page refuses a form with 400 when:

- the source URL does not parse, or has no host;
- the source URL's path holds a `..` segment, as `/v1/image/` refuses one;
- a width, height, quality or expiry is not a number, or is out of range;
- the fit is not one of the five fit modes;
- pixa would fetch the source URL in another form (below).

The page refuses a source URL that pixa would fetch in another form, and shows
the URL pixa would fetch. A URL with a user name and password is one example:
pixa does not send them. A URL ending in `?` with no query is another. A path
holding `%2F` is another: pixa would fetch it as `/`. So is a path holding `(`:
pixa would fetch it as `%28` (see Image URLs).

The image is made only when the URL is requested. A side that would come out
above 8192, or an AVIF output of more than 16,777,216 pixels, is refused with
400 then (see Image URLs).

Every login is made with the one signing key. So logging out ends every login
session, in every browser, and a copied session cookie too. pixa keeps a random
value in its database. It makes the value when the database has none, and
replaces it at each logout. A login session carries the value current at its
login. Once the value has changed, the session is refused, after a restart too.
The check compares no time, so a login after a logout works whatever the host's
clock says. The 30 days are still measured by the host's clock.

The token holds the source's host, path and query, and the size, format,
quality, fit and expiry. It is encrypted with a key derived from `signing_key`.
The host is kept in lower case and an empty path as `/`, as for `/v1/image/`. So
`https://CDN.Example.com` and `https://cdn.example.com/` are one source, fetched
and cached once. The source URL's scheme is not kept. The image is fetched like
any other (see Image URLs), and the blocked networks still apply.

The expiry is fixed in the token when the URL is made. It cannot be changed or
revoked afterwards. Until then the image is served with a `max-age` that ends at
the expiry (see Image URLs). After it, the URL answers 410 `URL has expired`. A
URL made to last forever holds no expiry. It stops working only when
`signing_key` changes. Changing the key makes every encrypted URL already handed
out answer 400, and ends every login session.

### Source Hosts

Hosts in `allowlist_hosts` are served without a signature. Any other host needs
one. An entry is written in one of two ways:

- `cdn.example.com`: an exact match, only that host;
- `.example.com`: a suffix match, which matches `cdn.example.com`,
  `images.example.com` and `example.com` itself.

An IP address is matched exactly. Write an IPv6 address without brackets. An
entry that is neither a host name nor an IP address aborts startup. A host name
here is letters, digits, hyphens, underscores and dots, with at most one leading
dot. So an entry with a port, or a `*.` wildcard, aborts startup.

pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
So `CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
port needs a signature even when the host is allowlisted. So does one whose host
ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host
must be one of:

- a host name or an IPv4 address: letters, digits, hyphens, underscores and
  dots;
- an IPv6 address in brackets.

Either may be followed by `:` and a port. Any other host is refused with 400.
One holding `@` is an example. So is one with a `?`, `#`, `%`, `\` or space
written percent-encoded.

### Image Metadata

pixa decodes and re-encodes every image it serves, and removes all metadata from
the output. This cannot be turned off. The metadata removed is:

- EXIF, which can hold the GPS position, the camera make, model and serial
  number, the capture time and an embedded thumbnail;
- XMP and IPTC, two other metadata formats;
- the ICC colour profile.

The rest of the rules:

- The `orig` format means the source's own format, not the source's bytes. An
  `orig` image is re-encoded and stripped like any other.
- An image with an EXIF orientation is turned upright first, so it displays the
  same without the tag. A requested size applies to the upright image.
- An image with an ICC profile is first converted to sRGB, the standard colour
  space, since clients show an image with no profile as sRGB. Colours outside
  sRGB are clipped: the most saturated ones in a Display P3 photo, for example.
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
  writes an EXIF block of its own, which pixa cannot stop. The block holds the
  image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
  Nothing in it comes from the source. The Docker image has libvips 8.16, so its
  JPEG XL output carries this block.

### Limits

**Shared work.** Several requests for one variant can miss the cache at the same
time. They then share one upstream fetch, or one read of the cached source, and
one conversion. The first request does the work. The others wait for its image
or its error, and hold no upstream connection or processing slot of their own. A
waiting request stops waiting when its own client goes away. It is then
answered 499. The work goes on for the others even when the first request's
client goes away. The shared fetch sends the first request's ID upstream. The
lines logged for the fetch and the conversion carry that ID.

**Timeouts.** `downstream_timeout` (default `60s`) bounds the whole answer to
one request. The upstream fetch counts toward it. So do the waits for a
connection slot and for a processing slot. The wait for a shared
`upstream_connections` slot ends after 10 seconds, and so does the wait for a
processing slot. The wait for an `upstream_connections_per_host` slot ends only
with `downstream_timeout`. Keep `downstream_timeout` longer than
`upstream_fetch_timeout` plus 20 seconds.

A conversion whose request's `downstream_timeout` has ended stops waiting for a
processing slot. If its `downstream_timeout` ends after it has a slot but before
it begins to encode, it gives the slot back. For a shared conversion, the first
request's deadline counts. When a shared conversion stops this way, each request
waiting for it is answered 503 `server busy, try again later`, even if its own
`downstream_timeout` has not ended. govips, the library pixa drives libvips
through, gives no way to stop an encode once it has begun. So an encode keeps
its slot until it finishes, which can be after `downstream_timeout`.

**Encode times.** The limits on image size bound how long a conversion can take.
They are 8192 on each side of the source and of the output, and 16,777,216
pixels for an AVIF output. Decoding an 8192x8192 source takes up to about 11
seconds. The WebP and JPEG XL encoders run below libvips' default effort. AVIF
runs at the lowest effort govips can set. With one libvips thread, as pixa runs
it, encoding 8192x8192 random pixels takes up to about:

- 1 second as JPEG;
- 15 seconds as PNG;
- 16 seconds as GIF;
- 23 seconds as WebP with an alpha channel, and 16 without;
- 27 seconds as JPEG XL with an alpha channel, and 13 without.

An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
85 it would take about 96 seconds with an alpha channel, and 52 without. These
times hold at any quality and at 8 or 16 bits per sample. No encode within the
limits takes more than about 40 seconds of the default `downstream_timeout`
of 60.

**Processing and connections.** `max_concurrent_processing` images are decoded
and encoded at once. A request that finds every slot in use waits up to 10
seconds for one. If none frees up, it is answered 503
`server busy, try again later`. `upstream_connections` bounds the fetches in
progress to all upstream hosts together, and `upstream_connections_per_host`
those to one host. A fetch holds its connection until its image has been
processed. A fetch that finds every shared connection in use waits up to 10
seconds the same way, then answers 503. A fetch that finds every connection to
its host in use waits for one until `downstream_timeout` ends.

**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
per client address. An attempt over the limit is refused with 429 and a
`Retry-After` header. One client may send `image_requests_per_minute` requests
per minute (default 600) to `/v1/image/` and `/v1/e/` together, `GET` and
`HEAD`, cache hits included. A request over the limit is refused with 429 and a
`Retry-After` header. A browser's CORS preflight is not counted. `0` turns the
limit off. Both limits count an IPv6 client by its `/64`.

**The client address.** Behind a reverse proxy, pixa takes the client address
from `X-Forwarded-For` only when the request arrives from an address in
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The
address the request arrives from is not always the proxy's own. A proxy on the
Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of
the container's Docker network, such as `172.17.0.1` on the default bridge. A
proxy that connects through another of the host's addresses is seen with that
address. To be sure, find the address as described under `trusted_proxies` (see
Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the
private address ranges. With that default, a client with a private address is
itself trusted. It can then choose the address it is counted by, through its own
`X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes
this. Behind a content delivery network (CDN) or proxy whose addresses are not
in `trusted_proxies`, all requests arriving from one address count as one
client. Such a deployment needs a higher `image_requests_per_minute`, or `0`.

**Request limits.** pixa reads a request's headers within 10 seconds and the
whole request within 30 seconds. Headers are limited to 8 KiB. An idle
connection is closed after 120 seconds. A form body over 1 MiB is refused
with 413.

### Storage

- **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
  each source URL
- **Database**: `<state_dir>/state.sqlite3`
- **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`

`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name. The database is one SQLite file. SQLite is the database library pixa
embeds; nothing else needs to run. By default the database is in WAL mode,
SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal`
and `state.sqlite3-shm` beside the file while pixa runs.

The image cache keeps these tables in the database:

- `source_content`: a row for each source, by source key, with its size and when
  it was last used
- `variant_content`: a row for each variant, by variant key (column
  `cache_key`), with its size and when it was last used
- `cache_usage`: the total size of the cached sources and variants, which
  eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when
  it expires

The `logout` table holds the random value that a logout replaces (see Encrypted
URLs).

pixa reads a cached source's format from its file's first bytes. When they match
none of the source formats, pixa fetches the source again. Two source URLs that
serve the same bytes are stored twice, once under each URL. A source is stored
by writing its file and then its row. When the row cannot be written, the file
is removed again.

Each source and variant file is written under a temporary name and synced to
disk before it is renamed into place. So a crash never leaves part of a file
under its own name.

When pixa starts, and then at the spacing below, it checks the cached files
against the database. It removes each source or variant whose file is empty, or
not the size its row records. It logs a warning naming the file. A later request
for that image fetches the source or makes the variant again. The check also
drops the rows of variants whose files are missing. When pixa cannot read a
variant file's size for another reason, it keeps the variant's row and logs a
warning naming the variant key and the error. During a check, pixa skips a
directory it cannot list and a file whose size it cannot read. It logs a warning
naming the path and the error, and goes on with the rest of the check. A file
renamed or removed after pixa listed its directory gets no warning.

A check reads the database once for every cached file, so the checks are spaced
out. After a check ends, pixa waits at least ten times as long as that check
took. Eviction runs every five minutes (below). The next check starts at the
first of those runs after that wait. After a check that took one minute, the
next one starts 10 to 15 minutes after that check ended.

A cache hit opens the variant's file and nothing else. It makes no database
call. The content type comes from the file's first bytes, which are those of the
format pixa wrote it in. A file whose first bytes are not those of a format pixa
writes is not served, and the image is converted again. A hit hands the time it
was served to a background writer and does not wait for the writer. Eviction
orders by that time. The hit is counted in memory, for the
`pixa_cache_hits_total` metric. A miss that uses a cached source also hands that
writer the time it used the source. Once a second the writer writes the waiting
times in one transaction. Up to 10,000 access times of variants, and as many of
sources, can wait to be written. A time handed over beyond that is dropped. Its
image then only looks less recently used to eviction. `script/bench` measures
how many hits pixa serves per second (see Load Test).

Eviction removes the least recently used sources and variants until the cache is
within `cache_max_bytes`. It runs after each store, and every five minutes. It
also removes the expired rows of the negative cache every five minutes, at most
1000 per statement, so a request waits behind 1000 deletions at most.

### Configuration

Every setting can be given as an environment variable, in a config file
(`--config`), or both. The file is YAML, a plain-text format. A variable present
in the environment wins over the file, even when it is empty. The file wins over
the built-in default. pixa reads a variable's value as a quoted string in the
file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts
startup. A setting that holds a list takes comma-separated entries, with the
spaces around each trimmed. An empty variable is an empty list.

The file's `env:` section is the one exception to that order. pixa sets each
variable named there while it loads the file. So that variable wins over the
environment pixa started with, and over the file's key for the same setting.

pixa refuses to start when a value does not parse or is invalid. The error names
the setting. It also refuses to start when a key in the file is set to null. A
key written as `key:` with nothing after it is null. Omit the key to use the
default. It refuses to start when the file has a key that is not in the list
below, and the error names the key. It refuses to start when a variable starts
with `PIXA_` but is not in the list below, such as a misspelled one or
`PIXA_PORT`. The error names the variable. The variables set by the file's
`env:` section are checked the same way.

pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`.

pixa reads at most one config file. It reads the file given with `--config` (or
`-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
file does not exist, cannot be read or does not parse. Without `--config` or
`PIXA_CONFIG_PATH`, pixa looks in `/etc/pixa/`, then in `~/.config/pixa/`, then
in the working directory. In each directory it looks for `config.yml`, then
`config.yaml`. It reads the first file it finds. It skips a missing file without
a message. pixa refuses to start when a file it finds cannot be read or does not
parse, and the error names the file. It also refuses to start when it may not
enter one of these directories, whether or not the file is there, since it
cannot tell. With no file, pixa uses the environment and the defaults.

Durations are written as Go durations, such as `30s` or `2m`, and must be
positive. A bare number has no unit and aborts startup. Sizes are a whole number
of bytes. A CIDR range is an address range written as an address and a prefix
length, such as `10.0.0.0/8`. An address matches a range that contains it.

`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain
IPv4 address. A range written in that form is read as the IPv4 range it covers:
`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers
more than IPv4 addresses and aborts startup.

Each item below is one setting, named by its config key:

- `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed
  URLs, encrypted URLs and login. It must be 32 characters or longer. The
  placeholder in `configs/config.example.yml` is refused. pixa uses the key as
  it is for two things: as the HMAC key of signed URLs (see Signed URLs), and as
  the password of the login page. From it, pixa derives four more keys at
  startup, with HKDF-SHA256, a key derivation function. One signs the session
  cookie and one encrypts it. One encrypts the token of an encrypted URL. One
  signs the token cookie of the login and generator pages' forms.
- `port`: variable `PORT`, default `8080`. The port to listen on, from 1
  to 65535.
- `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory
  for the database and the disk cache. pixa creates it at startup when it is
  missing. It refuses to start when it cannot create or write the directory.
- `db_url`: variable `PIXA_DB_URL`, default
  `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
  database to open. The default keeps the database in WAL mode, SQLite's
  write-ahead log, which lets reads go on during a write. pixa adds
  `_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another
  program writing to the file then waits up to five seconds for it instead of
  failing. pixa opens one connection, so its own reads and writes run one at a
  time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
  one you set. An empty `db_url` aborts startup: omit the key to use the
  default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in
  the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one
  starting with `//`, the default starts with `file:////`, so SQLite does not
  read the first name in the path as a host.
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
  cache limit in bytes. When the key is omitted, pixa works the limit out at
  startup. It takes the free space on the filesystem holding
  `<state_dir>/cache/`, adds the bytes the cache already holds, and takes 75% of
  the sum. The default is at least 500 MiB. A value you set is used exactly as
  given. `0` turns the disk cache off: every request then fetches and converts
  its image, and only the negative cache is used. A negative value aborts
  startup.
- `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream
  hosts whose images pixa serves without a signature, written as under Source
  Hosts. Every image on a listed host is served to anyone, so list only hosts
  you control. An entry that is neither a host name nor an IP address aborts
  startup.
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
  whose pages may not show pixa's images, to stop other sites embedding them.
  Entries are written and matched as for `allowlist_hosts`. A request whose
  `Referer` header names a listed host is refused with 403. That applies to
  `/v1/image/` and `/v1/e/`. It happens before the signature or token is
  checked, and before the cache or the upstream host is used. The request is
  refused even when the image is cached. A request with no `Referer` is served,
  as many clients send none. So is one whose `Referer` does not parse as a URL
  with a host. So a site whose pages send no `Referer`, for example with
  `Referrer-Policy: no-referrer`, is not stopped. The login and generator pages
  are not covered.
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
  ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
  aborts startup. The built-in ranges always apply. They are:
    - `0.0.0.0/8`, this network, `0.0.0.0` included;
    - `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
    - `100.64.0.0/10`, carrier-grade NAT;
    - `127.0.0.0/8`, loopback;
    - `169.254.0.0/16`, link-local;
    - `192.0.0.0/24`, IETF protocol assignments;
    - `198.18.0.0/15`, benchmarking;
    - `224.0.0.0/4`, multicast;
    - `::`, the unspecified address;
    - `::1`, loopback;
    - `64:ff9b::/96` and `64:ff9b:1::/48`, NAT64;
    - `fc00::/7`, the unique local (private) range;
    - `fe80::/10`, link-local;
    - `ff00::/8`, multicast.
- `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
  private ranges: `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`. CIDR
  ranges of the reverse proxies in front of pixa. `X-Forwarded-For` is believed
  only when the request reaches pixa from an address in one of these ranges. The
  client address is then the rightmost entry of the header that is not itself a
  trusted proxy. Otherwise the header is ignored and the address the request
  came from is used. So a client connecting from outside these ranges cannot
  choose its address. An explicitly empty list (`[]`, or an empty variable)
  trusts no one. A list you set replaces the default. An invalid CIDR aborts
  startup. Set this to the address pixa sees for requests that come through your
  proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or
  to trust nothing else (see Limits). To find that address, set this to `[]`,
  send a request through the proxy, and read `remoteIP` in pixa's log line for
  it.
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
  fetches every source over plain HTTP, and follows redirects to plain HTTP. It
  is for testing only.
- `upstream_connections_per_host`: variable
  `PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
  one upstream host at once. It must be at least 1. A fetch that finds every
  connection to its host in use waits for one until `downstream_timeout` ends
  (see Limits).
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
  The most connections to all upstream hosts together at once, on top of the
  limit per host. It must be at least 1. A fetch holds its connection until its
  image has been processed (see Limits).
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
  default half the CPUs Go uses (`GOMAXPROCS`), rounded down, and at least 1.
  The most images decoded and encoded at once. `GOMAXPROCS` follows a
  container's CPU limit. A value that is not a whole number of at least 1 aborts
  startup. Each image can take a lot of memory (see Running under upaas).
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
  default `600`. How many requests one client may send per minute to
  `/v1/image/` and `/v1/e/` together (see Limits). `0` turns the limit off. A
  negative or non-integer value aborts startup.
- `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default
  `30s`. The time allowed for one fetch from an upstream host. A fetch whose
  response headers have not arrived by then answers 504.
- `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`,
  default `52428800` (50 MiB). The largest upstream response accepted, in bytes,
  from 1 to `1073741824` (1 GiB). It also limits the image data pixa decodes. A
  cached source larger than it is fetched again.
- `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The
  time allowed for answering one client request. Keep it longer than
  `upstream_fetch_timeout` plus 20 seconds (see Limits).
- `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`,
  default `*`. The origin a browser lets read the responses of `/v1/image/` and
  `/v1/e/`, sent as the CORS `Access-Control-Allow-Origin` header. No other
  route sends it. `*` is any site. Otherwise it is one `http` or `https` origin
  such as `https://example.com`. Its host is a lowercase host name (letters,
  digits, hyphens and dots, with a letter in its last part) or an IP address
  (IPv6 in brackets, in its shortest form). It may have a port from 1 to 65535
  that has no leading zero and is not the scheme's default. Any other value
  aborts startup, including an origin with another scheme, such as a browser
  extension's.
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
  username for `/metrics`. pixa serves `/metrics`, and measures every HTTP
  request, only when the username and the password are both set. Setting one
  without the other aborts startup.
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
  password for `/metrics`. Set it together with the username.
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. Sentry is an error
  reporting service, and the DSN is the URL it gives you. A DSN turns error
  reporting on. It must be a URL with a scheme and a host, or empty.
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
  plain-HTTP local development (see Routes).
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
  `true`, `/v1/image/` and `/v1/e/` answer every `GET` and `HEAD` request with
  503, a `Retry-After` header of 300 seconds and a JSON error body. The health
  check still answers 200 and reports `"maintenance_mode": true`. It stays 200
  because the image's Docker `HEALTHCHECK` requests it, and upaas marks a deploy
  failed when its container is unhealthy. The login and generator pages and
  `/metrics` keep working.

`configs/config.example.yml` names every setting with its default.

### Architecture

- **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the
  program's parts in dependency order and starts and stops them. `internal/app`
  puts every part of `pixad` in one `fx` graph. `internal/providers` builds the
  image cache, the image proxy and the other parts that serve images and logins.
  Each setting's default is declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above:
    - `handlers`: the routes, which read the image request from the URL, check
      the signed or encrypted URL, and make the login session
    - `imagerequest`: the image request, with the source key and the variant key
    - `imageproxy`: the image proxy
    - `imgcache`: the image cache, which keeps its sources in one `FileStore`
      and its variants in another
    - `httpfetcher`: fetching sources from upstream hosts
    - `magic`: reading an image's format from its first bytes
    - `imageprocessor`: making a variant from a source with libvips
    - `signature`: checking the `sig` of a signed URL
    - `encurl`: making and reading the tokens of encrypted URLs
    - `seal`: the keys derived from `signing_key`, and the encryption of those
      tokens
    - `session`: the login session
- **HTTP router**: `chi` (`github.com/go-chi/chi/v5`)
- **Image processing**: govips, a Go wrapper for libvips
- **Database**: SQLite via `modernc.org/sqlite`
- **Static assets**: embedded in the binary via `//go:embed`
- **Metrics**: Prometheus, at `/metrics`. Prometheus is a monitoring system that
  reads metrics from that route. pixa serves `/metrics` and measures HTTP
  requests only when `metrics.username` and `metrics.password` are set.
  `/metrics` holds:
    - the duration of HTTP requests
    - the size of HTTP responses
    - the HTTP requests in flight
    - the Go runtime and process metrics
    - `pixa_cache_hits_total`: image requests served from the cache
    - `pixa_cache_misses_total`: image requests not served from the cache
    - `pixa_upstream_fetches_total`: fetches of a source image that read any
      bytes
    - `pixa_upstream_fetch_bytes_total`: the bytes those fetches read
    - `pixa_transforms_total`: images converted

    A request's `handler` label is the pattern of the route it matches, such as
    `/v1/image/*`, never its path. Requests that match no route share the label
    `unmatched`: those the router answers 404 or 405, and CORS preflights. A
    request's `method` label is its method, for these nine: `GET`, `HEAD`,
    `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any
    other method is labelled `other`.

- **Logging**: the standard library's `slog`, as JSON on standard output, or as
  text when standard output is a terminal.

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for the development
workflow, and most `Makefile` targets are thin shims that call them. We provide:

- `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and
  download the Go modules. It can be run again at any time. With `--cgo`, it
  installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
  libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker,
  as `make build` and `script/bench` do, needs those. Docker itself must already
  be installed.
- `script/setup` — make a fresh clone ready for development: `script/bootstrap`,
  then `script/install-precommit`.
- `script/projectname` — output the project name (`pixa`).
- `script/test` — run the test suite: build the `test` phase of the
  `Dockerfile`, writing no image.
- `script/lint` — run `golangci-lint`: build the `lint` phase of the
  `Dockerfile`, writing no image. The linter never runs on the host.
- `script/fmt` — format the Go code with `gofmt` and the markdown with
  `prettier` (writes).
- `script/fmt-check` — check the same formatting (read-only), on the host.
- `script/check` — run `script/test`, `script/lint` and `script/fmt-check`.
- `script/docker` — build the Docker image tagged via `script/projectname`, with
  the version from `git describe`. The image's build stage depends on the `lint`
  and `test` phases, so this runs them too.
- `script/docker-smoke` — build the image, start it, and wait for it to be
  healthy.
- `script/loadtest` — measure `pixad`'s throughput, latency and peak memory. It
  is a benchmark, not part of `script/check` (see Load Test).
- `script/bench` — run the Go benchmark that serves cache hits in parallel
  through the image handler, with Go on the host after `script/bootstrap --cgo`.
  It is a benchmark, not part of `script/check` (see Load Test).
- `script/cibuild` — the CI entrypoint: run `script/bootstrap` (without
  `--cgo`), then `script/check`, then build the image as `script/docker` does.
- `script/precommit` — the pre-commit checks: the `go mod tidy` guard, then
  `script/check`.
- `script/install-precommit` — install the git pre-commit hook that runs
  `script/precommit`.

Every `docker build` in these scripts passes a new value of the `CACHEBUST`
build argument. Every stage of the `Dockerfile` declares it with an
`ARG CACHEBUST` line. So every `RUN` step after that line, the lint and test
steps included, runs on every build instead of coming from the build cache. The
steps above that line install system packages and download the Go modules.
Docker keeps their layers until the base image, the step itself, `go.mod` or
`go.sum` changes.

Go's build cache is kept from build to build: the steps that compile mount it,
so a package whose inputs are unchanged is not compiled again. The test step
runs `go test`, and reruns it with `-v` after a failure. Both runs pass
`-count=1`, so no stored test result is replayed. The lint step also mounts
`golangci-lint`'s cache. That cache reuses a package's findings only while its
files, its imports, `go.mod`, `.golangci.yml` and the `golangci-lint` version
are unchanged.

The `test` phase runs on the Debian Go image, with Debian's libvips and libheif.
The image pixa ships is built and runs on Alpine.

These scripts are the standard copies from `sneak/prompts`, kept identical to
them:

- `script/check`
- `script/cibuild`
- `script/docker`
- `script/lint`
- `script/test`
- `script/setup`
- `script/install-precommit`

`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
step kept before `prettier`. `prettier` formats the markdown only, not the HTML
templates, as it cannot parse a Go template action inside a tag.

## Load Test

`script/loadtest` (or `make loadtest`) measures how fast `pixad` answers and how
much memory it uses. It is a benchmark, not a check: `script/check` does not run
it. It needs Docker and Go.

```bash
script/loadtest           # 10 seconds per scenario, 4 clients
script/loadtest 30s 32    # 30 seconds per scenario, 32 clients
```

It builds the image with `script/docker`. It also builds the load tool,
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
starts a new `pixad` container and a new origin container,
`cmd/loadtest-origin`. The origin is an upstream host that answers every path
with the same generated 1600x1200 JPEG. vegeta then sends requests from the
given number of clients. Each client sends its next request as soon as its last
one is answered. Every request asks for an image resized to 400x300 WebP:

- `hit`: the same image every time, put in the cache first;
- `miss`: a new source image every time, so `pixad` fetches and converts each
  one;
- `herd`: every client asks for each new source image at the same moment. The
  clients then share one fetch and one conversion (see Limits).

`pixad` refuses upstream hosts with private or local addresses, so the
containers share a Docker network in `203.0.113.0/24`, a range set aside for
documentation. A second run on the same Docker host while one is going fails, as
it cannot create that network.

For each scenario the script prints vegeta's report and two lines of its own:

- `Requests [total, rate, throughput]`: the requests sent, how many were sent
  per second, and how many were answered successfully per second;
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
  to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th
  and 99th percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other
  numbers are not for the scenario described, such as `503` when `pixad` was
  busy;
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
- `pixad peak memory (VmHWM)`: the peak resident memory of the `pixad` process
  since its container started, in kB; for `hit` it includes the request that put
  the image in the cache;
- `requests to the origin`: the fetches `pixad` made. That is one for `hit`, one
  per request for `miss`, and one per image for `herd`: the requests sent
  divided by the number of clients.

The numbers depend on the machine and on whatever else runs on it.

`script/bench` (or `make bench`) measures cache hits alone, without the network
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
cached image through the `/v1/image/` handler from several goroutines at once,
one per CPU. A goroutine is a Go thread. The benchmark reports the hits served
per second as `hits/s`. Each run starts with an empty cache and an empty
database, a SQLite file of its own, and puts the image in the cache before it
measures. The benchmark fails if serving the hits ran any statement on the
database. Like the load test it is not part of `script/check`. It runs Go on the
host, as the load test does, and compiles pixa there. So it needs the C compiler
and libraries that `script/bootstrap --cgo` installs, but not Docker.

## TODO

See [`TODO.md`](TODO.md) for the full prioritized task list.

## License

GPL-3.0. See [`LICENSE`](LICENSE).

## Author

[@sneak](https://sneak.berlin)