pixa/README.md

# pixa

pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
proxies images from upstream sources, optionally resizing or transforming them,
and serves the results. Both source and transformed images are cached to disk so
that subsequent requests are served without origin fetches or additional
processing.

## Getting Started

```bash
# clone and build
git clone https://git.eeqj.de/sneak/pixa.git
cd pixa
make build

# run with a config file: copy the example and set a real signing key
# (the example placeholder is refused at startup), e.g. with
# openssl rand -base64 32
cp configs/config.example.yml config.yml
$EDITOR config.yml   # replace the signing_key placeholder
./bin/pixad --config config.yml

# or build and run via Docker
make docker
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
```

The example config allowlists no host (`allowlist_hosts`), so pixa serves an
image only for a signed or encrypted URL. Every image on an allowlisted host is
served to anyone without a signature, at any size and in any format, so
allowlist only hosts you control, where nobody else can put images.

A container takes its settings from environment variables (see Configuration
below for the list). Only `PIXA_SIGNING_KEY` is required; if it is unset the
container exits at startup naming the variable. Everything else has a built-in
default. A config file mounted at `/etc/pixa/config.yml` is optional: it is read
when present, and an environment variable wins over the same setting in it.

## Deployment

pixa listens on plain HTTP and runs behind a reverse proxy that terminates TLS.
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, chosen because
it is the smallest correct one: Caddy gets the TLS certificate itself and does
everything in this list without further settings. The reverse proxy must:

- terminate TLS, as the login and generator pages work only over HTTPS (see
  Routes);
- pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses
  a form from those pages unless `Origin` or `Referer` names the host in `Host`,
  builds encrypted URLs from `Host`, and checks `Referer` against
  `referer_blocklist`;
- set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to
  the address pixa sees the proxy's requests come from, so the login limit and
  `image_requests_per_minute` count each client by its own address (see
  `trusted_proxies` under Configuration);
- wait for pixa's answer for at least `downstream_timeout` (default `60s`), the
  longest pixa takes to fetch, convert and send an image.

It may also refuse `/metrics`, as the example does, so that only a scraper that
reaches pixa directly can read it; pixa itself asks for the metrics username and
password there.

pixa does the rest itself: it checks signatures and encrypted URLs, applies the
allowlist, refuses upstream hosts with private or local addresses, limits login
attempts, each client's image requests, upstream response size and image
dimensions, and sends the security headers, `Strict-Transport-Security`
included, with every response.

The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
database and the disk cache:

- It needs a persistent volume: without one, every restart starts with an empty
  cache. Losing the directory also ends every login session (see Encrypted
  URLs). In the container, the startup script gives the directory to the user
  pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
  must be able to write the directory.
- `cache_max_bytes` limits the source and transformed images together. The
  database and files still being written come on top, and eviction runs in the
  background, so the cache can pass the limit for a while: leave room on the
  volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each
  time pixa starts, is 75% of the sum of the space free on the volume and the
  space the cached images already take, so a restart keeps the limit the cache
  had, but anything else that fills or frees space on the volume moves it.

A load balancer's health check can request `/.well-known/healthcheck.json`,
which answers 200 whenever pixa is running, in maintenance mode too (see
`maintenance_mode`).

On SIGTERM or SIGINT pixa stops accepting connections. The requests in progress
and the images being processed get 5 seconds to finish. A request not finished
by then is cut off. pixa then writes the waiting last-used times of cached
images to the database (see Storage) and exits. It exits with 1 when:

- images were still being processed after those 5 seconds,
- some of those times could not be written, or
- another part of pixa failed to stop.

Otherwise it exits with 0. `docker stop` waits 10 seconds before it kills the
container.

Outside Docker, pixa needs libvips (the image has 8.16) and libheif to run, as
it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
support, which on Alpine is the `vips-jxl` package and which the nix and brew
packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04
on. Building pixa also needs the development files of libvips and libheif,
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these;
the `Dockerfile` installs its own packages. Plain `script/bootstrap`, which
`script/setup` and `script/cibuild` run, installs git, make and Go, and Node,
Yarn and the prettier pinned in `yarn.lock` for formatting the markdown, but
none of the C libraries: the checks compile pixa in Docker. Docker itself must
already be installed.

## Running under upaas

What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:

- **Port:** pixa listens on container port `8080`.
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its database and
  cache. Creating the host directory when it is missing is upaas's job, tracked
  in https://git.eeqj.de/sneak/upaas/issues/235.
- **Environment variables:**
    - `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs and
      login, 32+ characters, for example from `openssl rand -base64 32`
    - `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
      comma-separated; default none. Every image on a listed host is served to
      anyone without a signature, so list only hosts you control, where nobody
      else can put images.
    - `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it. Set
      it: the volume is a directory on the host's filesystem, shared with the
      host and every other app. The default is 75% of the sum of that
      filesystem's free space and what the cache holds, and at least 500 MiB, so
      the cache can grow to three quarters or more of the space they all have
      left, and to all of it when that is 500 MiB or less.
    - the rest are in the list under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
  the images it processes at once, plus `upstream_connections` times
  `upstream_max_response_size` for the source images it has fetched that wait
  for processing. 4.2 GiB is the most resident memory was measured to grow while
  pixa, with pixad's libvips settings, processed one 8192x8192 source, the
  largest it accepts, saving it at full size to each output format in turn, one
  at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
  an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
  JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
  WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
  times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
  GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
  `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
  use.
- **Temporary files:** while pixa processes a large source, libvips writes its
  decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
  written, so a listing of `/tmp` does not show it, but it keeps its space, on
  disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
  GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
  well as written to that file, and the 4.2 GiB above includes that memory. For
  an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
  JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
  32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
  processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
  This `/tmp` is in the container's own filesystem, not on the volume at
  `/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
  does not include these files; where `/tmp` is a tmpfs, they are memory on top
  of it.
- **Health check:** the image's `HEALTHCHECK` requests
  `/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
  after a deploy and marks the deploy failed unless it is `healthy`. The probe
  uses the port from `PORT` (default `8080`), so a port changed only in a
  mounted config file is not seen by it: change the port with `PORT`.

## Rationale

Image-heavy web applications need a fast, caching reverse proxy that can resize
and transcode images on the fly. pixa fills that role as a single,
self-contained binary with no external runtime dependencies beyond libvips. It
supports HMAC-SHA256 signed URLs with expiration to prevent abuse, and
allowlisted source hosts for open access.

## Design

### Concepts

These terms are the names the code uses:

- **Source**: the image at one upstream URL, named by its host, path and query.
- **Source key**: the SHA-256 of a source's host, path and query, which names
  the source's file and its row in the database.
- **Transform**: the output size, format, quality and fit an image request asks
  for; with the format `auto`, pixa chooses the output format from the `Accept`
  header.
- **Image request**: a source and a transform, which each image route reads from
  its URL.
- **Variant**: the image pixa serves for an image request, made from its source
  with its transform.
- **Variant key**: the SHA-256 of a variant's source, size, format, quality and
  fit, which names the variant's file and its row in the database, and gives its
  `ETag`.
- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its
  expiry `exp` with `signing_key`.
- **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any
  expiry, encrypted with a key derived from `signing_key`.
- **Login session**: an encrypted cookie that keeps a browser logged in to the
  generator page for 30 days, or until a logout replaces the random value pixa
  keeps in the `logout` table (see Encrypted URLs).
- **Negative cache**: the source URLs pixa does not fetch again for 5 minutes,
  after one of these failed fetches:
    - the upstream answered with an error status
    - the body's first bytes were not those of the format its `Content-Type`
      names
- **Image cache**: the part of pixa that keeps the sources, the variants and the
  negative cache (see Storage).
- **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached
  sources and variants may reach together.
- **Image proxy**: the part of pixa that serves a variant from the image cache
  when it can, and otherwise makes it from the source.

### Storage

- **Sources**: `<state_dir>/cache/sources/<ab>/<cd>/<source key>`, one file for
  each source URL
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
- **Variants**: `<state_dir>/cache/variants/<ab>/<cd>/<variant key>`

`<ab>` and `<cd>` are the first and second pairs of characters of the file's
name.

The image cache keeps these tables in the database:

- `source_content`: a row for each source, by source key, with its size and when
  it was last used
- `variant_content`: a row for each variant, by variant key (column
  `cache_key`), with its size and when it was last used
- `cache_usage`: the total size of the cached sources and variants, which
  eviction reads
- `negative_cache`: a row for each source URL in the negative cache, with when
  it expires

pixa reads a cached source image's format from its file's first bytes. When they
match none of the source formats, pixa fetches the source again. Two source URLs
that serve the same bytes are stored twice, once under each URL. A source image
is stored by writing its file and then its row; when the row cannot be written,
the file is removed again.

Each source and variant file is written under a temporary name and synced to
disk before it is renamed into place. So a crash never leaves part of a file
under its own name. When pixa starts, and every five minutes after, it removes
each source or variant whose file is empty, or not the size its row records. It
logs a warning naming the file. A later request for that image fetches the
source or makes the variant again. Looking for those files reads the database
once for every cached file, so pixa skips each five-minute look that would start
less than ten times as long after the last look ended as that look took. After a
look that took one minute, the next one starts 10 to 15 minutes after that look
ended.

Each look also drops the rows of variants whose files are missing. When pixa
cannot read a variant file's size for another reason, it keeps the variant's
row. It logs a warning naming the variant key and the error.

During a look, pixa skips a directory it cannot list and a file whose size it
cannot read. It logs a warning naming the path and the error, and goes on with
the rest of the look. A file renamed or removed after pixa listed its directory
gets no warning.

pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, a cache hit opens the variant's file and nothing else, and
makes no database call. The content type comes from the file's first bytes,
which are those of the format pixa wrote it in; a file whose first bytes are not
those of a format pixa writes is not served, and the image is converted again. A
hit hands the time it was served to a background writer and does not wait for
the writer. Eviction orders by that time. The hit is counted in memory. A miss
that uses a cached source image also hands that writer the time it used the
image. Once a second that writer writes the waiting times in one transaction. Up
to 10,000 access times of variants, and as many of sources, can wait to be
written. A time handed over beyond that is dropped. Its image then only looks
less recently used to eviction. `script/bench` measures how many hits pixa
serves per second (see Load Test).

### Routes

pixa answers these routes; any other path answers 404. A path in this list asked
with a method the list does not give answers 405, except `/static/<file>`, which
answers any method as it answers `GET`. A browser's CORS preflight request
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
path under `/v1/` answers 200, in maintenance mode too.

- `GET /` — the login page, or the URL generator page with a login session (see
  Encrypted URLs). Needs: nothing. Answers: 200.
- `POST /` — log in with the signing key typed into the login page. Needs: the
  login page's form (below). Answers: 303 to `/` with a login session cookie
  that lasts 30 days for the right key; 200 with the login page and an error for
  a wrong key; 429 over the login limit (below).
- `POST /generate` — make an encrypted URL from the generator page's form.
  Needs: a login session and the generator page's form (below); without a login
  session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
  with the page naming a field that is not valid; 500 when the URL cannot be
  made.
- `POST /logout` — log out with the generator page's Logout button: end every
  login session, in every browser (see Encrypted URLs), and remove the session
  cookie. Needs: a login session and the generator page's form (below); without
  a login session it only removes the cookie. Answers: 303 to `/`; 500 when the
  logout cannot be saved in the database.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
  `/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
  and converted (below). Needs: a signature, unless the host is allowlisted (see
  Source Hosts). Answers:
    - 200 with the image.
    - 304 when `If-None-Match` matches the image's `ETag`.
    - 400 for a URL or parameter that is not valid.
    - 400 when the format is `auto` and the `Accept` header is not valid.
    - 406 when the format is `auto` and `Accept` allows none of the formats it
      chooses from.
    - 401 for a missing or wrong signature.
    - 401 for a missing `exp`.
    - 401 for an `exp` in the past.
    - 403 when the request's `Referer` names a host in `referer_blocklist`,
      checked before the signature, the cache and the upstream fetch.
    - 403 when the upstream host, or a host it redirects to, is `localhost` or
      ends in `.localhost` or `.local`.
    - 403 when the upstream host, or a host it redirects to, has an address in a
      blocked network (see `blocked_networks`).
    - 429 over `image_requests_per_minute` (below).
    - 499, with no body, when the client goes away before its image is ready,
      unless its request is the one making the image (below).
    - 415 when the upstream answers with a `Content-Type` that is not one of the
      source formats (below).
    - 415 when the upstream sends a body whose first bytes are not those of the
      format its `Content-Type` names.
    - 415 when the upstream sends an image whose first bytes libvips reads as
      another format.
    - 502 when the upstream answers with an error status.
    - 502 for 5 minutes after that for the same source URL, when pixa has
      neither the image nor its source image in its cache.
    - 502 likewise for 5 minutes after a body whose first bytes are not those of
      its `Content-Type`.
    - 503 when pixa is busy.
    - 503 when pixa is in maintenance mode.
    - 504 when the upstream has not sent its response headers within
      `upstream_fetch_timeout`.
    - 500 when that time runs out while the image itself is still arriving.
    - 500 for any other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
  (see Encrypted URLs). Needs: nothing but the URL. Answers:
    - 200 with the image.
    - 304 when `If-None-Match` matches the image's `ETag`.
    - 400 for a token that does not decrypt.
    - 400 for a token that asks for a size or fit that is not valid.
    - 410 once the token has expired.
    - 400 when the token's format is `auto` and the `Accept` header is not
      valid.
    - These answers, for the same causes as on `/v1/image/`:
        - 406
        - 403
        - 415
        - 429
        - 499, with no body
        - 502
        - 503
        - 504
        - 500
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
  nothing. Answers: 200.
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
  `uptime_seconds`, `uptime_human`, `version`, `appname` and `maintenance_mode`.
  Needs: nothing. Answers: 200, always.
- `GET /static/<file>` — the stylesheet and script the login and generator pages
  load. Needs: nothing. Answers: 200, or 404 for a file that does not exist.
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
  authentication with `metrics.username` and `metrics.password`. Answers: 200;
  401 without them; 404 when they are not set, as the route then does not exist.

Every response carries an `X-Request-ID` header holding the request's ID, which
a client can quote when reporting a problem: the request's own `X-Request-ID`,
as a reverse proxy in front of pixa may send, when it is at most 64 letters,
digits, `-`, `_` or `.`; otherwise a random one pixa makes for the request,
which tells nothing about the machine or the other requests. pixa's log line for
the request carries the same ID as `request_id`, and so do the lines it logs
when it fetches, converts and serves an image; the fetch sends it to the
upstream host as `X-Request-ID`.

The `POST` routes accept only a form that pixa's own page served: the page puts
a token in the form and sets a cookie to match, and a request without both is
refused with 403, so another site cannot submit the form from a visitor's
browser. The login and generator pages are meant to be opened over HTTPS: while
`debug` is off, a form sent from a page opened over plain HTTP is refused with
403, and while it is on, so is one sent from a page opened over HTTPS. Plain
HTTP is for development on the browser's own machine: the login session cookie
is always marked `Secure`, and over plain HTTP a browser keeps such a cookie
only for its own machine (`localhost`), if at all. A form is also refused with
403 when the page's host is not the `Host` header pixa receives, so a reverse
proxy in front of pixa must pass that header on unchanged. A form body over 1
MiB is refused with 413. The image routes answer the errors listed for them with
JSON holding `error`, `status` and `timestamp`.

An image URL has one of these forms, the second with no format:

```
/v1/image/<host>/<path>/<size>.<format>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
/v1/image/<host>/<path>/<size>?sig=<signature>&exp=<expiration>&q=<quality>&fit=<fit>
```

Images are only fetched from origins using TLS with valid certificates, unless
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
testing only.

A request whose query string cannot be decoded, or gives any parameter more than
once, is refused with 400.

- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
  `avif`, `jxl` (JPEG XL), `gif`, or `auto` (below). A URL with no format (the
  second form, with no dot after the size) is served as JPEG XL, the default, as
  with `jxl`
- `<size>`: `orig` or `<width>x<height>` (e.g. `800x600`)
- `sig` and `exp`: the signature and its expiry, needed unless the host is
  allowlisted (see Signature Specification)
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
  both optional (values under Signature Specification). Both are part of what is
  cached, so each value of either is a separate cached image.

A source URL's query follows its path after `%3F`, the encoded `?`:
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
percent-escapes in the image URL's path once, then reads the host up to the
first `/`, the size and format after the last `/`, and between them the source
path up to the first `?` and the source query after it. So the `?` before the
query, any `?` in the query, and every `%` in the source path or query must be
percent-encoded, as `%3F` and `%25`. Any other character may be written as
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
the source path nor the source query may hold a `#`, and an image URL that
writes one as `%23` is refused with 400. For example,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 WebP is
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
signed as shown under Signature Specification.

The source path is the path with its percent-escapes decoded, and pixa fetches
it percent-encoded again: every character other than letters, digits, `-`, `.`,
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
percent-escape. So `my%20cat.jpg` in the image URL is the source path
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
`/100%25.png`. The source query is the source URL's query as that URL writes it,
its percent-escapes included, and is sent as it is, except that the characters a
browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control
characters and characters outside ASCII) and a `%` that does not begin a
percent-escape are percent-encoded. pixa signs and caches the query in that
form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are
the same source query, `name=my%20cat`.

A size with one side `0` has that side scaled to keep the source's proportions,
and is refused with 400 when that side would come out above 8192, the most
either side may be; so is a size with `fit=outside`, which scales the image to
cover the size and so past it on one side, when that side would come out
above 8192.

An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
have, is refused with 400 before anything is decoded, as its encode could take
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
not `image/jxl` is refused such a size.

The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
`Content-Type` must name its format, and the image's first bytes must match it.
An upstream answering with any other `Content-Type`, SVG included, is refused
with 415. So is a body whose first bytes are not those of the format its
`Content-Type` names, such as a JPEG sent as `image/png`, text sent as an image,
or an empty body. pixa then does not fetch that source URL again for 5 minutes,
as after an upstream error status, and meanwhile answers 502 for an image that
would need that fetch. An image whose first bytes libvips reads as another
format, such as an SVG whose first bytes pass the check for AVIF, is refused
with 415 too.

With the format `auto`, pixa chooses the format for each request from its
`Accept` header, in this order:

1. JPEG XL, when the header names `image/jxl`;
2. AVIF, when it names `image/avif`;
3. WebP, when it names `image/webp`;
4. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names
   allows it, or when there is no `Accept` header or it is empty.

An entry with `q=0` refuses its format; other `q` values do not change the
order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them
also send `image/*` and `*/*`. pixa never sends a format the client refused:
when the header allows none of the four, the answer is 406, and a header that
does not parse, or has a `q` that is not a number from 0 to 1, is refused
with 400. The signature, or the token of an encrypted URL, covers `auto` itself,
so one URL serves every client. Each format chosen is cached as a separate
image, and every answer that depends on `Accept` (the image, a 304, and the 400
and 406 above) carries `Vary: Accept`, so a shared cache keeps the formats apart
too.

An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
is the whole seconds left until then, at most one year, so no browser or proxy
cache keeps the image after pixa would refuse the URL. A URL with no expiry gets
one year. `immutable` only stops a client revalidating while its copy is fresh.

When several requests for the same image, size, format, quality and fit miss the
cache at once, they share one upstream fetch (or one read of the cached source)
and one transcode: the first request does the work, and the others wait for its
image or its error, holding no upstream connection or processing slot of their
own. A waiting request stops waiting when its own client goes away. The work
goes on for the others even if the first request's client goes away. The shared
fetch sends the first request's ID upstream, and the lines logged for the fetch
and the transcode carry that ID.

A transcode whose request's `downstream_timeout` has ended (for a shared one,
the first request's) stops waiting for a processing slot, or, once it has one,
gives it back without beginning to encode. A request waiting for a shared
transcode that stops this way is answered 503 with the error
`server busy, try again later`, though its own `downstream_timeout` may not have
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
encode keeps its slot until it finishes, which can be after
`downstream_timeout`. The limit of 8192 on each side of the source and of the
output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
encoders run below libvips' default effort, and AVIF at the lowest effort govips
can set, so that no encode takes more than 40 seconds of the default
`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
8192x8192 random pixels with an alpha channel take up to about 23 seconds as
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
up to about 11 seconds.

The login form (`POST /`) is limited to 5 attempts per minute per client
address, counting an IPv6 client by its /64; an attempt over the limit is
refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
address comes from `X-Forwarded-For` only when the address pixa sees for
requests that come through the proxy is in `trusted_proxies`; otherwise all
users behind the proxy are counted as one client. That address is not always the
proxy's own: a proxy on the Docker host that connects to pixa over `127.0.0.1`
is seen as the gateway of the container's Docker network, such as `172.17.0.1`
on the default bridge, and one that connects through another of the host's
addresses is seen with that address. To be sure, read it as `remoteIP` in pixa's
request log while it is not in `trusted_proxies` (see `trusted_proxies` under
Configuration). With the default `trusted_proxies` (the RFC 1918 ranges), a
client with a private address can choose the address it is counted by through
its own `X-Forwarded-For`, whether it connects directly or through the proxy,
because its own address is trusted too. Setting `trusted_proxies` to only the
address pixa sees for requests that come through the proxy closes this.

One client may send `image_requests_per_minute` requests per minute
(default 600) to the image routes, `/v1/image/` and `/v1/e/` together, `GET` and
`HEAD`, cache hits included; a request over the limit is refused with 429 and a
`Retry-After` header. Clients are counted as for the login limit. A browser's
CORS preflight is not counted. Behind a CDN or proxy whose addresses are not in
`trusted_proxies`, all requests arriving from one proxy or CDN address count as
one client, so such a deployment needs a higher limit, or `0`, which turns the
limit off.

### Encrypted URLs

An encrypted URL is an image URL made on pixa's own web page by someone who
knows the signing key. It works for any upstream host, allowlisted or not,
without a signature, and whoever gets it can neither read the source URL from it
nor change what it asks for.

1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
   Routes) and log in with the signing key (`signing_key`). The login session
   lasts 30 days, or until someone logs out with the generator page's Logout
   button. Every login is made with the one signing key, so logging out ends
   every login session, in every browser, a copy of the session cookie included.
   pixa keeps a random value in its database, which it makes when the database
   has none and replaces at each logout. A login session carries the value
   current at its login and is refused once the value has changed, also after a
   restart (see Deployment). The check for a logout compares no time, so a login
   after a logout works whatever the host's clock says; the 30 days are still
   measured by the host's clock.
2. On the generator page, give the source image's URL, the width and height, the
   format, quality and fit, and how long the URL lasts, then submit the form
   (`POST /generate`). The format is JPEG XL unless another is chosen; a form
   sent with an empty format, or none, also makes a JPEG XL URL. Width and
   height both empty or `0` keep the original size; if only one of them is empty
   or `0`, that side is scaled to keep the image's proportions. A URL whose
   scaled side would come out above 8192 answers 400, and so does an AVIF URL of
   more than 16,777,216 pixels or a source URL whose path holds a `..` segment.
   So does a source URL that the image URL would fetch written another way, and
   the page then shows the URL pixa would fetch: one with a user name and
   password, which pixa does not send, or one whose path holds `%2F`, fetched as
   `/`, or `(`, fetched as `%28`, as pixa keeps the source path decoded and
   fetches it percent-encoded again (see Routes).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
   it expires. `<host>` is the host the page was opened on, and the URL starts
   with `http` instead while `debug` is on. The name after the token is ignored
   and only gives the URL a file extension, `jpg` for `orig` and `auto`, and
   `jxl` for a form with no format.

The token holds the source's host, path and query and the size, format, quality,
fit and expiry, encrypted with a key derived from `signing_key`. The host is
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
fetched and cached once. The source URL's scheme is not kept: the image is
fetched like any other (see Routes), and the blocked networks still apply.

How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
The expiry is fixed in the token when the URL is made and cannot be changed or
revoked afterwards. Until then the image is served with a `max-age` that ends at
the expiry (see Routes); after it the URL answers 410 `URL has expired`. A URL
made to last forever stops working only when `signing_key` changes: changing it
makes every encrypted URL already handed out answer 400, and ends every login
session.

### Image Metadata

pixa decodes and re-encodes every image it serves, and removes all metadata from
the output: EXIF (GPS position, camera make, model and serial number, capture
time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be
turned off.

- The `orig` format means the source's own format, not the source's bytes: an
  `orig` image is re-encoded and stripped like any other.
- An image with an EXIF orientation is turned upright first, so it displays the
  same without the tag; a requested size applies to the upright image.
- An image with an ICC profile is converted to sRGB first, since clients show an
  image with no profile as sRGB. Colours outside sRGB, such as the most
  saturated ones in a Display P3 photo, are clipped.
- The one exception is JPEG XL with libvips 8.16 and later: libvips then writes
  an EXIF block of its own into the image, as govips cannot ask libvips to leave
  it out. It holds the image's size and otherwise fixed values, such as an
  orientation of 1 and a resolution of 72 dpi, and nothing from the source.

### Source Hosts

Source hosts may be allowlisted in the configuration. Non-allowlisted hosts
require an HMAC-SHA256 signature.

#### Signature Specification

Signatures use HMAC-SHA256 and include an expiration timestamp to prevent replay
attacks. Signatures are **exact match only**: every component (host, path,
query, dimensions, format, expiration, quality, fit) must match exactly what was
signed. No suffix matching, wildcard matching, or partial matching is supported.

**Signed data format** (colon-separated):

```
HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit")
```

Where:

- `host` — source host as the URL writes it up to the first `/`, in lower case,
  keeping an IPv6 address's brackets and any `:` and port after it (e.g.
  `cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
- `path` — source path, with its percent-escapes decoded as under Routes (e.g.
  `/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with
  each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is
  signed as `/photos/a%3Ab.jpg`
- `query` — source query string as under Routes, empty string if none, with `%`
  and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
  query and path `/a` with query `b:` are signed differently
- `width` — requested width in pixels, or `0`: the width is then worked out from
  the source's proportions and the requested height, rounded down but at least
  1, or is the source's own width when the height is `0` too (the `orig` size)
- `height` — requested height in pixels, or `0`: the height is then worked out
  from the source's proportions and the requested width, rounded down but at
  least 1, or is the source's own height when the width is `0` too (the `orig`
  size)
- `format` — output format, one of those listed under Routes, with `original`
  signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
  format has the signature of the same URL ending in `.jxl`; `auto` is signed as
  `auto`, not as the format chosen for the request
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when the
  signature expires; a request whose `exp` is not a whole number, an empty
  `exp=` included, is refused with 400
- `quality` — the URL's `q` query parameter, a whole number from 1 to 100, or
  `85` when the URL has no `q`; a request whose `q` is anything else is refused
  with 400
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
  outside), or `cover` when the URL has no `fit`; a request whose `fit` is
  anything else, an empty `fit=` included, is refused with 400. `outside` scales
  the image to cover the requested size, keeping its proportions, so it can come
  out larger than that size on one side: an 8192x1 source at `100x0`, which is
  100x1 by its proportions, comes out at 8192x1

The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
compares it exactly, so a signature encoded without padding, as Node's
`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401.

**Example:** with the signing key `example-signing-key-for-documentation`,
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with expiration
1704067200, default quality and fit:

1. Build input:
   `cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
2. Compute HMAC-SHA256 of it with the signing key
3. Base64URL-encode the result, keeping the `=` padding:
   `-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=`
4. URL:
   `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=&exp=1704067200`

For the same image at quality 40 with fit `contain`, the input ends in
`:40:contain`, the signature is `5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=`,
and the URL is
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.

For a source URL with a query,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
800x600 WebP with the same key and expiration and the default quality and fit:

1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
   `cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
   `%25` (see Routes):
   `/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`

**Allowlist patterns:**

- **Exact match**: `cdn.example.com` — matches only that host
- **Suffix match**: `.example.com` — matches `cdn.example.com`,
  `images.example.com`, and `example.com`

An IP address is matched exactly; write an IPv6 address without brackets. An
entry that is neither a host name (letters, digits, hyphens, underscores and
dots, with at most one leading dot) nor an IP address, such as one with a port
or a `*.` wildcard, aborts startup.

pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
`CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
even when the host is allowlisted. The host must be a host name or an IPv4
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
brackets, either one optionally followed by `:` and a port. Any other host, such
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
refused with 400.

### Configuration

Every setting can be given as an environment variable, in a YAML config file
(`--config`), or both. A variable present in the environment wins over the file,
even when it is empty, and the file wins over the built-in default. The one
exception is a variable set in the file's `env:` section. pixa sets that
variable while loading the file, so the variable wins over the environment pixa
started with. It also wins over the file's key for the same setting. A
variable's value is parsed as the same text in the file would be. A setting that
holds a list takes comma-separated entries, with the spaces around each trimmed.
An empty variable is an empty list. A value that does not parse or is invalid
aborts startup, naming the variable. pixa refuses to start when a variable
starts with `PIXA_` but is not in the list below, such as a misspelled one or
`PIXA_PORT`. The error names the variable, as it names an unknown config key.
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`. The variables set by the file's `env:` section are checked the same
way.

pixa reads at most one config file. It reads the file given with `--config` (or
`-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that
file does not exist, cannot be read or does not parse. Without `--config` or
`PIXA_CONFIG_PATH`, pixa looks in `/etc/pixa/`, then in `~/.config/pixa/`, then
in the working directory. In each directory it looks for `config.yml`, then
`config.yaml`. It reads the first file it finds. It skips a missing file without
a message. pixa refuses to start when a file it finds cannot be read or does not
parse, and the error names the file. It also refuses to start when it may not
enter one of these directories, whether or not the file is there, since it
cannot tell. With no file, pixa uses the environment and the defaults.

Each item below is one setting, named by its config key:

- `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed
  URLs, encrypted URLs and login. It must be 32 characters or longer.
- `port`: variable `PORT`, default `8080`. The port to listen on.
- `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory
  for the database and the disk cache.
- `db_url`: variable `PIXA_DB_URL`, default `state.sqlite3` in the state
  directory. The URL of the SQLite database.
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default 75% of the sum of
  the free space and what the cache holds. The disk cache limit in bytes. `0`
  turns the disk cache off.
- `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream
  hosts whose images pixa serves without a signature.
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. The
  image routes refuse a request with 403 when its `Referer` header names one of
  these hosts.
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. The CIDR
  ranges pixa never fetches from, on top of the built-in ones.
- `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918
  ranges. The CIDR ranges of the proxies whose `X-Forwarded-For` header pixa
  believes.
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. Allows plain-HTTP
  upstream hosts. It is for testing only.
- `upstream_connections_per_host`: variable
  `PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to
  one upstream host at once.
- `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`.
  The most connections to all upstream hosts together at once.
- `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`,
  default half the CPUs, at least 1. The most images processed at once.
- `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`,
  default `600`. How many image requests one client may send per minute. `0`
  turns the limit off.
- `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default
  `30s`. The time allowed for one fetch from an upstream host.
- `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`,
  default 50 MiB. The largest upstream response accepted, in bytes.
- `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The
  time allowed for answering one client request.
- `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`,
  default `*`. The CORS origin allowed to read image responses. It is `*` or one
  origin.
- `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The
  username for `/metrics`. pixa serves `/metrics` only when the username and the
  password are both set.
- `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The
  password for `/metrics`. Set it together with the username.
- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. The Sentry DSN for
  error reporting. An empty DSN turns error reporting off.
- `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and
  plain-HTTP local development.
- `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While
  `true`, pixa answers image requests with 503. The health check still
  answers 200.

Key settings in more detail:

- `access_control_allow_origin` — the origin a browser lets read the responses
  of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
  `Access-Control-Allow-Origin` header; no other route sends it. `*`, the
  default, is any site; otherwise one `http` or `https` origin such as
  `https://example.com`, whose host is a lowercase host name (letters, digits,
  hyphens and dots, with a letter in its last part) or an IP address (IPv6 in
  brackets, in its shortest form), with an optional port 1-65535 that has no
  leading zero and is not the scheme's default. Any other value, including
  another scheme such as a browser extension's, aborts startup
- `allowlist_hosts` — list of allowed upstream hosts
- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
  stop other sites hotlinking them. Entries are written and matched as for
  `allowlist_hosts` (see Allowlist patterns), and an entry that is neither a
  host name nor an IP address aborts startup. A request to `/v1/image/` or
  `/v1/e/` whose `Referer` header names a listed host is refused with 403 before
  its signature or token is checked and before the cache or the upstream host is
  used, so it fetches nothing, and it is refused even when the image is cached.
  A request with no `Referer`, or one that does not parse as a URL with a host,
  is served, as many clients send none. So this is easily got around: a site
  whose pages send no `Referer` (for example with
  `Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
  and generator pages. Default: empty
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
  to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
  benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6
  address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
  address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
  plain IPv4 address. An entry written in that form is read as the IPv4 range it
  covers, so `::ffff:10.0.0.0/104` refuses what `10.0.0.0/8` refuses; such an
  entry shorter than `/96` covers more than IPv4 addresses and aborts startup
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
  pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
  of these ranges; the logged and login-recorded client address is then the
  rightmost forwarded entry that is not itself a trusted proxy. Otherwise the
  direct peer address is used and the header is ignored, so a client connecting
  directly from an address outside these ranges cannot spoof its address. An
  omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
  `172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a
  private network; an explicitly empty list (`[]`) trusts no one, and an
  explicit list replaces the default. An invalid CIDR aborts startup. An IPv6
  address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
  address it holds, so an IPv6 range such as `::/0` trusts neither it nor a
  plain IPv4 address. An entry written in that form is read as the IPv4 range it
  covers, so `::ffff:10.0.0.0/104` trusts what `10.0.0.0/8` trusts; such an
  entry shorter than `/96` covers more than IPv4 addresses and aborts startup.
  Set this to the address pixa sees for requests that come through your proxy,
  such as `172.17.0.1/32`, when the defaults do not cover it, or to trust
  nothing else (see the login limit and `image_requests_per_minute` under
  Routes). For a proxy on the Docker host that connects to pixa over
  `127.0.0.1`, that address is the gateway of the container's Docker network
  (`172.17.0.1` on the default bridge), not the proxy's own address; a proxy
  that connects through another of the host's addresses is seen with that
  address. To be sure which address it is, set this to `[]` (or
  `PIXA_TRUSTED_PROXIES` to empty), send a request through the proxy, and read
  `remoteIP` in pixa's request log line for it
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
  as a duration such as `30s` (the default) or `2m`
- `upstream_max_response_size` — largest upstream response accepted, in bytes;
  default `52428800` (50 MiB). It also limits the image data pixa decodes
- `downstream_timeout` — time allowed for answering one client request, as a
  duration; default `60s`. The upstream fetch counts toward it, and so do the
  waits for an upstream connection and for a processing slot (up to 10 seconds
  each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
- `signing_key` — the secret that protects signed URLs, encrypted URLs and
  logins. pixa uses it as it is for two things:
    - the HMAC key of signed URLs (see Signature Specification)
    - the password of the login page

    pixa derives every other key it uses from it at startup, with HKDF-SHA256:
    - a key that signs the session cookie
    - a key that encrypts the session cookie
    - a key that encrypts the token of an encrypted URL
    - a key that signs the CSRF token cookie of the login and generator pages

- `db_url` — the SQLite database to open; omitted, it is
  `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
  database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
  `#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`
  or one starting with `//` it starts `file:////`, so SQLite does not read the
  first name in the path as a host. pixa opens one connection to it, so its own
  reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to
  any `db_url`, so a write that finds another program writing to the file waits
  up to five seconds for it instead of failing. WAL mode comes only from the
  URL: keep `_pragma=journal_mode(WAL)` in one you set
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
  cache entirely; omitted defaults to 75% of the sum of the free space on the
  filesystem containing `<state_dir>/cache/` and the bytes of source and
  transformed images the cache already holds, worked out at startup (minimum 500
  MiB)
- `upstream_connections` — the most connections to upstream hosts at once, all
  hosts together, on top of `upstream_connections_per_host`; default `64`. A
  fetch holds its connection until its image has been processed. A fetch that
  finds all of them in use waits up to 10 seconds for one to free up; if none
  does, and `downstream_timeout` has not ended first, the request is answered
  503 with the error `server busy, try again later`
- `max_concurrent_processing` — the most images decoded and encoded at once;
  default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's
  CPU limit), rounded down, and at least 1. A value that is not a whole number
  of at least 1 aborts startup. A request that finds all of them in use waits up
  to 10 seconds for one to free up; if none does, and `downstream_timeout` has
  not ended first, it is answered 503 the same way. Each one can take a lot of
  memory: see "Running under upaas"
- `image_requests_per_minute` — how many requests one client may send to the
  image routes per minute (see the limit on them under Routes); default `600`.
  `0` turns the limit off, and a negative or non-integer value aborts startup.
  Behind a CDN or proxy whose addresses are not in `trusted_proxies`, all
  requests arriving from one proxy or CDN address count as one client, so raise
  the limit there, or set `0`
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
  `/v1/e/`) answer every request for an image with 503, a `Retry-After` header
  and a JSON error body. The health check (`/.well-known/healthcheck.json`)
  still answers 200 and reports `"maintenance_mode": true`. It stays 200 because
  the image's Docker `HEALTHCHECK` requests it: a 503 there would make the
  container unhealthy, and upaas marks a deploy failed when its container is
  unhealthy. The login and URL generator pages and `/metrics` keep working

See `configs/config.example.yml` for all options with defaults.

### Architecture

- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in
  one fx graph. `internal/providers` builds the image cache, the image proxy and
  the other parts that serve images and logins. Each setting's default is
  declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above:
    - `handlers`: the routes, which read the image request from the URL, check
      the signed or encrypted URL, and make the login session
    - `imagerequest`: the image request, with the source key and the variant key
    - `imageproxy`: the image proxy
    - `imgcache`: the image cache, which keeps its sources in one `FileStore`
      and its variants in another
    - `httpfetcher`: fetching sources from upstream hosts
    - `magic`: reading an image's format from its first bytes
    - `imageprocessor`: making a variant from a source with libvips
    - `signature`: checking the `sig` of a signed URL
    - `encurl`: making and reading the tokens of encrypted URLs
    - `seal`: the keys derived from `signing_key`, and the encryption of those
      tokens
    - `session`: the login session
- **HTTP router**: go-chi
- **Image processing**: govips (CGO wrapper for libvips)
- **Database**: SQLite via modernc.org/sqlite
- **Static assets**: embedded via `//go:embed`
- **Metrics**: Prometheus, at `/metrics`. pixa serves `/metrics` and measures
  HTTP requests only when `metrics.username` and `metrics.password` are set.
  `/metrics` holds:
    - the duration of HTTP requests
    - the size of HTTP responses
    - the HTTP requests in flight
    - the Go runtime and process metrics
    - `pixa_cache_hits_total`: image requests served from the cache
    - `pixa_cache_misses_total`: image requests not served from the cache
    - `pixa_upstream_fetches_total`: fetches of a source image that read any
      bytes
    - `pixa_upstream_fetch_bytes_total`: the bytes those fetches read
    - `pixa_transforms_total`: images converted

    A request's `handler` label is the pattern of the route it matches, such as
    `/v1/image/*`, never its path. Requests that match no route share the label
    `unmatched`: those the router answers 404 or 405, and CORS preflights. A
    request's `method` label is its method when that is `GET`, `HEAD`, `POST`,
    `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for
    any other method.

- **Logging**: stdlib slog

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. We
provide:

- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and
  download the Go modules (idempotent); with `--cgo`, the C compiler and the
  libvips (with its JPEG XL support) and libheif libraries that compiling and
  testing pixa need instead of Node, Yarn and prettier
- `script/setup` — make a fresh clone ready for development (bootstrap, then
  install-precommit)
- `script/projectname` — output the project name ("pixa")
- `script/test` — run the test suite: build the `test` phase of the
  `Dockerfile`, writing no image
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
  writing no image; the linter never runs on the host
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
  (writes)
- `script/fmt-check` — check the same formatting (read-only), on the host
- `script/check` — run test, lint, and fmt-check
- `script/docker` — build the Docker image tagged via `script/projectname`, with
  the version from `git describe`; the image's build stage depends on the `lint`
  and `test` phases, so this runs them too
- `script/docker-smoke` — build the image, start it, wait for it to be healthy
- `script/loadtest` — measure pixad's throughput, latency and peak memory; a
  benchmark, not part of `script/check` (see Load Test)
- `script/bench` — run the Go benchmark that serves cache hits in parallel
  through the image handler, with Go on the host after `script/bootstrap --cgo`;
  a benchmark, not part of `script/check` (see Load Test)
- `script/cibuild` — CI entrypoint: run `script/bootstrap` (without `--cgo`),
  then `script/check`, then build the image as `script/docker` does
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
  `script/check`)
- `script/install-precommit` — install the git pre-commit hook that runs
  `script/precommit`

Every `docker build` in these scripts passes a new value of the `CACHEBUST`
build argument, which every stage of the `Dockerfile` declares, so every `RUN`
step after that line, the lint and test steps included, runs on every build
instead of coming from the build cache. Above the line are the steps that
install system packages and download the Go modules; Docker keeps their layers
until the base image, the step itself, `go.mod` or `go.sum` changes. Go's build
cache is kept from build to build: the steps that compile mount it, so a package
whose inputs are unchanged is not compiled again, and both `go test` runs pass
`-count=1` so that no stored test result is replayed. The lint step also mounts
golangci-lint's cache, which reuses a package's findings only when its files,
those of the packages it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged. The `test`
phase runs on the Debian Go image, with Debian's libvips and libheif; the image
pixa ships is built and runs on Alpine. `script/check`, `script/cibuild`,
`script/docker`, `script/lint`, `script/test`, `script/setup` and
`script/install-precommit` are the standard copies from `sneak/prompts`, kept
identical to them. `script/fmt` and `script/fmt-check` are the standard copies
with pixa's `gofmt` step kept before prettier. prettier formats the markdown
only, not the HTML templates, as it cannot parse a Go template action inside a
tag.

## Load Test

`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how
much memory it uses. It is a benchmark, not a check: `script/check` does not run
it. It needs Docker and Go.

```bash
script/loadtest           # 10 seconds per scenario, 4 clients
script/loadtest 30s 32    # 30 seconds per scenario, 32 clients
```

It builds the image with `script/docker` and the load tool,
[vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario
starts a new pixad container and a new origin container, `cmd/loadtest-origin`:
an upstream host that answers every path with the same generated 1600x1200 JPEG.
vegeta then sends requests from the given number of clients, each sending its
next request as soon as its last one is answered, all for an image resized to
400x300 WebP:

- `hit`: the same image every time, put in the cache first;
- `miss`: a new source image every time, so pixad fetches and converts each one;
- `herd`: each new source image once per client in a row, so that all clients
  ask for it at the same time and share one fetch and one conversion (see
  Routes).

pixad refuses upstream hosts with private or local addresses, so the containers
share a Docker network in `203.0.113.0/24`, a range set aside for documentation.
A second run on the same Docker host while one is going fails, as it cannot
create that network.

For each scenario the script prints vegeta's report and two lines of its own:

- `Requests [total, rate, throughput]`: the requests sent, how many were sent
  per second, and how many were answered successfully per second; the last is
  the number to compare with the target under Storage;
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
  to the end of its answer; `50`, `95` and `99` are the 50th, 95th and 99th
  percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other
  numbers are not for the scenario described, such as `503` when pixad was busy;
- `Bytes In`: `0`, as vegeta is told not to keep the images it receives;
- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since
  its container started, in kB; for `hit` it includes the request that put the
  image in the cache;
- `requests to the origin`: the fetches pixad made: one for `hit`, one per
  request for `miss`, and one per image for `herd`, that is the requests sent
  divided by the number of clients.

The numbers depend on the machine and on whatever else runs on it. The first
measurement, made on a shared machine with few clients, is in `TODO.md`; it says
nothing about the target.

`script/bench` (or `make bench`) measures cache hits alone, without the network
and with no pixad process: a Go benchmark in `internal/handlers` serves one
cached image through the `/v1/image/` handler from as many goroutines at once as
the machine has CPUs, and reports the hits served per second as `hits/s`. Each
run starts with an empty cache and an empty database, a SQLite file of its own,
and puts the image in the cache before it measures. The benchmark fails if
serving the hits ran any statement on the database. Like the load test it is not
part of `script/check`. It runs Go on the host, as the load test does, and
compiles pixa there, so it needs the C compiler and libraries that
`script/bootstrap --cgo` installs, but not Docker.

## TODO

See [TODO.md](TODO.md) for the full prioritized task list.

## License

GPL-3.0. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)