From 8669b5cf04bf50950d1b830daf28fd99c752d366 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Tue, 27 Oct 2026 01:42:48 +0000 Subject: [PATCH] Rewrite README.md sentence by sentence (closes #327) Rewrites README.md from scratch against the code on next, for https://git.eeqj.de/sneak/pixa/issues/327: what pixa is and how to use it, deployment, design, the settings reference, then development and testing. The Configuration list keeps one item per setting, as TestReadmeNamesEverySetting reads it; the test is unchanged. The statements where the old text disagreed with the code are listed in the PR body. Model: fable-5-1 --- README.md | 1782 +++++++++++++++++++++++++++++------------------------ 1 file changed, 968 insertions(+), 814 deletions(-) diff --git a/README.md b/README.md index c6593a2..918ce9c 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,15 @@ # pixa pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that -proxies images from upstream sources, optionally resizing or transforming them, -and serves the results. Both source and transformed images are cached to disk so -that subsequent requests are served without origin fetches or additional -processing. +fetches images from other servers, resizes and converts them, and serves the +results. + +A web page asks pixa for an image at a size and in a format. pixa fetches the +original image, makes the requested image from it, and serves that. It keeps +both on disk. Later requests for the same image are served from disk, without a +fetch or a conversion. + +In this document, the upstream host is the server that holds an original image. ## Getting Started @@ -14,167 +19,200 @@ git clone https://git.eeqj.de/sneak/pixa.git cd pixa make build -# run with a config file: copy the example and set a real signing key -# (the example placeholder is refused at startup), e.g. with -# openssl rand -base64 32 +# make a config file from the example and put a real signing key in it cp configs/config.example.yml config.yml -$EDITOR config.yml # replace the signing_key placeholder +openssl rand -base64 32 # prints a new signing key +$EDITOR config.yml # replace the signing_key placeholder with it ./bin/pixad --config config.yml -# or build and run via Docker +# or build and run the Docker image make docker docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest ``` -The example config allowlists no host (`allowlist_hosts`), so pixa serves an -image only for a signed or encrypted URL. Every image on an allowlisted host is -served to anyone without a signature, at any size and in any format, so +`make build` compiles pixa on your machine into `bin/pixad`. `pixad` is the +server program. Compiling needs the libraries `script/bootstrap --cgo` installs +(see Deployment). + +The signing key is pixa's one required secret. It must be 32 characters or +longer. The placeholder in the example file is refused at startup. + +Once pixa runs, it serves images through URLs of this form: + +``` +/v1/image///. +``` + +For example, this URL serves `https://cdn.example.com/photos/cat.jpg` at 800x600 +as WebP: + +``` +/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp +``` + +A URL for a host in `allowlist_hosts` needs nothing more. A URL for any other +host needs a signature (see Signed URLs). Someone who knows the signing key can +also make an encrypted URL on pixa's own web page (see Encrypted URLs). An +encrypted URL works for any host and hides the original image's URL. + +The example config allowlists no host. Every image on an allowlisted host is +served to anyone, without a signature, at any size and in any format. So allowlist only hosts you control, where nobody else can put images. -A container takes its settings from environment variables (see Configuration -below for the list). Only `PIXA_SIGNING_KEY` is required; if it is unset the -container exits at startup naming the variable. Everything else has a built-in -default. A config file mounted at `/etc/pixa/config.yml` is optional: it is read -when present, and an environment variable wins over the same setting in it. +A container takes its settings from environment variables (see Configuration). +Only `PIXA_SIGNING_KEY` is required. Without it, the container exits at startup +and names the variable. Every other setting has a built-in default. A config +file mounted at `/etc/pixa/config.yml` is read when present. An environment +variable wins over the same setting in the file. ## Deployment -pixa listens on plain HTTP and runs behind a reverse proxy that terminates TLS. -[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, chosen because -it is the smallest correct one: Caddy gets the TLS certificate itself and does -everything in this list without further settings. The reverse proxy must: +pixa listens on plain HTTP. It runs behind a reverse proxy: a server in front of +it that receives the public requests and passes them on. The reverse proxy +terminates TLS, the encryption behind `https`. -- terminate TLS, as the login and generator pages work only over HTTPS (see - Routes); -- pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses - a form from those pages unless `Origin` or `Referer` names the host in `Host`, - builds encrypted URLs from `Host`, and checks `Referer` against - `referer_blocklist`; -- set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to - the address pixa sees the proxy's requests come from, so the login limit and - `image_requests_per_minute` count each client by its own address (see - `trusted_proxies` under Configuration); -- wait for pixa's answer for at least `downstream_timeout` (default `60s`), the - longest pixa takes to fetch, convert and send an image. +The reverse proxy must: -It may also refuse `/metrics`, as the example does, so that only a scraper that -reaches pixa directly can read it; pixa itself asks for the metrics username and -password there. +- terminate TLS. The login and generator pages work only over HTTPS (see + Routes). +- pass the `Host`, `Origin` and `Referer` headers on unchanged. pixa refuses a + form from its pages unless `Origin` or `Referer` names the host in `Host`. It + builds encrypted URLs from `Host`. It checks `Referer` against + `referer_blocklist`. +- set `X-Forwarded-For` to the client's address. `trusted_proxies` must hold the + proxy's address as pixa sees it. The login limit and + `image_requests_per_minute` then count each client by its own address. +- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That + is the longest pixa takes to fetch, convert and send an image. -pixa does the rest itself: it checks signatures and encrypted URLs, applies the -allowlist, refuses upstream hosts with private or local addresses, limits login -attempts, each client's image requests, upstream response size and image -dimensions, and sends the security headers, `Strict-Transport-Security` -included, with every response. +[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server. +Caddy does everything in this list without further settings, and gets its own +TLS certificate. + +The reverse proxy may also refuse `/metrics`, as the example does. Only a +scraper that reaches pixa directly can then read it. pixa itself asks for +`metrics.username` and `metrics.password` there. + +pixa does the rest itself: + +- it checks signatures and encrypted URLs; +- it applies the allowlist; +- it refuses upstream hosts with private or local addresses; +- it limits login attempts and each client's image requests; +- it limits the size of upstream responses and the dimensions of images; +- it sends the security headers, `Strict-Transport-Security` included, with + every response. The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the database and the disk cache: -- It needs a persistent volume: without one, every restart starts with an empty +- It needs a persistent volume. Without one, every restart starts with an empty cache. Losing the directory also ends every login session (see Encrypted - URLs). In the container, the startup script gives the directory to the user - pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user - must be able to write the directory. -- `cache_max_bytes` limits the source and transformed images together. The - database and files still being written come on top, and eviction runs in the - background, so the cache can pass the limit for a while: leave room on the - volume beyond it. -- Set `cache_max_bytes` for a lasting deployment. Its default, worked out each - time pixa starts, is 75% of the sum of the space free on the volume and the - space the cached images already take, so a restart keeps the limit the cache - had, but anything else that fills or frees space on the volume moves it. + URLs). +- In the container, the startup script gives the directory to the user pixa runs + as, uid 65532, and sets its mode to `750`. Outside the container, the user + running pixa must be able to write it. +- `cache_max_bytes` limits the cached images. The limit does not count the + database, or files still being written. Eviction runs in the background, so + the cache can pass the limit for a while. Leave room on the volume beyond it. +- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at + each start from the free space on the volume (see Configuration). Anything + else that fills or frees space on the volume moves that default. -A load balancer's health check can request `/.well-known/healthcheck.json`, -which answers 200 whenever pixa is running, in maintenance mode too (see +A load balancer's health check can request `/.well-known/healthcheck.json`. It +answers 200 whenever pixa is running, in maintenance mode too (see `maintenance_mode`). -On SIGTERM or SIGINT pixa stops accepting connections. The requests in progress -and the images being processed get 5 seconds to finish. A request not finished -by then is cut off. pixa then writes the waiting last-used times of cached -images to the database (see Storage) and exits. It exits with 1 when: +On `SIGTERM` or `SIGINT`, pixa stops accepting connections. The requests in +progress and the images being processed get 5 seconds to finish. A request not +finished by then is cut off. pixa then saves the last-used times it has not yet +written (see Storage), and exits. It exits with status 1 when: -- images were still being processed after those 5 seconds, -- some of those times could not be written, or +- images were still being processed after those 5 seconds; +- some of those times could not be saved; - another part of pixa failed to stop. Otherwise it exits with 0. `docker stop` waits 10 seconds before it kills the container. -Outside Docker, pixa needs libvips (the image has 8.16) and libheif to run, as -it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL -support, which on Alpine is the `vips-jxl` package and which the nix and brew -packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04 -on. Building pixa also needs the development files of libvips and libheif, -`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these; -the `Dockerfile` installs its own packages. Plain `script/bootstrap`, which -`script/setup` and `script/cibuild` run, installs git, make and Go, and Node, -Yarn and the prettier pinned in `yarn.lock` for formatting the markdown, but -none of the C libraries: the checks compile pixa in Docker. Docker itself must -already be installed. +Outside Docker, pixa needs these at run time: + +- libvips, the image library pixa converts images with, including its JPEG XL + support. pixa does not start without that support. On Alpine it is the + `vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do + the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on. +- libheif, which libvips needs for AVIF. +- the system's CA certificates, which HTTPS fetches need. + +Compiling pixa needs the development files of libvips and libheif, `pkg-config` +and a C compiler. `script/bootstrap --cgo` installs them, and libvips and +libheif with them. The `Dockerfile` installs its own packages. ## Running under upaas -What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs: +[upaas](https://git.eeqj.de/sneak/upaas) is the deployment tool pixa is meant to +run under. Its app for pixa needs: - **Port:** pixa listens on container port `8080`. - **Volume:** container path `/var/lib/pixa`, where pixa keeps its database and - cache. Creating the host directory when it is missing is upaas's job, tracked - in https://git.eeqj.de/sneak/upaas/issues/235. + cache. - **Environment variables:** - - `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs and - login, 32+ characters, for example from `openssl rand -base64 32` + - `PIXA_SIGNING_KEY` (required): the secret for signed URLs, encrypted URLs + and login. It is 32 characters or more, for example from + `openssl rand -base64 32`. - `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature, comma-separated; default none. Every image on a listed host is served to - anyone without a signature, so list only hosts you control, where nobody - else can put images. - - `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it. Set - it: the volume is a directory on the host's filesystem, shared with the - host and every other app. The default is 75% of the sum of that - filesystem's free space and what the cache holds, and at least 500 MiB, so - the cache can grow to three quarters or more of the space they all have - left, and to all of it when that is 500 MiB or less. - - the rest are in the list under Configuration below + anyone, so list only hosts you control, where nobody else can put images. + - `PIXA_CACHE_MAX_BYTES`: the disk cache limit in bytes; `0` turns the disk + cache off. Set it. The volume is a directory on the host's filesystem. The + host and every other app use that filesystem too. The default is 75% of + the sum of that filesystem's free space and what the cache already holds. + So by default the cache may grow to three quarters of the space the host + and the other apps have left. The default is also at least 500 MiB. So + when that space is 500 MiB or less, the cache may take all of it. + - the rest are in the list under Configuration. - **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for - the images it processes at once, plus `upstream_connections` times - `upstream_max_response_size` for the source images it has fetched that wait - for processing. 4.2 GiB is the most resident memory was measured to grow while - pixa, with pixad's libvips settings, processed one 8192x8192 source, the - largest it accepts, saving it at full size to each output format in turn, one - at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG, - an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point - JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as - WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64 - times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1 - GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and - `PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may - use. -- **Temporary files:** while pixa processes a large source, libvips writes its - decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is - written, so a listing of `/tmp` does not show it, but it keeps its space, on - disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a - GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as - well as written to that file, and the 4.2 GiB above includes that memory. For - an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB - JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of - 32-bit floating-point RGBA, the largest measured: up to 1 GiB per image - processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs. - This `/tmp` is in the container's own filesystem, not on the volume at - `/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above - does not include these files; where `/tmp` is a tmpfs, they are memory on top - of it. + the images it processes at once. On top of that, the fetched source images + waiting for processing can take `upstream_connections` times + `upstream_max_response_size`. 4.2 GiB is a measured value. It is the largest + rise in resident memory measured while pixa saved one 8192x8192 source, the + largest it accepts, to each output format at full size. The rise was largest + for a 16-bit JPEG XL of random pixels saved as WebP. At the defaults the total + is half the CPUs pixa sees times 4.2 GiB, plus 64 times 50 MiB. On a host + where pixa sees 48 CPUs, that is 24 times 4.2 GiB plus 3.1 GiB, about 104 GiB. + Set `PIXA_MAX_CONCURRENT_PROCESSING` and `PIXA_UPSTREAM_CONNECTIONS` so that + this fits the memory the container may use. +- **Temporary files:** while pixa processes a large source, libvips writes the + decoded pixels to a file in `/tmp`. The file is removed from the directory + once it is written, so a listing of `/tmp` does not show it. It keeps its + space until the image is done: on disk, or in memory where `/tmp` is a tmpfs. + For a GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory + as well, and the 4.2 GiB above includes that memory. For an 8192x8192 source + the file was measured at: + - 192 MiB for an 8-bit RGB JPEG; + - 512 MiB for a 16-bit RGBA PNG or JPEG XL; + - 1 GiB for a 32-bit floating-point RGBA JPEG XL. + + So count up to 1 GiB per image processed at once: 24 GiB at the defaults on + a host where pixa sees 48 CPUs. This `/tmp` is in the container's own + filesystem, not on the volume at `/var/lib/pixa`. `cache_max_bytes` does not + count these files, and the 4.2 GiB above does not include them. + - **Health check:** the image's `HEALTHCHECK` requests `/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds after a deploy and marks the deploy failed unless it is `healthy`. The probe - uses the port from `PORT` (default `8080`), so a port changed only in a - mounted config file is not seen by it: change the port with `PORT`. + takes the port from `PORT` (default `8080`). It does not see a port changed + only in a mounted config file, so change the port with `PORT`. ## Rationale -Image-heavy web applications need a fast, caching reverse proxy that can resize -and transcode images on the fly. pixa fills that role as a single, -self-contained binary with no external runtime dependencies beyond libvips. It -supports HMAC-SHA256 signed URLs with expiration to prevent abuse, and -allowlisted source hosts for open access. +A web application that shows images needs each one at several sizes and in +several formats. pixa makes them on request from one original, and keeps what it +made. The application stores one image per picture and hands out URLs. pixa is +one binary. At run time it needs only the image libraries and the CA +certificates named under Deployment. Signed URLs with an expiry keep strangers +from using it to resize anything they like. Allowlisted hosts give open access +where that is wanted. ## Design @@ -183,30 +221,30 @@ allowlisted source hosts for open access. These terms are the names the code uses: - **Source**: the image at one upstream URL, named by its host, path and query. -- **Source key**: the SHA-256 of a source's host, path and query, which names - the source's file and its row in the database. -- **Transform**: the output size, format, quality and fit an image request asks - for; with the format `auto`, pixa chooses the output format from the `Accept` - header. -- **Image request**: a source and a transform, which each image route reads from +- **Source key**: the SHA-256 hash of a source's host, path and query. SHA-256 + is a hash function: it turns any text into a fixed-length fingerprint. The + source key names the source's file and its row in the database. +- **Transform**: the output size, format, quality and fit that an image request + asks for. With the format `auto`, pixa chooses the output format from the + `Accept` header. +- **Image request**: a source and a transform. Each image route reads one from its URL. - **Variant**: the image pixa serves for an image request, made from its source with its transform. -- **Variant key**: the SHA-256 of a variant's source, size, format, quality and - fit, which names the variant's file and its row in the database, and gives its - `ETag`. -- **Signed URL**: a `/v1/image/` URL whose `sig` signs its image request and its - expiry `exp` with `signing_key`. +- **Variant key**: the SHA-256 hash of a variant's source, size, format, quality + and fit. It names the variant's file and its row in the database, and gives + its `ETag`. +- **Signed URL**: a `/v1/image/` URL whose `sig` parameter signs its image + request and its expiry `exp` with `signing_key`. - **Encrypted URL**: a `/v1/e/` URL whose token holds its image request and any expiry, encrypted with a key derived from `signing_key`. - **Login session**: an encrypted cookie that keeps a browser logged in to the - generator page for 30 days, or until a logout replaces the random value pixa - keeps in the `logout` table (see Encrypted URLs). + generator page. It lasts 30 days, or until a logout (see Encrypted URLs). - **Negative cache**: the source URLs pixa does not fetch again for 5 minutes, - after one of these failed fetches: - - the upstream answered with an error status + after a fetch failed in one of these ways: + - the upstream answered with an error status; - the body's first bytes were not those of the format its `Content-Type` - names + names. - **Image cache**: the part of pixa that keeps the sources, the variants and the negative cache (see Storage). - **Disk cache limit** (`cache_max_bytes`): the size in bytes that the cached @@ -214,95 +252,31 @@ These terms are the names the code uses: - **Image proxy**: the part of pixa that serves a variant from the image cache when it can, and otherwise makes it from the source. -### Storage - -- **Sources**: `/cache/sources///`, one file for - each source URL -- **Database**: `/state.sqlite3` (SQLite) -- **Variants**: `/cache/variants///` - -`` and `` are the first and second pairs of characters of the file's -name. - -The image cache keeps these tables in the database: - -- `source_content`: a row for each source, by source key, with its size and when - it was last used -- `variant_content`: a row for each variant, by variant key (column - `cache_key`), with its size and when it was last used -- `cache_usage`: the total size of the cached sources and variants, which - eviction reads -- `negative_cache`: a row for each source URL in the negative cache, with when - it expires - -pixa reads a cached source image's format from its file's first bytes. When they -match none of the source formats, pixa fetches the source again. Two source URLs -that serve the same bytes are stored twice, once under each URL. A source image -is stored by writing its file and then its row; when the row cannot be written, -the file is removed again. - -Each source and variant file is written under a temporary name and synced to -disk before it is renamed into place. So a crash never leaves part of a file -under its own name. When pixa starts, and every five minutes after, it removes -each source or variant whose file is empty, or not the size its row records. It -logs a warning naming the file. A later request for that image fetches the -source or makes the variant again. Looking for those files reads the database -once for every cached file, so pixa skips each five-minute look that would start -less than ten times as long after the last look ended as that look took. After a -look that took one minute, the next one starts 10 to 15 minutes after that look -ended. - -Each look also drops the rows of variants whose files are missing. When pixa -cannot read a variant file's size for another reason, it keeps the variant's -row. It logs a warning naming the variant key and the error. - -During a look, pixa skips a directory it cannot list and a file whose size it -cannot read. It logs a warning naming the path and the error, and goes on with -the rest of the look. A file renamed or removed after pixa listed its directory -gets no warning. - -pixa's target is 1-5k r/s, which has not been measured at that rate (see Load -Test). Toward it, a cache hit opens the variant's file and nothing else, and -makes no database call. The content type comes from the file's first bytes, -which are those of the format pixa wrote it in; a file whose first bytes are not -those of a format pixa writes is not served, and the image is converted again. A -hit hands the time it was served to a background writer and does not wait for -the writer. Eviction orders by that time. The hit is counted in memory. A miss -that uses a cached source image also hands that writer the time it used the -image. Once a second that writer writes the waiting times in one transaction. Up -to 10,000 access times of variants, and as many of sources, can wait to be -written. A time handed over beyond that is dropped. Its image then only looks -less recently used to eviction. `script/bench` measures how many hits pixa -serves per second (see Load Test). - ### Routes -pixa answers these routes; any other path answers 404. A path in this list asked -with a method the list does not give answers 405, except `/static/`, which -answers any method as it answers `GET`. A browser's CORS preflight request -(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any -path under `/v1/` answers 200, in maintenance mode too. +pixa answers these routes. Any other path answers 404. A listed path answers 405 +to any method the list does not give for it. -- `GET /` — the login page, or the URL generator page with a login session (see - Encrypted URLs). Needs: nothing. Answers: 200. -- `POST /` — log in with the signing key typed into the login page. Needs: the - login page's form (below). Answers: 303 to `/` with a login session cookie - that lasts 30 days for the right key; 200 with the login page and an error for - a wrong key; 429 over the login limit (below). -- `POST /generate` — make an encrypted URL from the generator page's form. - Needs: a login session and the generator page's form (below); without a login - session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400 - with the page naming a field that is not valid; 500 when the URL cannot be - made. -- `POST /logout` — log out with the generator page's Logout button: end every - login session, in every browser (see Encrypted URLs), and remove the session - cookie. Needs: a login session and the generator page's form (below); without - a login session it only removes the cookie. Answers: 303 to `/`; 500 when the +- `GET /` — the login page, or the generator page with a login session (see + Encrypted URLs). Needs nothing. Answers 200. +- `POST /` — log in with the signing key typed into the login page. Needs the + login page's form (below). For the right key it answers 303 to `/` and sets + the login session cookie. For a wrong key it answers 200 with the login page + and an error. Answers 429 over the login limit (see Limits). +- `POST /generate` — make an encrypted URL from the generator page's form. Needs + a login session and the generator page's form. Without a login session it + answers 303 to `/`. Answers 200 with the page showing the URL. Answers 400 + with the page naming a field that is not valid. Answers 500 when the URL + cannot be made. +- `POST /logout` — log out with the generator page's Logout button. That ends + every login session, in every browser (see Encrypted URLs), and removes the + cookie. Needs a login session and the generator page's form. Without a login + session it only removes the cookie. Answers 303 to `/`. Answers 500 when the logout cannot be saved in the database. - `GET` or `HEAD` `/v1/image///.`, or `/v1/image///` with no format — an image, fetched, resized - and converted (below). Needs: a signature, unless the host is allowlisted (see - Source Hosts). Answers: + and converted (see Image URLs). Needs a signature, unless the host is + allowlisted (see Source Hosts). Answers: - 200 with the image. - 304 when `If-None-Match` matches the image's `ETag`. - 400 for a URL or parameter that is not valid. @@ -312,34 +286,34 @@ path under `/v1/` answers 200, in maintenance mode too. - 401 for a missing or wrong signature. - 401 for a missing `exp`. - 401 for an `exp` in the past. - - 403 when the request's `Referer` names a host in `referer_blocklist`, - checked before the signature, the cache and the upstream fetch. + - 403 when the request's `Referer` names a host in `referer_blocklist`. This + is checked before the signature, the cache and the upstream fetch. - 403 when the upstream host, or a host it redirects to, is `localhost` or ends in `.localhost` or `.local`. - 403 when the upstream host, or a host it redirects to, has an address in a blocked network (see `blocked_networks`). - - 429 over `image_requests_per_minute` (below). + - 429 over `image_requests_per_minute` (see Limits). - 499, with no body, when the client goes away before its image is ready, - unless its request is the one making the image (below). + unless its request is the one making the image (see Limits). - 415 when the upstream answers with a `Content-Type` that is not one of the - source formats (below). + source formats (see Image URLs). - 415 when the upstream sends a body whose first bytes are not those of the format its `Content-Type` names. - 415 when the upstream sends an image whose first bytes libvips reads as another format. - 502 when the upstream answers with an error status. - 502 for 5 minutes after that for the same source URL, when pixa has - neither the image nor its source image in its cache. + neither the image nor its source in its cache. - 502 likewise for 5 minutes after a body whose first bytes are not those of its `Content-Type`. - - 503 when pixa is busy. - - 503 when pixa is in maintenance mode. + - 503 when pixa is busy (see Limits). + - 503 in maintenance mode. - 504 when the upstream has not sent its response headers within `upstream_fetch_timeout`. - 500 when that time runs out while the image itself is still arriving. - 500 for any other failure. - `GET` or `HEAD` `/v1/e//` — an image through an encrypted URL - (see Encrypted URLs). Needs: nothing but the URL. Answers: + (see Encrypted URLs). Needs nothing but the URL. Answers: - 200 with the image. - 304 when `If-None-Match` matches the image's `ETag`. - 400 for a token that does not decrypt. @@ -357,395 +331,578 @@ path under `/v1/` answers 200, in maintenance mode too. - 503 - 504 - 500 -- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs: - nothing. Answers: 200. +- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs + nothing. Answers 200. - `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`, `uptime_seconds`, `uptime_human`, `version`, `appname` and `maintenance_mode`. - Needs: nothing. Answers: 200, always. + Needs nothing. Answers 200, always. - `GET /static/` — the stylesheet and script the login and generator pages - load. Needs: nothing. Answers: 200, or 404 for a file that does not exist. -- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic - authentication with `metrics.username` and `metrics.password`. Answers: 200; - 401 without them; 404 when they are not set, as the route then does not exist. + load. Needs nothing. Answers 200, or 404 for a file that does not exist. It + answers any other method as it answers `GET`. +- `GET /metrics` — Prometheus metrics (see Architecture). Needs HTTP basic + authentication with `metrics.username` and `metrics.password`. Answers 200. + Answers 401 without them. Answers 404 when they are not set, as the route then + does not exist. -Every response carries an `X-Request-ID` header holding the request's ID, which -a client can quote when reporting a problem: the request's own `X-Request-ID`, -as a reverse proxy in front of pixa may send, when it is at most 64 letters, -digits, `-`, `_` or `.`; otherwise a random one pixa makes for the request, -which tells nothing about the machine or the other requests. pixa's log line for -the request carries the same ID as `request_id`, and so do the lines it logs -when it fetches, converts and serves an image; the fetch sends it to the -upstream host as `X-Request-ID`. +A browser's CORS preflight request to any path under `/v1/` answers 200, in +maintenance mode too. CORS is the browser rule that lets a page on one site read +responses from another. The preflight is the `OPTIONS` request the browser sends +first. -The `POST` routes accept only a form that pixa's own page served: the page puts -a token in the form and sets a cookie to match, and a request without both is -refused with 403, so another site cannot submit the form from a visitor's -browser. The login and generator pages are meant to be opened over HTTPS: while -`debug` is off, a form sent from a page opened over plain HTTP is refused with -403, and while it is on, so is one sent from a page opened over HTTPS. Plain -HTTP is for development on the browser's own machine: the login session cookie -is always marked `Secure`, and over plain HTTP a browser keeps such a cookie -only for its own machine (`localhost`), if at all. A form is also refused with -403 when the page's host is not the `Host` header pixa receives, so a reverse -proxy in front of pixa must pass that header on unchanged. A form body over 1 -MiB is refused with 413. The image routes answer the errors listed for them with -JSON holding `error`, `status` and `timestamp`. +Every response carries an `X-Request-Id` header with the request's ID. A client +can quote it when reporting a problem. A reverse proxy may send an +`X-Request-Id` header with the request. pixa keeps that ID when it is 1 to 64 +letters, digits, `-`, `_` or `.`. Otherwise pixa makes a random ID, which tells +nothing about the machine or the other requests. pixa's log line for the request +carries the same ID as `request_id`. So do the lines it logs when it fetches, +converts and serves an image. The fetch sends it to the upstream host as +`X-Request-Id`. -An image URL has one of these forms, the second with no format: +The `POST` routes accept only a form that pixa's own page served. The page puts +a token in the form and sets a cookie to match. A request without both is +refused with 403. So another site cannot submit the form from a visitor's +browser. A form is also refused with 403 when the page's host is not the `Host` +header pixa receives. So a reverse proxy in front of pixa must pass that header +on unchanged. A form body over 1 MiB is refused with 413. + +The login and generator pages are meant to be opened over HTTPS. While `debug` +is off, a form sent from a page opened over plain HTTP is refused with 403. +While `debug` is on, a form sent from a page opened over HTTPS is refused +instead. Plain HTTP is for development on the browser's own machine. The login +session cookie is always marked `Secure`. Over plain HTTP a browser keeps such a +cookie only for its own machine (`localhost`), if at all. + +The image routes answer the errors listed for them with a JSON body holding +`error`, `status` and `timestamp`. + +### Image URLs + +An image URL has one of these forms. The second has no format: ``` /v1/image///.?sig=&exp=&q=&fit= /v1/image///?sig=&exp=&q=&fit= ``` -Images are only fetched from origins using TLS with valid certificates, unless -`allow_http` is set: then pixa fetches every image over plain HTTP, which is for -testing only. - -A request whose query string cannot be decoded, or gives any parameter more than -once, is refused with 400. - -- ``: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`, - `avif`, `jxl` (JPEG XL), `gif`, or `auto` (below). A URL with no format (the - second form, with no dot after the size) is served as JPEG XL, the default, as - with `jxl` -- ``: `orig` or `x` (e.g. `800x600`) +- ``: the upstream host, with a port when the source URL has one (see + Source Hosts). +- ``: the source URL's path, and its query after `%3F` (below). +- ``: `orig` or `x`, for example `800x600`. A side of `0` + is worked out from the other side and the source's proportions (below). `orig` + is `0x0`: the source's own size. +- ``: `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`, `avif`, + `jxl` (JPEG XL), `gif`, or `auto` (below). `orig` is the source's own format. + A URL with no format is served as JPEG XL, the default, as with `jxl`. - `sig` and `exp`: the signature and its expiry, needed unless the host is - allowlisted (see Signature Specification) -- `q` and `fit`: the output quality and how the image is fitted to ``, - both optional (values under Signature Specification). Both are part of what is - cached, so each value of either is a separate cached image. + allowlisted (see Signed URLs). +- `q`: the output quality, a whole number from 1 to 100. It is `85` when the URL + has no `q`. +- `fit`: how the image is fitted to ``: `cover`, `contain`, `fill`, + `inside` or `outside` (below). It is `cover` when the URL has no `fit`. -A source URL's query follows its path after `%3F`, the encoded `?`: -`/v1/image//%3F/.`. pixa decodes the -percent-escapes in the image URL's path once, then reads the host up to the -first `/`, the size and format after the last `/`, and between them the source -path up to the first `?` and the source query after it. So the `?` before the -query, any `?` in the query, and every `%` in the source path or query must be -percent-encoded, as `%3F` and `%25`. Any other character may be written as -itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither -the source path nor the source query may hold a `#`, and an image URL that -writes one as `%23` is refused with 400. For example, +`q` and `fit` are part of what is cached, so each value of either is a separate +cached image. A request is refused with 400 when its query string cannot be +decoded, or when it gives any parameter more than once. A request with a `q`, +`fit` or `exp` that is empty or not valid is refused with 400 too. + +pixa fetches sources over HTTPS only, and only from hosts with valid +certificates. With `allow_http` set, it fetches every source over plain HTTP +instead. That is for testing only. + +**The source path and query.** A source URL's query follows its path after +`%3F`, the encoded `?`: + +``` +/v1/image//%3F/. +``` + +pixa decodes the percent-escapes in the image URL's path once. It then reads the +host up to the first `/`, and the size and format after the last `/`. Between +them it reads the source path up to the first `?`, and the source query after +it. So these must be percent-encoded, as `%3F` and `%25`: + +- the `?` before the query; +- any `?` in the query; +- every `%` in the source path or query. + +Any other character may be written as itself or percent-encoded. `/`, `;`, `,`, +`=` and `&` work either way. Neither the source path nor the source query may +hold a `#`. An image URL that writes one as `%23` is refused with 400. For +example, the image `https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at -800x600 WebP is -`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`, -signed as shown under Signature Specification. +800x600 as WebP is: -The source path is the path with its percent-escapes decoded, and pixa fetches -it percent-encoded again: every character other than letters, digits, `-`, `.`, -`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a -percent-escape. So `my%20cat.jpg` in the image URL is the source path -`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path -`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as -`/100%25.png`. The source query is the source URL's query as that URL writes it, -its percent-escapes included, and is sent as it is, except that the characters a -browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control -characters and characters outside ASCII) and a `%` that does not begin a -percent-escape are percent-encoded. pixa signs and caches the query in that -form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are -the same source query, `name=my%20cat`. +``` +/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp +``` -A size with one side `0` has that side scaled to keep the source's proportions, -and is refused with 400 when that side would come out above 8192, the most -either side may be; so is a size with `fit=outside`, which scales the image to -cover the size and so past it on one side, when that side would come out -above 8192. +Signed URLs shows this URL with its signature. -An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048 -have, is refused with 400 before anything is decoded, as its encode could take -more than 40 seconds (see below). That includes `orig` of an AVIF source, and -`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but -not `image/jxl` is refused such a size. +pixa keeps the source path decoded and fetches it percent-encoded again. Every +character other than these is sent as a percent-escape: letters, digits, `-`, +`.`, `_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@`. So: -The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's -`Content-Type` must name its format, and the image's first bytes must match it. -An upstream answering with any other `Content-Type`, SVG included, is refused -with 415. So is a body whose first bytes are not those of the format its -`Content-Type` names, such as a JPEG sent as `image/png`, text sent as an image, -or an empty body. pixa then does not fetch that source URL again for 5 minutes, -as after an upstream error status, and meanwhile answers 502 for an image that -would need that fetch. An image whose first bytes libvips reads as another -format, such as an SVG whose first bytes pass the check for AVIF, is refused -with 415 too. +- `my%20cat.jpg` in the image URL is the source path `/my cat.jpg`, fetched as + `/my%20cat.jpg`; +- `my%2520cat.jpg` is the source path `/my%20cat.jpg`, fetched as + `/my%2520cat.jpg`; +- `100%25.png` is fetched as `/100%25.png`. -With the format `auto`, pixa chooses the format for each request from its -`Accept` header, in this order: +The source query is sent as the source URL writes it, its percent-escapes +included. Two kinds of character are percent-encoded first: those a browser +percent-encodes in a query, and a `%` that does not begin a percent-escape. The +first kind is a space, `"`, `'`, `<`, `>`, control characters and characters +outside ASCII. pixa signs and caches the query in that form. So +`img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are the same +source query, `name=my%20cat`. + +**Sizes.** The largest width or height is 8192, for the source and for the +output. A size with one side `0` has that side worked out from the other side +and the source's proportions. The side is rounded down, but is at least 1. When +it would come out above 8192, the request is refused with 400. The fit modes: + +- `cover`: scale the image to cover the size, keeping its proportions, and cut + out the centre. The output is exactly the size. +- `contain`: scale the image to fit inside the size, keeping its proportions. + The output can be smaller than the size on one side. +- `fill`: stretch the image to the size, without keeping its proportions. +- `inside`: as `contain`, but never enlarge the image. +- `outside`: scale the image to cover the size, keeping its proportions, without + cutting. The output can be larger than the size on one side. An 8192x1 source + at `100x0` is 100x1 by its proportions, but comes out at 8192x1. A side that + would come out above 8192 is refused with 400. + +An AVIF output of more than 16,777,216 pixels is refused with 400 before +anything is decoded. That is the number of pixels in 4096x4096 or 8192x2048. A +larger AVIF encode could take more than 40 seconds (see Limits). The limit +includes `orig` of an AVIF source, and `auto` when it chooses AVIF. So a client +whose `Accept` names `image/avif` but not `image/jxl` is refused such a size. + +**Source formats.** The source may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The +upstream's `Content-Type` must name its format, and the image's first bytes must +match it. An upstream answering with any other `Content-Type`, SVG included, is +refused with 415. So is a body whose first bytes are not those of the format its +`Content-Type` names: a JPEG sent as `image/png`, text sent as an image, or an +empty body. pixa then does not fetch that source URL again for 5 minutes, as +after an upstream error status. Meanwhile it answers 502 for an image that would +need that fetch. An image whose first bytes libvips reads as another format is +refused with 415 too. An SVG whose first bytes pass the check for AVIF is one +example. + +**The format `auto`.** With `auto`, pixa chooses the format for each request +from its `Accept` header, in this order: 1. JPEG XL, when the header names `image/jxl`; 2. AVIF, when it names `image/avif`; 3. WebP, when it names `image/webp`; -4. JPEG, when the first of `image/jpeg`, `image/*` and `*/*` that it names - allows it, or when there is no `Accept` header or it is empty. +4. JPEG, when the most specific of `image/jpeg`, `image/*` and `*/*` that the + header names has a `q` above 0, or when there is no `Accept` header or it is + empty. -An entry with `q=0` refuses its format; other `q` values do not change the +An entry with `q=0` refuses its format. Other `q` values do not change the order. JPEG XL, AVIF and WebP must be named, as clients that cannot show them -also send `image/*` and `*/*`. pixa never sends a format the client refused: -when the header allows none of the four, the answer is 406, and a header that -does not parse, or has a `q` that is not a number from 0 to 1, is refused -with 400. The signature, or the token of an encrypted URL, covers `auto` itself, -so one URL serves every client. Each format chosen is cached as a separate -image, and every answer that depends on `Accept` (the image, a 304, and the 400 -and 406 above) carries `Vary: Accept`, so a shared cache keeps the formats apart -too. +also send `image/*` and `*/*`. pixa never sends a format the client refused. +When the header allows none of the four, the answer is 406. A header that does +not parse, or has a `q` that is not a number from 0 to 1, is refused with 400. +The signature, or the token of an encrypted URL, covers `auto` itself, so one +URL serves every client. Each format chosen is cached as a separate image. Every +answer that depends on `Accept` carries `Vary: Accept`: the image, a 304, and +the 400 and 406 above. So a shared cache keeps the formats apart too. -An image is served with `Cache-Control: public, max-age=, immutable`. -When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age` -is the whole seconds left until then, at most one year, so no browser or proxy -cache keeps the image after pixa would refuse the URL. A URL with no expiry gets -one year. `immutable` only stops a client revalidating while its copy is fresh. +**Response headers.** An image is served with +`Cache-Control: public, max-age=, immutable`. When the URL has an +expiry (an `exp`, or the expiry of an encrypted URL), `max-age` is the whole +seconds left until then, at most one year. So no browser or proxy cache keeps +the image after pixa would refuse the URL. A URL with no expiry gets one year. +`immutable` only stops a client revalidating while its copy is fresh. The +response also carries `X-Pixa-Cache`: `HIT` for an image served from the cache, +and `MISS` for one made for this request. The `ETag` is the first 16 characters +of the variant key, in quotes. -When several requests for the same image, size, format, quality and fit miss the -cache at once, they share one upstream fetch (or one read of the cached source) -and one transcode: the first request does the work, and the others wait for its -image or its error, holding no upstream connection or processing slot of their -own. A waiting request stops waiting when its own client goes away. The work -goes on for the others even if the first request's client goes away. The shared -fetch sends the first request's ID upstream, and the lines logged for the fetch -and the transcode carry that ID. +### Signed URLs -A transcode whose request's `downstream_timeout` has ended (for a shared one, -the first request's) stops waiting for a processing slot, or, once it has one, -gives it back without beginning to encode. A request waiting for a shared -transcode that stops this way is answered 503 with the error -`server busy, try again later`, though its own `downstream_timeout` may not have -ended. govips gives pixa no way to stop libvips once an encode has begun, so an -encode keeps its slot until it finishes, which can be after -`downstream_timeout`. The limit of 8192 on each side of the source and of the -output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes. -With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels -takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or -16 bits per sample, with or without an alpha channel. The WebP and JPEG XL -encoders run below libvips' default effort, and AVIF at the lowest effort govips -can set, so that no encode takes more than 40 seconds of the default -`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample, -8192x8192 random pixels with an alpha channel take up to about 23 seconds as -WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as -AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha -channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds -up to about 11 seconds. +A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix +time (seconds since 1 January 1970) when it expires, and `sig`, its signature. +The signature is an HMAC-SHA256: a hash of the request's details, keyed with +`signing_key`, that only someone who knows the key can make. Signatures match +exactly: every part of the signed data must be exactly what was signed. There is +no suffix, wildcard or partial matching. -The login form (`POST /`) is limited to 5 attempts per minute per client -address, counting an IPv6 client by its /64; an attempt over the limit is -refused with 429 and a `Retry-After` header. Behind a reverse proxy the client -address comes from `X-Forwarded-For` only when the address pixa sees for -requests that come through the proxy is in `trusted_proxies`; otherwise all -users behind the proxy are counted as one client. That address is not always the -proxy's own: a proxy on the Docker host that connects to pixa over `127.0.0.1` -is seen as the gateway of the container's Docker network, such as `172.17.0.1` -on the default bridge, and one that connects through another of the host's -addresses is seen with that address. To be sure, read it as `remoteIP` in pixa's -request log while it is not in `trusted_proxies` (see `trusted_proxies` under -Configuration). With the default `trusted_proxies` (the RFC 1918 ranges), a -client with a private address can choose the address it is counted by through -its own `X-Forwarded-For`, whether it connects directly or through the proxy, -because its own address is trusted too. Setting `trusted_proxies` to only the -address pixa sees for requests that come through the proxy closes this. - -One client may send `image_requests_per_minute` requests per minute -(default 600) to the image routes, `/v1/image/` and `/v1/e/` together, `GET` and -`HEAD`, cache hits included; a request over the limit is refused with 429 and a -`Retry-After` header. Clients are counted as for the login limit. A browser's -CORS preflight is not counted. Behind a CDN or proxy whose addresses are not in -`trusted_proxies`, all requests arriving from one proxy or CDN address count as -one client, so such a deployment needs a higher limit, or `0`, which turns the -limit off. - -### Encrypted URLs - -An encrypted URL is an image URL made on pixa's own web page by someone who -knows the signing key. It works for any upstream host, allowlisted or not, -without a signature, and whoever gets it can neither read the source URL from it -nor change what it asks for. - -1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see - Routes) and log in with the signing key (`signing_key`). The login session - lasts 30 days, or until someone logs out with the generator page's Logout - button. Every login is made with the one signing key, so logging out ends - every login session, in every browser, a copy of the session cookie included. - pixa keeps a random value in its database, which it makes when the database - has none and replaces at each logout. A login session carries the value - current at its login and is refused once the value has changed, also after a - restart (see Deployment). The check for a logout compares no time, so a login - after a logout works whatever the host's clock says; the 30 days are still - measured by the host's clock. -2. On the generator page, give the source image's URL, the width and height, the - format, quality and fit, and how long the URL lasts, then submit the form - (`POST /generate`). The format is JPEG XL unless another is chosen; a form - sent with an empty format, or none, also makes a JPEG XL URL. Width and - height both empty or `0` keep the original size; if only one of them is empty - or `0`, that side is scaled to keep the image's proportions. A URL whose - scaled side would come out above 8192 answers 400, and so does an AVIF URL of - more than 16,777,216 pixels or a source URL whose path holds a `..` segment. - So does a source URL that the image URL would fetch written another way, and - the page then shows the URL pixa would fetch: one with a user name and - password, which pixa does not send, or one whose path holds `%2F`, fetched as - `/`, or `(`, fetched as `%28`, as pixa keeps the source path decoded and - fetches it percent-encoded again (see Routes). -3. The page shows the URL, `https:///v1/e//img.`, and when - it expires. `` is the host the page was opened on, and the URL starts - with `http` instead while `debug` is on. The name after the token is ignored - and only gives the URL a file extension, `jpg` for `orig` and `auto`, and - `jxl` for a form with no format. - -The token holds the source's host, path and query and the size, format, quality, -fit and expiry, encrypted with a key derived from `signing_key`. The host is -kept in lower case and an empty path as `/`, as for `/v1/image/`, so -`https://CDN.Example.com` and `https://cdn.example.com/` are one source image, -fetched and cached once. The source URL's scheme is not kept: the image is -fetched like any other (see Routes), and the blocked networks still apply. - -How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never. -The expiry is fixed in the token when the URL is made and cannot be changed or -revoked afterwards. Until then the image is served with a `max-age` that ends at -the expiry (see Routes); after it the URL answers 410 `URL has expired`. A URL -made to last forever stops working only when `signing_key` changes: changing it -makes every encrypted URL already handed out answer 400, and ends every login -session. - -### Image Metadata - -pixa decodes and re-encodes every image it serves, and removes all metadata from -the output: EXIF (GPS position, camera make, model and serial number, capture -time, embedded thumbnail), XMP, IPTC and the ICC colour profile. This cannot be -turned off. - -- The `orig` format means the source's own format, not the source's bytes: an - `orig` image is re-encoded and stripped like any other. -- An image with an EXIF orientation is turned upright first, so it displays the - same without the tag; a requested size applies to the upright image. -- An image with an ICC profile is converted to sRGB first, since clients show an - image with no profile as sRGB. Colours outside sRGB, such as the most - saturated ones in a Display P3 photo, are clipped. -- The one exception is JPEG XL with libvips 8.16 and later: libvips then writes - an EXIF block of its own into the image, as govips cannot ask libvips to leave - it out. It holds the image's size and otherwise fixed values, such as an - orientation of 1 and a resolution of 72 dpi, and nothing from the source. - -### Source Hosts - -Source hosts may be allowlisted in the configuration. Non-allowlisted hosts -require an HMAC-SHA256 signature. - -#### Signature Specification - -Signatures use HMAC-SHA256 and include an expiration timestamp to prevent replay -attacks. Signatures are **exact match only**: every component (host, path, -query, dimensions, format, expiration, quality, fit) must match exactly what was -signed. No suffix matching, wildcard matching, or partial matching is supported. - -**Signed data format** (colon-separated): +The signed data is one line of nine parts, joined with `:`: ``` HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit") ``` -Where: +- `host`: the source host as the URL writes it up to the first `/`, in lower + case. An IPv6 address keeps its brackets, and a port is kept after its `:`. + Examples: `cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`. +- `path`: the source path with its percent-escapes decoded, as under Image URLs, + then with each `%` written as `%25` and each `:` as `%3A`. So + `/photos/my%20cat.jpg` in the URL is signed as `/photos/my cat.jpg`, and + `/photos/a:b.jpg` as `/photos/a%3Ab.jpg`. +- `query`: the source query as under Image URLs, with `%` and `:` written as in + `path`. It is empty when there is no query. Neither part then holds a `:`, so + path `/a:b` with no query and path `/a` with query `b:` are signed + differently. +- `width` and `height`: the requested size in pixels. A side of `0` is signed as + `0`, and `orig` as `0` and `0`. +- `format`: the output format as listed under Image URLs, with `original` signed + as `orig`, `jpg` as `jpeg`, and no format as `jxl`. So a URL with no format + has the signature of the same URL ending in `.jxl`. `auto` is signed as + `auto`, not as the format chosen for the request. +- `expiration`: the URL's `exp`, the Unix time in seconds when the signature + expires. A request whose `exp` is not a whole number is refused with 400. An + empty `exp=` is one example. +- `quality`: the URL's `q`, or `85` when the URL has no `q`. +- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`. -- `host` — source host as the URL writes it up to the first `/`, in lower case, - keeping an IPv6 address's brackets and any `:` and port after it (e.g. - `cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`) -- `path` — source path, with its percent-escapes decoded as under Routes (e.g. - `/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with - each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is - signed as `/photos/a%3Ab.jpg` -- `query` — source query string as under Routes, empty string if none, with `%` - and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no - query and path `/a` with query `b:` are signed differently -- `width` — requested width in pixels, or `0`: the width is then worked out from - the source's proportions and the requested height, rounded down but at least - 1, or is the source's own width when the height is `0` too (the `orig` size) -- `height` — requested height in pixels, or `0`: the height is then worked out - from the source's proportions and the requested width, rounded down but at - least 1, or is the source's own height when the width is `0` too (the `orig` - size) -- `format` — output format, one of those listed under Routes, with `original` - signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no - format has the signature of the same URL ending in `.jxl`; `auto` is signed as - `auto`, not as the format chosen for the request -- `expiration` — the URL's `exp` query parameter, the Unix timestamp when the - signature expires; a request whose `exp` is not a whole number, an empty - `exp=` included, is refused with 400 -- `quality` — the URL's `q` query parameter, a whole number from 1 to 100, or - `85` when the URL has no `q`; a request whose `q` is anything else is refused - with 400 -- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside, - outside), or `cover` when the URL has no `fit`; a request whose `fit` is - anything else, an empty `fit=` included, is refused with 400. `outside` scales - the image to cover the requested size, keeping its proportions, so it can come - out larger than that size on one side: an 8192x1 source at `100x0`, which is - 100x1 by its proportions, comes out at 8192x1 +The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of +RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa +compares it exactly. A signature encoded without padding is refused with 401. +Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding. -The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of -RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa -compares it exactly, so a signature encoded without padding, as Node's -`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401. +**Example.** With the signing key `example-signing-key-for-documentation`, +resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at +1704067200, with the default quality and fit: -**Example:** with the signing key `example-signing-key-for-documentation`, -resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with expiration -1704067200, default quality and fit: - -1. Build input: +1. Build the signed data: `cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover` -2. Compute HMAC-SHA256 of it with the signing key -3. Base64URL-encode the result, keeping the `=` padding: +2. Compute its HMAC-SHA256 with the signing key. +3. Encode the result as base64url, keeping the `=` padding: `-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=` -4. URL: +4. The URL: `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=-ay7KHpfqmtIGbibDGbUuBDkymi-Ymdn0NkC6j5EJag=&exp=1704067200` -For the same image at quality 40 with fit `contain`, the input ends in -`:40:contain`, the signature is `5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=`, +For the same image at quality 40 with fit `contain`, the signed data ends in +`:40:contain`. The signature is `5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=`, and the URL is `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`. For a source URL with a query, `https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at -800x600 WebP with the same key and expiration and the default quality and fit: +800x600 WebP with the same key, expiry, quality and fit: -1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`: +1. The signed data, with the query's `:` written as `%3A` and its `%` as `%25`: `cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover` -2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=` -3. URL, with the `?` before the query written as `%3F` and the query's `%` as - `%25` (see Routes): +2. The signature: `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=` +3. The URL, with the `?` before the query written as `%3F` and the query's `%` + as `%25`: `/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200` -**Allowlist patterns:** +### Encrypted URLs -- **Exact match**: `cdn.example.com` — matches only that host -- **Suffix match**: `.example.com` — matches `cdn.example.com`, - `images.example.com`, and `example.com` +An encrypted URL is an image URL made on pixa's own web page by someone who +knows the signing key. It works for any upstream host, allowlisted or not, +without a signature. Whoever gets it can neither read the source URL from it nor +change what it asks for. -An IP address is matched exactly; write an IPv6 address without brackets. An -entry that is neither a host name (letters, digits, hyphens, underscores and -dots, with at most one leading dot) nor an IP address, such as one with a port -or a `*.` wildcard, aborts startup. +1. Open `/` in a browser over HTTPS and log in with the signing key. The login + session lasts 30 days, or until someone logs out with the generator page's + Logout button. +2. On the generator page, fill in the form. Its fields are listed below. Then + submit it (`POST /generate`). +3. The page shows the URL, `https:///v1/e//img.`, and when + it expires. `` is the host the page was opened on. While `debug` is on, + the URL starts with `http` instead. The name after the token is ignored. It + only gives the URL a file extension: `jpg` for `orig` and `auto`, and `jxl` + for a form with no format. -pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so -`CDN.Example.com` is matched, fetched, cached and counted against -`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has -a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature -even when the host is allowlisted. The host must be a host name or an IPv4 -address (letters, digits, hyphens, underscores and dots) or an IPv6 address in -brackets, either one optionally followed by `:` and a port. Any other host, such -as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is -refused with 400. +The form's fields: + +- The source image's URL. Its scheme is not kept (below). +- The width and height. Both empty or `0` keep the source's size. When only one + of them is empty or `0`, that side is worked out from the source's proportions + when the image is made (see Image URLs). +- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. + `jxl` is selected unless another is chosen. A form sent with an empty format, + or none, also makes a JPEG XL URL. +- The quality, a whole number from 1 to 100. The page offers five values: `25`, + `50`, `70`, `85` and `100`. `85` is selected unless another is chosen. +- The fit, one of the five fit modes (see Image URLs). `cover` is selected + unless another is chosen. +- How long the URL lasts, in seconds, or `0` for never. The page offers seven + choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is + selected unless another is chosen. + +The page refuses a form with 400 when: + +- the source URL does not parse, or has no host; +- the source URL's path holds a `..` segment, as `/v1/image/` refuses one; +- a width, height, quality or expiry is not a number, or is out of range; +- the fit is not one of the five fit modes; +- pixa would fetch the source URL in another form (below). + +The page refuses a source URL that pixa would fetch in another form, and shows +the URL pixa would fetch. A URL with a user name and password is one example: +pixa does not send them. A URL ending in `?` with no query is another. A path +holding `%2F` is another: pixa would fetch it as `/`. So is a path holding `(`: +pixa would fetch it as `%28` (see Image URLs). + +The image is made only when the URL is requested. A side that would come out +above 8192, or an AVIF output of more than 16,777,216 pixels, is refused with +400 then (see Image URLs). + +Every login is made with the one signing key. So logging out ends every login +session, in every browser, and a copied session cookie too. pixa keeps a random +value in its database. It makes the value when the database has none, and +replaces it at each logout. A login session carries the value current at its +login. Once the value has changed, the session is refused, after a restart too. +The check compares no time, so a login after a logout works whatever the host's +clock says. The 30 days are still measured by the host's clock. + +The token holds the source's host, path and query, and the size, format, +quality, fit and expiry. It is encrypted with a key derived from `signing_key`. +The host is kept in lower case and an empty path as `/`, as for `/v1/image/`. So +`https://CDN.Example.com` and `https://cdn.example.com/` are one source, fetched +and cached once. The source URL's scheme is not kept. The image is fetched like +any other (see Image URLs), and the blocked networks still apply. + +The expiry is fixed in the token when the URL is made. It cannot be changed or +revoked afterwards. Until then the image is served with a `max-age` that ends at +the expiry (see Image URLs). After it, the URL answers 410 `URL has expired`. A +URL made to last forever holds no expiry. It stops working only when +`signing_key` changes. Changing the key makes every encrypted URL already handed +out answer 400, and ends every login session. + +### Source Hosts + +Hosts in `allowlist_hosts` are served without a signature. Any other host needs +one. An entry is written in one of two ways: + +- `cdn.example.com`: an exact match, only that host; +- `.example.com`: a suffix match, which matches `cdn.example.com`, + `images.example.com` and `example.com` itself. + +An IP address is matched exactly. Write an IPv6 address without brackets. An +entry that is neither a host name nor an IP address aborts startup. A host name +here is letters, digits, hyphens, underscores and dots, with at most one leading +dot. So an entry with a port, or a `*.` wildcard, aborts startup. + +pixa lower-cases the host of a `/v1/image/` URL before anything else reads it. +So `CDN.Example.com` is matched, fetched, cached and counted against +`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a +port needs a signature even when the host is allowlisted. So does one whose host +ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host +must be one of: + +- a host name or an IPv4 address: letters, digits, hyphens, underscores and + dots; +- an IPv6 address in brackets. + +Either may be followed by `:` and a port. Any other host is refused with 400. +One holding `@` is an example. So is one with a `?`, `#`, `%`, `\` or space +written percent-encoded. + +### Image Metadata + +pixa decodes and re-encodes every image it serves, and removes all metadata from +the output. This cannot be turned off. The metadata removed is: + +- EXIF, which can hold the GPS position, the camera make, model and serial + number, the capture time and an embedded thumbnail; +- XMP and IPTC, two other metadata formats; +- the ICC colour profile. + +The rest of the rules: + +- The `orig` format means the source's own format, not the source's bytes. An + `orig` image is re-encoded and stripped like any other. +- An image with an EXIF orientation is turned upright first, so it displays the + same without the tag. A requested size applies to the upright image. +- An image with an ICC profile is first converted to sRGB, the standard colour + space, since clients show an image with no profile as sRGB. Colours outside + sRGB are clipped: the most saturated ones in a Display P3 photo, for example. +- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then + writes an EXIF block of its own, which pixa cannot stop. The block holds the + image's size and fixed values: an orientation of 1 and a resolution of 72 dpi. + Nothing in it comes from the source. The Docker image has libvips 8.16, so its + JPEG XL output carries this block. + +### Limits + +**Shared work.** Several requests for one variant can miss the cache at the same +time. They then share one upstream fetch, or one read of the cached source, and +one conversion. The first request does the work. The others wait for its image +or its error, and hold no upstream connection or processing slot of their own. A +waiting request stops waiting when its own client goes away. It is then +answered 499. The work goes on for the others even when the first request's +client goes away. The shared fetch sends the first request's ID upstream. The +lines logged for the fetch and the conversion carry that ID. + +**Timeouts.** `downstream_timeout` (default `60s`) bounds the whole answer to +one request. The upstream fetch counts toward it. So do the waits for a +connection slot and for a processing slot. The wait for a shared +`upstream_connections` slot ends after 10 seconds, and so does the wait for a +processing slot. The wait for an `upstream_connections_per_host` slot ends only +with `downstream_timeout`. Keep `downstream_timeout` longer than +`upstream_fetch_timeout` plus 20 seconds. + +A conversion whose request's `downstream_timeout` has ended stops waiting for a +processing slot. If its `downstream_timeout` ends after it has a slot but before +it begins to encode, it gives the slot back. For a shared conversion, the first +request's deadline counts. When a shared conversion stops this way, each request +waiting for it is answered 503 `server busy, try again later`, even if its own +`downstream_timeout` has not ended. govips, the library pixa drives libvips +through, gives no way to stop an encode once it has begun. So an encode keeps +its slot until it finishes, which can be after `downstream_timeout`. + +**Encode times.** The limits on image size bound how long a conversion can take. +They are 8192 on each side of the source and of the output, and 16,777,216 +pixels for an AVIF output. Decoding an 8192x8192 source takes up to about 11 +seconds. The WebP and JPEG XL encoders run below libvips' default effort. AVIF +runs at the lowest effort govips can set. With one libvips thread, as pixa runs +it, encoding 8192x8192 random pixels takes up to about: + +- 1 second as JPEG; +- 15 seconds as PNG; +- 16 seconds as GIF; +- 23 seconds as WebP with an alpha channel, and 16 without; +- 27 seconds as JPEG XL with an alpha channel, and 13 without. + +An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to +about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality +85 it would take about 96 seconds with an alpha channel, and 52 without. These +times hold at any quality and at 8 or 16 bits per sample. No encode within the +limits takes more than about 40 seconds of the default `downstream_timeout` +of 60. + +**Processing and connections.** `max_concurrent_processing` images are decoded +and encoded at once. A request that finds every slot in use waits up to 10 +seconds for one. If none frees up, it is answered 503 +`server busy, try again later`. `upstream_connections` bounds the fetches in +progress to all upstream hosts together, and `upstream_connections_per_host` +those to one host. A fetch holds its connection until its image has been +processed. A fetch that finds every shared connection in use waits up to 10 +seconds the same way, then answers 503. A fetch that finds every connection to +its host in use waits for one until `downstream_timeout` ends. + +**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute +per client address. An attempt over the limit is refused with 429 and a +`Retry-After` header. One client may send `image_requests_per_minute` requests +per minute (default 600) to `/v1/image/` and `/v1/e/` together, `GET` and +`HEAD`, cache hits included. A request over the limit is refused with 429 and a +`Retry-After` header. A browser's CORS preflight is not counted. `0` turns the +limit off. Both limits count an IPv6 client by its `/64`. + +**The client address.** Behind a reverse proxy, pixa takes the client address +from `X-Forwarded-For` only when the request arrives from an address in +`trusted_proxies`. Otherwise all users behind the proxy count as one client. The +address the request arrives from is not always the proxy's own. A proxy on the +Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of +the container's Docker network, such as `172.17.0.1` on the default bridge. A +proxy that connects through another of the host's addresses is seen with that +address. To be sure, find the address as described under `trusted_proxies` (see +Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the +private address ranges. With that default, a client with a private address is +itself trusted. It can then choose the address it is counted by, through its own +`X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes +this. Behind a content delivery network (CDN) or proxy whose addresses are not +in `trusted_proxies`, all requests arriving from one address count as one +client. Such a deployment needs a higher `image_requests_per_minute`, or `0`. + +**Request limits.** pixa reads a request's headers within 10 seconds and the +whole request within 30 seconds. Headers are limited to 8 KiB. An idle +connection is closed after 120 seconds. A form body over 1 MiB is refused +with 413. + +### Storage + +- **Sources**: `/cache/sources///`, one file for + each source URL +- **Database**: `/state.sqlite3` +- **Variants**: `/cache/variants///` + +`` and `` are the first and second pairs of characters of the file's +name. The database is one SQLite file. SQLite is the database library pixa +embeds; nothing else needs to run. By default the database is in WAL mode, +SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal` +and `state.sqlite3-shm` beside the file while pixa runs. + +The image cache keeps these tables in the database: + +- `source_content`: a row for each source, by source key, with its size and when + it was last used +- `variant_content`: a row for each variant, by variant key (column + `cache_key`), with its size and when it was last used +- `cache_usage`: the total size of the cached sources and variants, which + eviction reads +- `negative_cache`: a row for each source URL in the negative cache, with when + it expires + +The `logout` table holds the random value that a logout replaces (see Encrypted +URLs). + +pixa reads a cached source's format from its file's first bytes. When they match +none of the source formats, pixa fetches the source again. Two source URLs that +serve the same bytes are stored twice, once under each URL. A source is stored +by writing its file and then its row. When the row cannot be written, the file +is removed again. + +Each source and variant file is written under a temporary name and synced to +disk before it is renamed into place. So a crash never leaves part of a file +under its own name. + +When pixa starts, and then at the spacing below, it checks the cached files +against the database. It removes each source or variant whose file is empty, or +not the size its row records. It logs a warning naming the file. A later request +for that image fetches the source or makes the variant again. The check also +drops the rows of variants whose files are missing. When pixa cannot read a +variant file's size for another reason, it keeps the variant's row and logs a +warning naming the variant key and the error. During a check, pixa skips a +directory it cannot list and a file whose size it cannot read. It logs a warning +naming the path and the error, and goes on with the rest of the check. A file +renamed or removed after pixa listed its directory gets no warning. + +A check reads the database once for every cached file, so the checks are spaced +out. After a check ends, pixa waits at least ten times as long as that check +took. Eviction runs every five minutes (below). The next check starts at the +first of those runs after that wait. After a check that took one minute, the +next one starts 10 to 15 minutes after that check ended. + +A cache hit opens the variant's file and nothing else. It makes no database +call. The content type comes from the file's first bytes, which are those of the +format pixa wrote it in. A file whose first bytes are not those of a format pixa +writes is not served, and the image is converted again. A hit hands the time it +was served to a background writer and does not wait for the writer. Eviction +orders by that time. The hit is counted in memory, for the +`pixa_cache_hits_total` metric. A miss that uses a cached source also hands that +writer the time it used the source. Once a second the writer writes the waiting +times in one transaction. Up to 10,000 access times of variants, and as many of +sources, can wait to be written. A time handed over beyond that is dropped. Its +image then only looks less recently used to eviction. `script/bench` measures +how many hits pixa serves per second (see Load Test). + +Eviction removes the least recently used sources and variants until the cache is +within `cache_max_bytes`. It runs after each store, and every five minutes. It +also removes the expired rows of the negative cache every five minutes, at most +1000 per statement, so a request waits behind 1000 deletions at most. ### Configuration -Every setting can be given as an environment variable, in a YAML config file -(`--config`), or both. A variable present in the environment wins over the file, -even when it is empty, and the file wins over the built-in default. The one -exception is a variable set in the file's `env:` section. pixa sets that -variable while loading the file, so the variable wins over the environment pixa -started with. It also wins over the file's key for the same setting. A -variable's value is parsed as the same text in the file would be. A setting that -holds a list takes comma-separated entries, with the spaces around each trimmed. -An empty variable is an empty list. A value that does not parse or is invalid -aborts startup, naming the variable. pixa refuses to start when a variable -starts with `PIXA_` but is not in the list below, such as a misspelled one or -`PIXA_PORT`. The error names the variable, as it names an unknown config key. +Every setting can be given as an environment variable, in a config file +(`--config`), or both. The file is YAML, a plain-text format. A variable present +in the environment wins over the file, even when it is empty. The file wins over +the built-in default. pixa reads a variable's value as a quoted string in the +file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts +startup. A setting that holds a list takes comma-separated entries, with the +spaces around each trimmed. An empty variable is an empty list. + +The file's `env:` section is the one exception to that order. pixa sets each +variable named there while it loads the file. So that variable wins over the +environment pixa started with, and over the file's key for the same setting. + +pixa refuses to start when a value does not parse or is invalid. The error names +the setting. It also refuses to start when a key in the file is set to null. A +key written as `key:` with nothing after it is null. Omit the key to use the +default. It refuses to start when the file has a key that is not in the list +below, and the error names the key. It refuses to start when a variable starts +with `PIXA_` but is not in the list below, such as a misspelled one or +`PIXA_PORT`. The error names the variable. The variables set by the file's +`env:` section are checked the same way. + pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like -`--config`. The variables set by the file's `env:` section are checked the same -way. +`--config`. pixa reads at most one config file. It reads the file given with `--config` (or `-c`), or else the one `PIXA_CONFIG_PATH` names. pixa refuses to start when that @@ -758,188 +915,170 @@ parse, and the error names the file. It also refuses to start when it may not enter one of these directories, whether or not the file is there, since it cannot tell. With no file, pixa uses the environment and the defaults. +Durations are written as Go durations, such as `30s` or `2m`, and must be +positive. A bare number has no unit and aborts startup. Sizes are a whole number +of bytes. A CIDR range is an address range written as an address and a prefix +length, such as `10.0.0.0/8`. An address matches a range that contains it. + +`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6 +address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4 +address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain +IPv4 address. A range written in that form is read as the IPv4 range it covers: +`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers +more than IPv4 addresses and aborts startup. + Each item below is one setting, named by its config key: - `signing_key`: variable `PIXA_SIGNING_KEY`, required. The secret for signed - URLs, encrypted URLs and login. It must be 32 characters or longer. -- `port`: variable `PORT`, default `8080`. The port to listen on. + URLs, encrypted URLs and login. It must be 32 characters or longer. The + placeholder in `configs/config.example.yml` is refused. pixa uses the key as + it is for two things: as the HMAC key of signed URLs (see Signed URLs), and as + the password of the login page. From it, pixa derives four more keys at + startup, with HKDF-SHA256, a key derivation function. One signs the session + cookie and one encrypts it. One encrypts the token of an encrypted URL. One + signs the token cookie of the login and generator pages' forms. +- `port`: variable `PORT`, default `8080`. The port to listen on, from 1 + to 65535. - `state_dir`: variable `PIXA_STATE_DIR`, default `/var/lib/pixa`. The directory - for the database and the disk cache. -- `db_url`: variable `PIXA_DB_URL`, default `state.sqlite3` in the state - directory. The URL of the SQLite database. -- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default 75% of the sum of - the free space and what the cache holds. The disk cache limit in bytes. `0` - turns the disk cache off. + for the database and the disk cache. pixa creates it at startup when it is + missing. It refuses to start when it cannot create or write the directory. +- `db_url`: variable `PIXA_DB_URL`, default + `file:/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite + database to open. The default keeps the database in WAL mode, SQLite's + write-ahead log, which lets reads go on during a write. pixa adds + `_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another + program writing to the file then waits up to five seconds for it instead of + failing. pixa opens one connection, so its own reads and writes run one at a + time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in + one you set. An empty `db_url` aborts startup: omit the key to use the + default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in + the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one + starting with `//`, the default starts with `file:////`, so SQLite does not + read the first name in the path as a host. +- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk + cache limit in bytes. When the key is omitted, pixa works the limit out at + startup. It takes the free space on the filesystem holding + `/cache/`, adds the bytes the cache already holds, and takes 75% of + the sum. The default is at least 500 MiB. A value you set is used exactly as + given. `0` turns the disk cache off: every request then fetches and converts + its image, and only the negative cache is used. A negative value aborts + startup. - `allowlist_hosts`: variable `PIXA_ALLOWLIST_HOSTS`, default none. The upstream - hosts whose images pixa serves without a signature. -- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. The - image routes refuse a request with 403 when its `Referer` header names one of - these hosts. -- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. The CIDR - ranges pixa never fetches from, on top of the built-in ones. + hosts whose images pixa serves without a signature, written as under Source + Hosts. Every image on a listed host is served to anyone, so list only hosts + you control. An entry that is neither a host name nor an IP address aborts + startup. +- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts + whose pages may not show pixa's images, to stop other sites embedding them. + Entries are written and matched as for `allowlist_hosts`. A request whose + `Referer` header names a listed host is refused with 403. That applies to + `/v1/image/` and `/v1/e/`. It happens before the signature or token is + checked, and before the cache or the upstream host is used. The request is + refused even when the image is cached. A request with no `Referer` is served, + as many clients send none. So is one whose `Referer` does not parse as a URL + with a host. So a site whose pages send no `Referer`, for example with + `Referrer-Policy: no-referrer`, is not stopped. The login and generator pages + are not covered. +- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR + ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR + aborts startup. The built-in ranges always apply. They are: + - `0.0.0.0/8`, this network, `0.0.0.0` included; + - `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges; + - `100.64.0.0/10`, carrier-grade NAT; + - `127.0.0.0/8`, loopback; + - `169.254.0.0/16`, link-local; + - `192.0.0.0/24`, IETF protocol assignments; + - `198.18.0.0/15`, benchmarking; + - `224.0.0.0/4`, multicast; + - `::`, the unspecified address; + - `::1`, loopback; + - `64:ff9b::/96` and `64:ff9b:1::/48`, NAT64; + - `fc00::/7`, the unique local (private) range; + - `fe80::/10`, link-local; + - `ff00::/8`, multicast. - `trusted_proxies`: variable `PIXA_TRUSTED_PROXIES`, default the RFC 1918 - ranges. The CIDR ranges of the proxies whose `X-Forwarded-For` header pixa - believes. -- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. Allows plain-HTTP - upstream hosts. It is for testing only. + private ranges: `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`. CIDR + ranges of the reverse proxies in front of pixa. `X-Forwarded-For` is believed + only when the request reaches pixa from an address in one of these ranges. The + client address is then the rightmost entry of the header that is not itself a + trusted proxy. Otherwise the header is ignored and the address the request + came from is used. So a client connecting from outside these ranges cannot + choose its address. An explicitly empty list (`[]`, or an empty variable) + trusts no one. A list you set replaces the default. An invalid CIDR aborts + startup. Set this to the address pixa sees for requests that come through your + proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or + to trust nothing else (see Limits). To find that address, set this to `[]`, + send a request through the proxy, and read `remoteIP` in pixa's log line for + it. +- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa + fetches every source over plain HTTP, and follows redirects to plain HTTP. It + is for testing only. - `upstream_connections_per_host`: variable `PIXA_UPSTREAM_CONNECTIONS_PER_HOST`, default `20`. The most connections to - one upstream host at once. + one upstream host at once. It must be at least 1. A fetch that finds every + connection to its host in use waits for one until `downstream_timeout` ends + (see Limits). - `upstream_connections`: variable `PIXA_UPSTREAM_CONNECTIONS`, default `64`. - The most connections to all upstream hosts together at once. + The most connections to all upstream hosts together at once, on top of the + limit per host. It must be at least 1. A fetch holds its connection until its + image has been processed (see Limits). - `max_concurrent_processing`: variable `PIXA_MAX_CONCURRENT_PROCESSING`, - default half the CPUs, at least 1. The most images processed at once. + default half the CPUs Go uses (`GOMAXPROCS`), rounded down, and at least 1. + The most images decoded and encoded at once. `GOMAXPROCS` follows a + container's CPU limit. A value that is not a whole number of at least 1 aborts + startup. Each image can take a lot of memory (see Running under upaas). - `image_requests_per_minute`: variable `PIXA_IMAGE_REQUESTS_PER_MINUTE`, - default `600`. How many image requests one client may send per minute. `0` - turns the limit off. + default `600`. How many requests one client may send per minute to + `/v1/image/` and `/v1/e/` together (see Limits). `0` turns the limit off. A + negative or non-integer value aborts startup. - `upstream_fetch_timeout`: variable `PIXA_UPSTREAM_FETCH_TIMEOUT`, default - `30s`. The time allowed for one fetch from an upstream host. + `30s`. The time allowed for one fetch from an upstream host. A fetch whose + response headers have not arrived by then answers 504. - `upstream_max_response_size`: variable `PIXA_UPSTREAM_MAX_RESPONSE_SIZE`, - default 50 MiB. The largest upstream response accepted, in bytes. + default `52428800` (50 MiB). The largest upstream response accepted, in bytes, + from 1 to `1073741824` (1 GiB). It also limits the image data pixa decodes. A + cached source larger than it is fetched again. - `downstream_timeout`: variable `PIXA_DOWNSTREAM_TIMEOUT`, default `60s`. The - time allowed for answering one client request. + time allowed for answering one client request. Keep it longer than + `upstream_fetch_timeout` plus 20 seconds (see Limits). - `access_control_allow_origin`: variable `PIXA_ACCESS_CONTROL_ALLOW_ORIGIN`, - default `*`. The CORS origin allowed to read image responses. It is `*` or one - origin. + default `*`. The origin a browser lets read the responses of `/v1/image/` and + `/v1/e/`, sent as the CORS `Access-Control-Allow-Origin` header. No other + route sends it. `*` is any site. Otherwise it is one `http` or `https` origin + such as `https://example.com`. Its host is a lowercase host name (letters, + digits, hyphens and dots, with a letter in its last part) or an IP address + (IPv6 in brackets, in its shortest form). It may have a port from 1 to 65535 + that has no leading zero and is not the scheme's default. Any other value + aborts startup, including an origin with another scheme, such as a browser + extension's. - `metrics.username`: variable `PIXA_METRICS_USERNAME`, default empty. The - username for `/metrics`. pixa serves `/metrics` only when the username and the - password are both set. + username for `/metrics`. pixa serves `/metrics`, and measures every HTTP + request, only when the username and the password are both set. Setting one + without the other aborts startup. - `metrics.password`: variable `PIXA_METRICS_PASSWORD`, default empty. The password for `/metrics`. Set it together with the username. -- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. The Sentry DSN for - error reporting. An empty DSN turns error reporting off. +- `sentry_dsn`: variable `PIXA_SENTRY_DSN`, default empty. Sentry is an error + reporting service, and the DSN is the URL it gives you. A DSN turns error + reporting on. It must be a URL with a scheme and a host, or empty. - `debug`: variable `PIXA_DEBUG`, default `false`. Turns on debug logging and - plain-HTTP local development. + plain-HTTP local development (see Routes). - `maintenance_mode`: variable `PIXA_MAINTENANCE_MODE`, default `false`. While - `true`, pixa answers image requests with 503. The health check still - answers 200. + `true`, `/v1/image/` and `/v1/e/` answer every `GET` and `HEAD` request with + 503, a `Retry-After` header of 300 seconds and a JSON error body. The health + check still answers 200 and reports `"maintenance_mode": true`. It stays 200 + because the image's Docker `HEALTHCHECK` requests it, and upaas marks a deploy + failed when its container is unhealthy. The login and generator pages and + `/metrics` keep working. -Key settings in more detail: - -- `access_control_allow_origin` — the origin a browser lets read the responses - of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS - `Access-Control-Allow-Origin` header; no other route sends it. `*`, the - default, is any site; otherwise one `http` or `https` origin such as - `https://example.com`, whose host is a lowercase host name (letters, digits, - hyphens and dots, with a letter in its last part) or an IP address (IPv6 in - brackets, in its shortest form), with an optional port 1-65535 that has no - leading zero and is not the scheme's default. Any other value, including - another scheme such as a browser extension's, aborts startup -- `allowlist_hosts` — list of allowed upstream hosts -- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to - stop other sites hotlinking them. Entries are written and matched as for - `allowlist_hosts` (see Allowlist patterns), and an entry that is neither a - host name nor an IP address aborts startup. A request to `/v1/image/` or - `/v1/e/` whose `Referer` header names a listed host is refused with 403 before - its signature or token is checked and before the cache or the upstream host is - used, so it fetches nothing, and it is refused even when the image is cached. - A request with no `Referer`, or one that does not parse as a URL with a host, - is served, as many clients send none. So this is easily got around: a site - whose pages send no `Referer` (for example with - `Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login - and generator pages. Default: empty -- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added - to the always-enforced built-in ranges (loopback, private, link-local, CGNAT, - benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6 - address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4 - address it holds, so an IPv6 range such as `::/0` refuses neither it nor a - plain IPv4 address. An entry written in that form is read as the IPv4 range it - covers, so `::ffff:10.0.0.0/104` refuses what `10.0.0.0/8` refuses; such an - entry shorter than `/96` covers more than IPv4 addresses and aborts startup -- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of - pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one - of these ranges; the logged and login-recorded client address is then the - rightmost forwarded entry that is not itself a trusted proxy. Otherwise the - direct peer address is used and the header is ignored, so a client connecting - directly from an address outside these ranges cannot spoof its address. An - omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`, - `172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a - private network; an explicitly empty list (`[]`) trusts no one, and an - explicit list replaces the default. An invalid CIDR aborts startup. An IPv6 - address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4 - address it holds, so an IPv6 range such as `::/0` trusts neither it nor a - plain IPv4 address. An entry written in that form is read as the IPv4 range it - covers, so `::ffff:10.0.0.0/104` trusts what `10.0.0.0/8` trusts; such an - entry shorter than `/96` covers more than IPv4 addresses and aborts startup. - Set this to the address pixa sees for requests that come through your proxy, - such as `172.17.0.1/32`, when the defaults do not cover it, or to trust - nothing else (see the login limit and `image_requests_per_minute` under - Routes). For a proxy on the Docker host that connects to pixa over - `127.0.0.1`, that address is the gateway of the container's Docker network - (`172.17.0.1` on the default bridge), not the proxy's own address; a proxy - that connects through another of the host's addresses is seen with that - address. To be sure which address it is, set this to `[]` (or - `PIXA_TRUSTED_PROXIES` to empty), send a request through the proxy, and read - `remoteIP` in pixa's request log line for it -- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host, - as a duration such as `30s` (the default) or `2m` -- `upstream_max_response_size` — largest upstream response accepted, in bytes; - default `52428800` (50 MiB). It also limits the image data pixa decodes -- `downstream_timeout` — time allowed for answering one client request, as a - duration; default `60s`. The upstream fetch counts toward it, and so do the - waits for an upstream connection and for a processing slot (up to 10 seconds - each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds -- `signing_key` — the secret that protects signed URLs, encrypted URLs and - logins. pixa uses it as it is for two things: - - the HMAC key of signed URLs (see Signature Specification) - - the password of the login page - - pixa derives every other key it uses from it at startup, with HKDF-SHA256: - - a key that signs the session cookie - - a key that encrypts the session cookie - - a key that encrypts the token of an encrypted URL - - a key that signs the CSRF token cookie of the login and generator pages - -- `db_url` — the SQLite database to open; omitted, it is - `file:/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the - database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or - `#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/` - or one starting with `//` it starts `file:////`, so SQLite does not read the - first name in the path as a host. pixa opens one connection to it, so its own - reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to - any `db_url`, so a write that finds another program writing to the file waits - up to five seconds for it instead of failing. WAL mode comes only from the - URL: keep `_pragma=journal_mode(WAL)` in one you set -- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk - cache entirely; omitted defaults to 75% of the sum of the free space on the - filesystem containing `/cache/` and the bytes of source and - transformed images the cache already holds, worked out at startup (minimum 500 - MiB) -- `upstream_connections` — the most connections to upstream hosts at once, all - hosts together, on top of `upstream_connections_per_host`; default `64`. A - fetch holds its connection until its image has been processed. A fetch that - finds all of them in use waits up to 10 seconds for one to free up; if none - does, and `downstream_timeout` has not ended first, the request is answered - 503 with the error `server busy, try again later` -- `max_concurrent_processing` — the most images decoded and encoded at once; - default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's - CPU limit), rounded down, and at least 1. A value that is not a whole number - of at least 1 aborts startup. A request that finds all of them in use waits up - to 10 seconds for one to free up; if none does, and `downstream_timeout` has - not ended first, it is answered 503 the same way. Each one can take a lot of - memory: see "Running under upaas" -- `image_requests_per_minute` — how many requests one client may send to the - image routes per minute (see the limit on them under Routes); default `600`. - `0` turns the limit off, and a negative or non-integer value aborts startup. - Behind a CDN or proxy whose addresses are not in `trusted_proxies`, all - requests arriving from one proxy or CDN address count as one client, so raise - the limit there, or set `0` -- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and - `/v1/e/`) answer every request for an image with 503, a `Retry-After` header - and a JSON error body. The health check (`/.well-known/healthcheck.json`) - still answers 200 and reports `"maintenance_mode": true`. It stays 200 because - the image's Docker `HEALTHCHECK` requests it: a 503 there would make the - container unhealthy, and upaas marks a deploy failed when its container is - unhealthy. The login and URL generator pages and `/metrics` keep working - -See `configs/config.example.yml` for all options with defaults. +`configs/config.example.yml` names every setting with its default. ### Architecture -- **Dependency injection**: Uber fx. `internal/app` puts every part of pixad in - one fx graph. `internal/providers` builds the image cache, the image proxy and - the other parts that serve images and logins. Each setting's default is - declared once, in `internal/config`. +- **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the + program's parts in dependency order and starts and stops them. `internal/app` + puts every part of `pixad` in one `fx` graph. `internal/providers` builds the + image cache, the image proxy and the other parts that serve images and logins. + Each setting's default is declared once, in `internal/config`. - **Packages**: these packages under `internal/` work with the concepts above: - `handlers`: the routes, which read the image request from the URL, check the signed or encrypted URL, and make the login session @@ -955,12 +1094,13 @@ See `configs/config.example.yml` for all options with defaults. - `seal`: the keys derived from `signing_key`, and the encryption of those tokens - `session`: the login session -- **HTTP router**: go-chi -- **Image processing**: govips (CGO wrapper for libvips) -- **Database**: SQLite via modernc.org/sqlite -- **Static assets**: embedded via `//go:embed` -- **Metrics**: Prometheus, at `/metrics`. pixa serves `/metrics` and measures - HTTP requests only when `metrics.username` and `metrics.password` are set. +- **HTTP router**: `chi` (`github.com/go-chi/chi/v5`) +- **Image processing**: govips, a Go wrapper for libvips +- **Database**: SQLite via `modernc.org/sqlite` +- **Static assets**: embedded in the binary via `//go:embed` +- **Metrics**: Prometheus, at `/metrics`. Prometheus is a monitoring system that + reads metrics from that route. pixa serves `/metrics` and measures HTTP + requests only when `metrics.username` and `metrics.password` are set. `/metrics` holds: - the duration of HTTP requests - the size of HTTP responses @@ -976,75 +1116,91 @@ See `configs/config.example.yml` for all options with defaults. A request's `handler` label is the pattern of the route it matches, such as `/v1/image/*`, never its path. Requests that match no route share the label `unmatched`: those the router answers 404 or 405, and CORS preflights. A - request's `method` label is its method when that is `GET`, `HEAD`, `POST`, - `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for - any other method. + request's `method` label is its method, for these nine: `GET`, `HEAD`, + `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any + other method is labelled `other`. -- **Logging**: stdlib slog +- **Logging**: the standard library's `slog`, as JSON on standard output, or as + text when standard output is a terminal. ## Entrypoints This repository adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: normalized scripts in `script/` are the entrypoints for the -development workflow, and the Makefile targets are thin shims that call them. We -provide: +standard. The scripts in `script/` are the entrypoints for the development +workflow, and most `Makefile` targets are thin shims that call them. We provide: -- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and - download the Go modules (idempotent); with `--cgo`, the C compiler and the - libvips (with its JPEG XL support) and libheif libraries that compiling and - testing pixa need instead of Node, Yarn and prettier -- `script/setup` — make a fresh clone ready for development (bootstrap, then - install-precommit) -- `script/projectname` — output the project name ("pixa") +- `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and + download the Go modules. It can be run again at any time. With `--cgo`, it + installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and + libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker, + as `make build` and `script/bench` do, needs those. Docker itself must already + be installed. +- `script/setup` — make a fresh clone ready for development: `script/bootstrap`, + then `script/install-precommit`. +- `script/projectname` — output the project name (`pixa`). - `script/test` — run the test suite: build the `test` phase of the - `Dockerfile`, writing no image -- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`, - writing no image; the linter never runs on the host -- `script/fmt` — format the Go code with gofmt and the markdown with prettier - (writes) -- `script/fmt-check` — check the same formatting (read-only), on the host -- `script/check` — run test, lint, and fmt-check + `Dockerfile`, writing no image. +- `script/lint` — run `golangci-lint`: build the `lint` phase of the + `Dockerfile`, writing no image. The linter never runs on the host. +- `script/fmt` — format the Go code with `gofmt` and the markdown with + `prettier` (writes). +- `script/fmt-check` — check the same formatting (read-only), on the host. +- `script/check` — run `script/test`, `script/lint` and `script/fmt-check`. - `script/docker` — build the Docker image tagged via `script/projectname`, with - the version from `git describe`; the image's build stage depends on the `lint` - and `test` phases, so this runs them too -- `script/docker-smoke` — build the image, start it, wait for it to be healthy -- `script/loadtest` — measure pixad's throughput, latency and peak memory; a - benchmark, not part of `script/check` (see Load Test) + the version from `git describe`. The image's build stage depends on the `lint` + and `test` phases, so this runs them too. +- `script/docker-smoke` — build the image, start it, and wait for it to be + healthy. +- `script/loadtest` — measure `pixad`'s throughput, latency and peak memory. It + is a benchmark, not part of `script/check` (see Load Test). - `script/bench` — run the Go benchmark that serves cache hits in parallel - through the image handler, with Go on the host after `script/bootstrap --cgo`; - a benchmark, not part of `script/check` (see Load Test) -- `script/cibuild` — CI entrypoint: run `script/bootstrap` (without `--cgo`), - then `script/check`, then build the image as `script/docker` does -- `script/precommit` — pre-commit checks (`go mod tidy` guard, then - `script/check`) + through the image handler, with Go on the host after `script/bootstrap --cgo`. + It is a benchmark, not part of `script/check` (see Load Test). +- `script/cibuild` — the CI entrypoint: run `script/bootstrap` (without + `--cgo`), then `script/check`, then build the image as `script/docker` does. +- `script/precommit` — the pre-commit checks: the `go mod tidy` guard, then + `script/check`. - `script/install-precommit` — install the git pre-commit hook that runs - `script/precommit` + `script/precommit`. Every `docker build` in these scripts passes a new value of the `CACHEBUST` -build argument, which every stage of the `Dockerfile` declares, so every `RUN` -step after that line, the lint and test steps included, runs on every build -instead of coming from the build cache. Above the line are the steps that -install system packages and download the Go modules; Docker keeps their layers -until the base image, the step itself, `go.mod` or `go.sum` changes. Go's build -cache is kept from build to build: the steps that compile mount it, so a package -whose inputs are unchanged is not compiled again, and both `go test` runs pass -`-count=1` so that no stored test result is replayed. The lint step also mounts -golangci-lint's cache, which reuses a package's findings only when its files, -those of the packages it imports, `go.mod`, the linters and their settings in -`.golangci.yml`, and the golangci-lint version are all unchanged. The `test` -phase runs on the Debian Go image, with Debian's libvips and libheif; the image -pixa ships is built and runs on Alpine. `script/check`, `script/cibuild`, -`script/docker`, `script/lint`, `script/test`, `script/setup` and -`script/install-precommit` are the standard copies from `sneak/prompts`, kept -identical to them. `script/fmt` and `script/fmt-check` are the standard copies -with pixa's `gofmt` step kept before prettier. prettier formats the markdown -only, not the HTML templates, as it cannot parse a Go template action inside a -tag. +build argument. Every stage of the `Dockerfile` declares it with an +`ARG CACHEBUST` line. So every `RUN` step after that line, the lint and test +steps included, runs on every build instead of coming from the build cache. The +steps above that line install system packages and download the Go modules. +Docker keeps their layers until the base image, the step itself, `go.mod` or +`go.sum` changes. + +Go's build cache is kept from build to build: the steps that compile mount it, +so a package whose inputs are unchanged is not compiled again. The test step +runs `go test`, and reruns it with `-v` after a failure. Both runs pass +`-count=1`, so no stored test result is replayed. The lint step also mounts +`golangci-lint`'s cache. That cache reuses a package's findings only while its +files, its imports, `go.mod`, `.golangci.yml` and the `golangci-lint` version +are unchanged. + +The `test` phase runs on the Debian Go image, with Debian's libvips and libheif. +The image pixa ships is built and runs on Alpine. + +These scripts are the standard copies from `sneak/prompts`, kept identical to +them: + +- `script/check` +- `script/cibuild` +- `script/docker` +- `script/lint` +- `script/test` +- `script/setup` +- `script/install-precommit` + +`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt` +step kept before `prettier`. `prettier` formats the markdown only, not the HTML +templates, as it cannot parse a Go template action inside a tag. ## Load Test -`script/loadtest` (or `make loadtest`) measures how fast pixad answers and how +`script/loadtest` (or `make loadtest`) measures how fast `pixad` answers and how much memory it uses. It is a benchmark, not a check: `script/check` does not run it. It needs Docker and Go. @@ -1053,65 +1209,63 @@ script/loadtest # 10 seconds per scenario, 4 clients script/loadtest 30s 32 # 30 seconds per scenario, 32 clients ``` -It builds the image with `script/docker` and the load tool, +It builds the image with `script/docker`. It also builds the load tool, [vegeta](https://github.com/tsenart/vegeta), from a pinned commit. Each scenario -starts a new pixad container and a new origin container, `cmd/loadtest-origin`: -an upstream host that answers every path with the same generated 1600x1200 JPEG. -vegeta then sends requests from the given number of clients, each sending its -next request as soon as its last one is answered, all for an image resized to -400x300 WebP: +starts a new `pixad` container and a new origin container, +`cmd/loadtest-origin`. The origin is an upstream host that answers every path +with the same generated 1600x1200 JPEG. vegeta then sends requests from the +given number of clients. Each client sends its next request as soon as its last +one is answered. Every request asks for an image resized to 400x300 WebP: - `hit`: the same image every time, put in the cache first; -- `miss`: a new source image every time, so pixad fetches and converts each one; -- `herd`: each new source image once per client in a row, so that all clients - ask for it at the same time and share one fetch and one conversion (see - Routes). +- `miss`: a new source image every time, so `pixad` fetches and converts each + one; +- `herd`: every client asks for each new source image at the same moment. The + clients then share one fetch and one conversion (see Limits). -pixad refuses upstream hosts with private or local addresses, so the containers -share a Docker network in `203.0.113.0/24`, a range set aside for documentation. -A second run on the same Docker host while one is going fails, as it cannot -create that network. +`pixad` refuses upstream hosts with private or local addresses, so the +containers share a Docker network in `203.0.113.0/24`, a range set aside for +documentation. A second run on the same Docker host while one is going fails, as +it cannot create that network. For each scenario the script prints vegeta's report and two lines of its own: - `Requests [total, rate, throughput]`: the requests sent, how many were sent - per second, and how many were answered successfully per second; the last is - the number to compare with the target under Storage; + per second, and how many were answered successfully per second; - `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request - to the end of its answer; `50`, `95` and `99` are the 50th, 95th and 99th - percentiles; + to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th + and 99th percentiles; - `Status Codes` and `Error Set`: anything other than `200` means the other - numbers are not for the scenario described, such as `503` when pixad was busy; + numbers are not for the scenario described, such as `503` when `pixad` was + busy; - `Bytes In`: `0`, as vegeta is told not to keep the images it receives; -- `pixad peak memory (VmHWM)`: the peak resident memory of pixad's process since - its container started, in kB; for `hit` it includes the request that put the - image in the cache; -- `requests to the origin`: the fetches pixad made: one for `hit`, one per - request for `miss`, and one per image for `herd`, that is the requests sent +- `pixad peak memory (VmHWM)`: the peak resident memory of the `pixad` process + since its container started, in kB; for `hit` it includes the request that put + the image in the cache; +- `requests to the origin`: the fetches `pixad` made. That is one for `hit`, one + per request for `miss`, and one per image for `herd`: the requests sent divided by the number of clients. -The numbers depend on the machine and on whatever else runs on it. The first -measurement, made on a shared machine with few clients, is in `TODO.md`; it says -nothing about the target. +The numbers depend on the machine and on whatever else runs on it. `script/bench` (or `make bench`) measures cache hits alone, without the network -and with no pixad process: a Go benchmark in `internal/handlers` serves one -cached image through the `/v1/image/` handler from as many goroutines at once as -the machine has CPUs, and reports the hits served per second as `hits/s`. Each -run starts with an empty cache and an empty database, a SQLite file of its own, -and puts the image in the cache before it measures. The benchmark fails if -serving the hits ran any statement on the database. Like the load test it is not -part of `script/check`. It runs Go on the host, as the load test does, and -compiles pixa there, so it needs the C compiler and libraries that -`script/bootstrap --cgo` installs, but not Docker. +and with no `pixad` process. A Go benchmark in `internal/handlers` serves one +cached image through the `/v1/image/` handler from several goroutines at once, +one per CPU. A goroutine is a Go thread. The benchmark reports the hits served +per second as `hits/s`. Each run starts with an empty cache and an empty +database, a SQLite file of its own, and puts the image in the cache before it +measures. The benchmark fails if serving the hits ran any statement on the +database. Like the load test it is not part of `script/check`. It runs Go on the +host, as the load test does, and compiles pixa there. So it needs the C compiler +and libraries that `script/bootstrap --cgo` installs, but not Docker. ## TODO -See [TODO.md](TODO.md) for the full prioritized task list. +See [`TODO.md`](TODO.md) for the full prioritized task list. ## License -GPL-3.0. See [LICENSE](LICENSE). +GPL-3.0. See [`LICENSE`](LICENSE). ## Author