Author SHA1 Message Date
clawbot 8669b5cf04 Rewrite README.md sentence by sentence (closes #327)
check / check (push) Canceled after 0s
Rewrites README.md from scratch against the code on next, for
#327: what pixa is and how to
use it, deployment, design, the settings reference, then development
and testing. The Configuration list keeps one item per setting, as
TestReadmeNamesEverySetting reads it; the test is unchanged. The
statements where the old text disagreed with the code are listed in
the PR body.

Model: fable-5-1
2026-10-27 03:07:29 +00:00
+160 -156
View File
@@ -9,7 +9,7 @@ original image, makes the requested image from it, and serves that. It keeps
both on disk. Later requests for the same image are served from disk, without a both on disk. Later requests for the same image are served from disk, without a
fetch or a conversion. fetch or a conversion.
This document calls the server that holds an original image the upstream host. In this document, the upstream host is the server that holds an original image.
## Getting Started ## Getting Started
@@ -85,9 +85,9 @@ The reverse proxy must:
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That - wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
is the longest pixa takes to fetch, convert and send an image. is the longest pixa takes to fetch, convert and send an image.
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy does [`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server.
everything in this list without further settings, and gets its own TLS Caddy does everything in this list without further settings, and gets its own
certificate. TLS certificate.
The reverse proxy may also refuse `/metrics`, as the example does. Only a The reverse proxy may also refuse `/metrics`, as the example does. Only a
scraper that reaches pixa directly can then read it. pixa itself asks for scraper that reaches pixa directly can then read it. pixa itself asks for
@@ -112,9 +112,9 @@ database and the disk cache:
- In the container, the startup script gives the directory to the user pixa runs - In the container, the startup script gives the directory to the user pixa runs
as, uid 65532, and sets its mode to `750`. Outside the container, the user as, uid 65532, and sets its mode to `750`. Outside the container, the user
running pixa must be able to write it. running pixa must be able to write it.
- `cache_max_bytes` limits the cached images. The database and files still being - `cache_max_bytes` limits the cached images. The limit does not count the
written come on top. Eviction runs in the background, so the cache can pass database, or files still being written. Eviction runs in the background, so
the limit for a while. Leave room on the volume beyond it. the cache can pass the limit for a while. Leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at - Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
each start from the free space on the volume (see Configuration). Anything each start from the free space on the volume (see Configuration). Anything
else that fills or frees space on the volume moves that default. else that fills or frees space on the volume moves that default.
@@ -139,8 +139,8 @@ Outside Docker, pixa needs these at run time:
- libvips, the image library pixa converts images with, including its JPEG XL - libvips, the image library pixa converts images with, including its JPEG XL
support. pixa does not start without that support. On Alpine it is the support. pixa does not start without that support. On Alpine it is the
`vips-jxl` package. The nix and brew packages of libvips include it. So do the `vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do
apt packages from Debian 12 and Ubuntu 24.04 on. the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on.
- libheif, which libvips needs for AVIF. - libheif, which libvips needs for AVIF.
- the system's CA certificates, which HTTPS fetches need. - the system's CA certificates, which HTTPS fetches need.
@@ -401,9 +401,9 @@ An image URL has one of these forms. The second has no format:
`inside` or `outside` (below). It is `cover` when the URL has no `fit`. `inside` or `outside` (below). It is `cover` when the URL has no `fit`.
`q` and `fit` are part of what is cached, so each value of either is a separate `q` and `fit` are part of what is cached, so each value of either is a separate
cached image. A request whose query string cannot be decoded, or gives any cached image. A request is refused with 400 when its query string cannot be
parameter more than once, is refused with 400. So is a `q`, `fit` or `exp` that decoded, or when it gives any parameter more than once. A request with a `q`,
is empty or not valid. `fit` or `exp` that is empty or not valid is refused with 400 too.
pixa fetches sources over HTTPS only, and only from hosts with valid pixa fetches sources over HTTPS only, and only from hosts with valid
certificates. With `allow_http` set, it fetches every source over plain HTTP certificates. With `allow_http` set, it fetches every source over plain HTTP
@@ -522,11 +522,11 @@ of the variant key, in quotes.
### Signed URLs ### Signed URLs
A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
time when it expires, and `sig`, its signature. The signature is an HMAC-SHA256: time (seconds since 1 January 1970) when it expires, and `sig`, its signature.
a hash of the request's details, keyed with `signing_key`, that only someone who The signature is an HMAC-SHA256: a hash of the request's details, keyed with
knows the key can make. Signatures match exactly: every part of the signed data `signing_key`, that only someone who knows the key can make. Signatures match
must be exactly what was signed. There is no suffix, wildcard or partial exactly: every part of the signed data must be exactly what was signed. There is
matching. no suffix, wildcard or partial matching.
The signed data is one line of nine parts, joined with `:`: The signed data is one line of nine parts, joined with `:`:
@@ -552,15 +552,15 @@ HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit"
has the signature of the same URL ending in `.jxl`. `auto` is signed as has the signature of the same URL ending in `.jxl`. `auto` is signed as
`auto`, not as the format chosen for the request. `auto`, not as the format chosen for the request.
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature - `expiration`: the URL's `exp`, the Unix time in seconds when the signature
expires. A request whose `exp` is not a whole number, an empty `exp=` expires. A request whose `exp` is not a whole number is refused with 400. An
included, is refused with 400. empty `exp=` is one example.
- `quality`: the URL's `q`, or `85` when the URL has no `q`. - `quality`: the URL's `q`, or `85` when the URL has no `q`.
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`. - `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.
The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
compares it exactly. A signature encoded without padding, as Node's `base64url` compares it exactly. A signature encoded without padding is refused with 401.
and Go's `base64.RawURLEncoding` do, is refused with 401. Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding.
**Example.** With the signing key `example-signing-key-for-documentation`, **Example.** With the signing key `example-signing-key-for-documentation`,
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
@@ -614,16 +614,16 @@ The form's fields:
- The width and height. Both empty or `0` keep the source's size. When only one - The width and height. Both empty or `0` keep the source's size. When only one
of them is empty or `0`, that side is worked out from the source's proportions of them is empty or `0`, that side is worked out from the source's proportions
when the image is made (see Image URLs). when the image is made (see Image URLs).
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. It - The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`.
is JPEG XL unless another is chosen. A form sent with an empty format, or `jxl` is selected unless another is chosen. A form sent with an empty format,
none, also makes a JPEG XL URL. or none, also makes a JPEG XL URL.
- The quality, a whole number from 1 to 100. The page offers five values, and - The quality, a whole number from 1 to 100. The page offers five values: `25`,
`85` unless another is chosen. `50`, `70`, `85` and `100`. `85` is selected unless another is chosen.
- The fit, one of the five fit modes (see Image URLs). It is `cover` unless - The fit, one of the five fit modes (see Image URLs). `cover` is selected
another is chosen. unless another is chosen.
- How long the URL lasts, in seconds, or `0` for never. The page offers never, 1 - How long the URL lasts, in seconds, or `0` for never. The page offers seven
minute, 1 hour, 1 day, 1 week, 30 days and 1 year, and never unless another is choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is
chosen. selected unless another is chosen.
The page refuses a form with 400 when: The page refuses a form with 400 when:
@@ -682,9 +682,9 @@ dot. So an entry with a port, or a `*.` wildcard, aborts startup.
pixa lower-cases the host of a `/v1/image/` URL before anything else reads it. pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
So `CDN.Example.com` is matched, fetched, cached and counted against So `CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a `upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
port, as in `cdn.example.com:8443`, needs a signature even when the host is port needs a signature even when the host is allowlisted. So does one whose host
allowlisted. So does one whose host ends in `:` with no port. The host must be ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host
one of: must be one of:
- a host name or an IPv4 address: letters, digits, hyphens, underscores and - a host name or an IPv4 address: letters, digits, hyphens, underscores and
dots; dots;
@@ -701,7 +701,7 @@ the output. This cannot be turned off. The metadata removed is:
- EXIF, which can hold the GPS position, the camera make, model and serial - EXIF, which can hold the GPS position, the camera make, model and serial
number, the capture time and an embedded thumbnail; number, the capture time and an embedded thumbnail;
- XMP and IPTC; - XMP and IPTC, two other metadata formats;
- the ICC colour profile. - the ICC colour profile.
The rest of the rules: The rest of the rules:
@@ -710,9 +710,9 @@ The rest of the rules:
`orig` image is re-encoded and stripped like any other. `orig` image is re-encoded and stripped like any other.
- An image with an EXIF orientation is turned upright first, so it displays the - An image with an EXIF orientation is turned upright first, so it displays the
same without the tag. A requested size applies to the upright image. same without the tag. A requested size applies to the upright image.
- An image with an ICC profile is converted to sRGB first, since clients show an - An image with an ICC profile is first converted to sRGB, the standard colour
image with no profile as sRGB. Colours outside sRGB are clipped: the most space, since clients show an image with no profile as sRGB. Colours outside
saturated ones in a Display P3 photo, for example. sRGB are clipped: the most saturated ones in a Display P3 photo, for example.
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then - JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
writes an EXIF block of its own, which pixa cannot stop. The block holds the writes an EXIF block of its own, which pixa cannot stop. The block holds the
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi. image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
@@ -739,13 +739,13 @@ with `downstream_timeout`. Keep `downstream_timeout` longer than
`upstream_fetch_timeout` plus 20 seconds. `upstream_fetch_timeout` plus 20 seconds.
A conversion whose request's `downstream_timeout` has ended stops waiting for a A conversion whose request's `downstream_timeout` has ended stops waiting for a
processing slot. Once it has one, it gives the slot back without beginning to processing slot. If its `downstream_timeout` ends after it has a slot but before
encode. For a shared conversion, the first request's deadline counts. When a it begins to encode, it gives the slot back. For a shared conversion, the first
shared conversion stops this way, each request waiting for it is answered 503 request's deadline counts. When a shared conversion stops this way, each request
`server busy, try again later`, even if its own `downstream_timeout` has not waiting for it is answered 503 `server busy, try again later`, even if its own
ended. govips, the library pixa drives libvips through, gives no way to stop an `downstream_timeout` has not ended. govips, the library pixa drives libvips
encode once it has begun. So an encode keeps its slot until it finishes, which through, gives no way to stop an encode once it has begun. So an encode keeps
can be after `downstream_timeout`. its slot until it finishes, which can be after `downstream_timeout`.
**Encode times.** The limits on image size bound how long a conversion can take. **Encode times.** The limits on image size bound how long a conversion can take.
They are 8192 on each side of the source and of the output, and 16,777,216 They are 8192 on each side of the source and of the output, and 16,777,216
@@ -762,20 +762,20 @@ it, encoding 8192x8192 random pixels takes up to about:
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
85 it would take about 96 seconds. These times hold at any quality and at 8 or 85 it would take about 96 seconds with an alpha channel, and 52 without. These
16 bits per sample. No encode within the limits takes more than about 40 seconds times hold at any quality and at 8 or 16 bits per sample. No encode within the
of the default `downstream_timeout` of 60. limits takes more than about 40 seconds of the default `downstream_timeout`
of 60.
**Processing and connections.** `max_concurrent_processing` images are decoded **Processing and connections.** `max_concurrent_processing` images are decoded
and encoded at once. A request that finds every slot in use waits up to 10 and encoded at once. A request that finds every slot in use waits up to 10
seconds for one. If none frees up, it is answered 503 seconds for one. If none frees up, it is answered 503
`server busy, try again later`, unless `downstream_timeout` has ended first. `server busy, try again later`. `upstream_connections` bounds the fetches in
`upstream_connections` bounds the fetches in progress to all upstream hosts progress to all upstream hosts together, and `upstream_connections_per_host`
together, and `upstream_connections_per_host` those to one host. A fetch holds those to one host. A fetch holds its connection until its image has been
its connection until its image has been processed. A fetch that finds every processed. A fetch that finds every shared connection in use waits up to 10
shared connection in use waits up to 10 seconds the same way, then answers 503. seconds the same way, then answers 503. A fetch that finds every connection to
A fetch that finds every connection to its host in use waits for one until its host in use waits for one until `downstream_timeout` ends.
`downstream_timeout` ends.
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute **Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
per client address. An attempt over the limit is refused with 429 and a per client address. An attempt over the limit is refused with 429 and a
@@ -789,18 +789,17 @@ limit off. Both limits count an IPv6 client by its `/64`.
from `X-Forwarded-For` only when the request arrives from an address in from `X-Forwarded-For` only when the request arrives from an address in
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The `trusted_proxies`. Otherwise all users behind the proxy count as one client. The
address the request arrives from is not always the proxy's own. A proxy on the address the request arrives from is not always the proxy's own. A proxy on the
Docker host that connects to pixa over `127.0.0.1` is seen as the gateway of the Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of
container's Docker network, such as `172.17.0.1` on the default bridge. A proxy the container's Docker network, such as `172.17.0.1` on the default bridge. A
that connects through another of the host's addresses is seen with that address. proxy that connects through another of the host's addresses is seen with that
To be sure, read the address as `remoteIP` in pixa's request log while it is not address. To be sure, find the address as described under `trusted_proxies` (see
in `trusted_proxies` (see `trusted_proxies` under Configuration). The default Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the
`trusted_proxies` is the RFC 1918 ranges, the private address ranges. With that private address ranges. With that default, a client with a private address is
default, a client with a private address can choose the address it is counted itself trusted. It can then choose the address it is counted by, through its own
by, through its own `X-Forwarded-For`, because its own address is trusted too. `X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes
Setting `trusted_proxies` to only the proxy's address closes this. Behind a CDN this. Behind a content delivery network (CDN) or proxy whose addresses are not
or proxy whose addresses are not in `trusted_proxies`, all requests arriving in `trusted_proxies`, all requests arriving from one address count as one
from one address count as one client. Such a deployment needs a higher client. Such a deployment needs a higher `image_requests_per_minute`, or `0`.
`image_requests_per_minute`, or `0`.
**Request limits.** pixa reads a request's headers within 10 seconds and the **Request limits.** pixa reads a request's headers within 10 seconds and the
whole request within 30 seconds. Headers are limited to 8 KiB. An idle whole request within 30 seconds. Headers are limited to 8 KiB. An idle
@@ -816,8 +815,9 @@ with 413.
`<ab>` and `<cd>` are the first and second pairs of characters of the file's `<ab>` and `<cd>` are the first and second pairs of characters of the file's
name. The database is one SQLite file. SQLite is the database library pixa name. The database is one SQLite file. SQLite is the database library pixa
embeds; nothing else needs to run. In WAL mode, the default, SQLite keeps embeds; nothing else needs to run. By default the database is in WAL mode,
`state.sqlite3-wal` and `state.sqlite3-shm` beside it while pixa runs. SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal`
and `state.sqlite3-shm` beside the file while pixa runs.
The image cache keeps these tables in the database: The image cache keeps these tables in the database:
@@ -855,22 +855,23 @@ naming the path and the error, and goes on with the rest of the check. A file
renamed or removed after pixa listed its directory gets no warning. renamed or removed after pixa listed its directory gets no warning.
A check reads the database once for every cached file, so the checks are spaced A check reads the database once for every cached file, so the checks are spaced
out. After a check ends, the next one waits at least ten times as long as that out. After a check ends, pixa waits at least ten times as long as that check
check took. After a check that took one minute, the next one starts 10 to 15 took. Eviction runs every five minutes (below). The next check starts at the
minutes after that check ended. first of those runs after that wait. After a check that took one minute, the
next one starts 10 to 15 minutes after that check ended.
A cache hit opens the variant's file and nothing else. It makes no database A cache hit opens the variant's file and nothing else. It makes no database
call. The content type comes from the file's first bytes, which are those of the call. The content type comes from the file's first bytes, which are those of the
format pixa wrote it in. A file whose first bytes are not those of a format pixa format pixa wrote it in. A file whose first bytes are not those of a format pixa
writes is not served, and the image is converted again. A hit hands the time it writes is not served, and the image is converted again. A hit hands the time it
was served to a background writer and does not wait for the writer. Eviction was served to a background writer and does not wait for the writer. Eviction
orders by that time. The hit is counted in memory. A miss that uses a cached orders by that time. The hit is counted in memory, for the
source also hands that writer the time it used the source. Once a second the `pixa_cache_hits_total` metric. A miss that uses a cached source also hands that
writer writes the waiting times in one transaction. Up to 10,000 access times of writer the time it used the source. Once a second the writer writes the waiting
variants, and as many of sources, can wait to be written. A time handed over times in one transaction. Up to 10,000 access times of variants, and as many of
beyond that is dropped. Its image then only looks less recently used to sources, can wait to be written. A time handed over beyond that is dropped. Its
eviction. `script/bench` measures how many hits pixa serves per second (see Load image then only looks less recently used to eviction. `script/bench` measures
Test). how many hits pixa serves per second (see Load Test).
Eviction removes the least recently used sources and variants until the cache is Eviction removes the least recently used sources and variants until the cache is
within `cache_max_bytes`. It runs after each store, and every five minutes. It within `cache_max_bytes`. It runs after each store, and every five minutes. It
@@ -879,25 +880,26 @@ also removes the expired rows of the negative cache every five minutes, at most
### Configuration ### Configuration
Every setting can be given as an environment variable, in a YAML config file Every setting can be given as an environment variable, in a config file
(`--config`), or both. A variable present in the environment wins over the file, (`--config`), or both. The file is YAML, a plain-text format. A variable present
even when it is empty. The file wins over the built-in default. A variable's in the environment wins over the file, even when it is empty. The file wins over
value is read as the same text in quotes in the file would be. A setting that the built-in default. pixa reads a variable's value as a quoted string in the
holds a list takes comma-separated entries, with the spaces around each trimmed. file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts
An empty variable is an empty list. startup. A setting that holds a list takes comma-separated entries, with the
spaces around each trimmed. An empty variable is an empty list.
The file's `env:` section is the one exception to that order. pixa sets each The file's `env:` section is the one exception to that order. pixa sets each
variable named there while it loads the file. So that variable wins over the variable named there while it loads the file. So that variable wins over the
environment pixa started with, and over the file's key for the same setting. environment pixa started with, and over the file's key for the same setting.
pixa refuses to start when a value does not parse or is invalid. The error names pixa refuses to start when a value does not parse or is invalid. The error names
the setting. It also refuses to start when a key in the file is set to null, the setting. It also refuses to start when a key in the file is set to null. A
`key:` with nothing after it included. Omit the key to use the default. It key written as `key:` with nothing after it is null. Omit the key to use the
refuses to start when the file has a key that is not in the list below, and the default. It refuses to start when the file has a key that is not in the list
error names the key. It refuses to start when a variable starts with `PIXA_` but below, and the error names the key. It refuses to start when a variable starts
is not in the list below, such as a misspelled one or `PIXA_PORT`. The error with `PIXA_` but is not in the list below, such as a misspelled one or
names the variable. The variables set by the file's `env:` section are checked `PIXA_PORT`. The error names the variable. The variables set by the file's
the same way. `env:` section are checked the same way.
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`. `--config`.
@@ -916,7 +918,14 @@ cannot tell. With no file, pixa uses the environment and the defaults.
Durations are written as Go durations, such as `30s` or `2m`, and must be Durations are written as Go durations, such as `30s` or `2m`, and must be
positive. A bare number has no unit and aborts startup. Sizes are a whole number positive. A bare number has no unit and aborts startup. Sizes are a whole number
of bytes. A CIDR range is an address range written as an address and a prefix of bytes. A CIDR range is an address range written as an address and a prefix
length, such as `10.0.0.0/8`. length, such as `10.0.0.0/8`. An address matches a range that contains it.
`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain
IPv4 address. A range written in that form is read as the IPv4 range it covers:
`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers
more than IPv4 addresses and aborts startup.
Each item below is one setting, named by its config key: Each item below is one setting, named by its config key:
@@ -936,16 +945,16 @@ Each item below is one setting, named by its config key:
- `db_url`: variable `PIXA_DB_URL`, default - `db_url`: variable `PIXA_DB_URL`, default
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
database to open. The default keeps the database in WAL mode, SQLite's database to open. The default keeps the database in WAL mode, SQLite's
write-ahead log, which lets reads go on during a write. In the default, write-ahead log, which lets reads go on during a write. pixa adds
`state_dir` is percent-encoded: a `%`, `?` or `#` in the name is written as `_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another
`%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one starting with `//`, program writing to the file then waits up to five seconds for it instead of
the default starts with `file:////`, so SQLite does not read the first name in failing. pixa opens one connection, so its own reads and writes run one at a
the path as a host. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`. A time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
write that finds another program writing to the file then waits up to five one you set. An empty `db_url` aborts startup: omit the key to use the
seconds for it instead of failing. pixa opens one connection, so its own reads default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in
and writes run one at a time. WAL mode comes only from the URL: keep the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one
`_pragma=journal_mode(WAL)` in one you set. An empty `db_url` aborts startup: starting with `//`, the default starts with `file:////`, so SQLite does not
omit the key to use the default. read the first name in the path as a host.
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk - `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
cache limit in bytes. When the key is omitted, pixa works the limit out at cache limit in bytes. When the key is omitted, pixa works the limit out at
startup. It takes the free space on the filesystem holding startup. It takes the free space on the filesystem holding
@@ -960,23 +969,19 @@ Each item below is one setting, named by its config key:
you control. An entry that is neither a host name nor an IP address aborts you control. An entry that is neither a host name nor an IP address aborts
startup. startup.
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts - `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
whose pages may not show pixa's images, to stop other sites hotlinking them. whose pages may not show pixa's images, to stop other sites embedding them.
Entries are written and matched as for `allowlist_hosts`. A request to Entries are written and matched as for `allowlist_hosts`. A request whose
`/v1/image/` or `/v1/e/` whose `Referer` header names a listed host is refused `Referer` header names a listed host is refused with 403. That applies to
with 403. That happens before its signature or token is checked, and before `/v1/image/` and `/v1/e/`. It happens before the signature or token is
the cache or the upstream host is used. It is refused even when the image is checked, and before the cache or the upstream host is used. The request is
cached. A request with no `Referer`, or one that does not parse as a URL with refused even when the image is cached. A request with no `Referer` is served,
a host, is served, as many clients send none. So a site whose pages send no as many clients send none. So is one whose `Referer` does not parse as a URL
`Referer`, for example with `Referrer-Policy: no-referrer`, is not stopped. with a host. So a site whose pages send no `Referer`, for example with
The login and generator pages are not covered. `Referrer-Policy: no-referrer`, is not stopped. The login and generator pages
are not covered.
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR - `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
aborts startup. An IPv6 address in the IPv4-mapped form (`::ffff:a.b.c.d`) is aborts startup. The built-in ranges always apply. They are:
matched only as the IPv4 address it holds. So an IPv6 range such as `::/0`
refuses neither it nor a plain IPv4 address. An entry written in that form is
read as the IPv4 range it covers: `::ffff:10.0.0.0/104` refuses what
`10.0.0.0/8` refuses. Such an entry shorter than `/96` covers more than IPv4
addresses and aborts startup. The built-in ranges always apply. They are:
- `0.0.0.0/8`, this network, `0.0.0.0` included; - `0.0.0.0/8`, this network, `0.0.0.0` included;
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges; - `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
- `100.64.0.0/10`, carrier-grade NAT; - `100.64.0.0/10`, carrier-grade NAT;
@@ -1000,12 +1005,11 @@ Each item below is one setting, named by its config key:
came from is used. So a client connecting from outside these ranges cannot came from is used. So a client connecting from outside these ranges cannot
choose its address. An explicitly empty list (`[]`, or an empty variable) choose its address. An explicitly empty list (`[]`, or an empty variable)
trusts no one. A list you set replaces the default. An invalid CIDR aborts trusts no one. A list you set replaces the default. An invalid CIDR aborts
startup. An IPv4-mapped entry or address is handled as under startup. Set this to the address pixa sees for requests that come through your
`blocked_networks`. Set this to the address pixa sees for requests that come proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or
through your proxy, such as `172.17.0.1/32`. Do so when the default does not to trust nothing else (see Limits). To find that address, set this to `[]`,
cover it, or to trust nothing else (see Limits). To find that address, set send a request through the proxy, and read `remoteIP` in pixa's log line for
this to `[]`, send a request through the proxy, and read `remoteIP` in pixa's it.
log line for it.
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa - `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
fetches every source over plain HTTP, and follows redirects to plain HTTP. It fetches every source over plain HTTP, and follows redirects to plain HTTP. It
is for testing only. is for testing only.
@@ -1070,11 +1074,11 @@ Each item below is one setting, named by its config key:
### Architecture ### Architecture
- **Dependency injection**: Uber fx, a library that builds the program's parts - **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the
in dependency order and starts and stops them. `internal/app` puts every part program's parts in dependency order and starts and stops them. `internal/app`
of `pixad` in one fx graph. `internal/providers` builds the image cache, the puts every part of `pixad` in one `fx` graph. `internal/providers` builds the
image proxy and the other parts that serve images and logins. Each setting's image cache, the image proxy and the other parts that serve images and logins.
default is declared once, in `internal/config`. Each setting's default is declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above: - **Packages**: these packages under `internal/` work with the concepts above:
- `handlers`: the routes, which read the image request from the URL, check - `handlers`: the routes, which read the image request from the URL, check
the signed or encrypted URL, and make the login session the signed or encrypted URL, and make the login session
@@ -1090,7 +1094,7 @@ Each item below is one setting, named by its config key:
- `seal`: the keys derived from `signing_key`, and the encryption of those - `seal`: the keys derived from `signing_key`, and the encryption of those
tokens tokens
- `session`: the login session - `session`: the login session
- **HTTP router**: go-chi - **HTTP router**: `chi` (`github.com/go-chi/chi/v5`)
- **Image processing**: govips, a Go wrapper for libvips - **Image processing**: govips, a Go wrapper for libvips
- **Database**: SQLite via `modernc.org/sqlite` - **Database**: SQLite via `modernc.org/sqlite`
- **Static assets**: embedded in the binary via `//go:embed` - **Static assets**: embedded in the binary via `//go:embed`
@@ -1112,9 +1116,9 @@ Each item below is one setting, named by its config key:
A request's `handler` label is the pattern of the route it matches, such as A request's `handler` label is the pattern of the route it matches, such as
`/v1/image/*`, never its path. Requests that match no route share the label `/v1/image/*`, never its path. Requests that match no route share the label
`unmatched`: those the router answers 404 or 405, and CORS preflights. A `unmatched`: those the router answers 404 or 405, and CORS preflights. A
request's `method` label is its method when that is `GET`, `HEAD`, `POST`, request's `method` label is its method, for these nine: `GET`, `HEAD`,
`PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any
any other method. other method is labelled `other`.
- **Logging**: the standard library's `slog`, as JSON on standard output, or as - **Logging**: the standard library's `slog`, as JSON on standard output, or as
text when standard output is a terminal. text when standard output is a terminal.
@@ -1124,25 +1128,25 @@ Each item below is one setting, named by its config key:
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for the development standard. The scripts in `script/` are the entrypoints for the development
workflow, and most Makefile targets are thin shims that call them. We provide: workflow, and most `Makefile` targets are thin shims that call them. We provide:
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier, and - `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and
download the Go modules. It can be run again at any time. With `--cgo`, it download the Go modules. It can be run again at any time. With `--cgo`, it
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
libheif instead of Node, Yarn and prettier. Compiling pixa outside Docker, as libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker,
`make build` and `script/bench` do, needs those. Docker itself must already be as `make build` and `script/bench` do, needs those. Docker itself must already
installed. be installed.
- `script/setup` — make a fresh clone ready for development: bootstrap, then - `script/setup` — make a fresh clone ready for development: `script/bootstrap`,
install-precommit. then `script/install-precommit`.
- `script/projectname` — output the project name (`pixa`). - `script/projectname` — output the project name (`pixa`).
- `script/test` — run the test suite: build the `test` phase of the - `script/test` — run the test suite: build the `test` phase of the
`Dockerfile`, writing no image. `Dockerfile`, writing no image.
- `script/lint` — run `golangci-lint`: build the `lint` phase of the - `script/lint` — run `golangci-lint`: build the `lint` phase of the
`Dockerfile`, writing no image. The linter never runs on the host. `Dockerfile`, writing no image. The linter never runs on the host.
- `script/fmt` — format the Go code with `gofmt` and the markdown with prettier - `script/fmt` — format the Go code with `gofmt` and the markdown with
(writes). `prettier` (writes).
- `script/fmt-check` — check the same formatting (read-only), on the host. - `script/fmt-check` — check the same formatting (read-only), on the host.
- `script/check` — run test, lint, and fmt-check. - `script/check` — run `script/test`, `script/lint` and `script/fmt-check`.
- `script/docker` — build the Docker image tagged via `script/projectname`, with - `script/docker` — build the Docker image tagged via `script/projectname`, with
the version from `git describe`. The image's build stage depends on the `lint` the version from `git describe`. The image's build stage depends on the `lint`
and `test` phases, so this runs them too. and `test` phases, so this runs them too.
@@ -1191,7 +1195,7 @@ them:
- `script/install-precommit` - `script/install-precommit`
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt` `script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
step kept before prettier. prettier formats the markdown only, not the HTML step kept before `prettier`. `prettier` formats the markdown only, not the HTML
templates, as it cannot parse a Go template action inside a tag. templates, as it cannot parse a Go template action inside a tag.
## Load Test ## Load Test
@@ -1229,8 +1233,8 @@ For each scenario the script prints vegeta's report and two lines of its own:
- `Requests [total, rate, throughput]`: the requests sent, how many were sent - `Requests [total, rate, throughput]`: the requests sent, how many were sent
per second, and how many were answered successfully per second; per second, and how many were answered successfully per second;
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request - `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
to the end of its answer. `50`, `95` and `99` are the 50th, 95th and 99th to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th
percentiles; and 99th percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other - `Status Codes` and `Error Set`: anything other than `200` means the other
numbers are not for the scenario described, such as `503` when `pixad` was numbers are not for the scenario described, such as `503` when `pixad` was
busy; busy;
@@ -1246,22 +1250,22 @@ The numbers depend on the machine and on whatever else runs on it.
`script/bench` (or `make bench`) measures cache hits alone, without the network `script/bench` (or `make bench`) measures cache hits alone, without the network
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
cached image through the `/v1/image/` handler from as many goroutines at once as cached image through the `/v1/image/` handler from several goroutines at once,
the machine has CPUs. It reports the hits served per second as `hits/s`. Each one per CPU. A goroutine is a Go thread. The benchmark reports the hits served
run starts with an empty cache and an empty database, a SQLite file of its own, per second as `hits/s`. Each run starts with an empty cache and an empty
and puts the image in the cache before it measures. The benchmark fails if database, a SQLite file of its own, and puts the image in the cache before it
serving the hits ran any statement on the database. Like the load test it is not measures. The benchmark fails if serving the hits ran any statement on the
part of `script/check`. It runs Go on the host, as the load test does, and database. Like the load test it is not part of `script/check`. It runs Go on the
compiles pixa there. So it needs the C compiler and libraries that host, as the load test does, and compiles pixa there. So it needs the C compiler
`script/bootstrap --cgo` installs, but not Docker. and libraries that `script/bootstrap --cgo` installs, but not Docker.
## TODO ## TODO
See [TODO.md](TODO.md) for the full prioritized task list. See [`TODO.md`](TODO.md) for the full prioritized task list.
## License ## License
GPL-3.0. See [LICENSE](LICENSE). GPL-3.0. See [`LICENSE`](LICENSE).
## Author ## Author