Author SHA1 Message Date
clawbot 44f3192a03 Rewrite README.md sentence by sentence (closes #327)
check / check (push) Canceled after 0s
Rewrites README.md from scratch against the code on next, for
#327: what pixa is and how to use
it, deployment, design, the settings reference, then development and
testing. The `### Configuration` heading and its one-item-per-setting
list stay as TestReadmeNamesEverySetting reads them.

Rework after review: all 38 findings of
#345 (comment) are
addressed, the decode and encode times are restored, and 13 further
sentences were fixed in a full pass.

Model: fable-5-1
2026-10-27 02:29:34 +00:00
+156 -160
View File
@@ -9,7 +9,7 @@ original image, makes the requested image from it, and serves that. It keeps
both on disk. Later requests for the same image are served from disk, without a both on disk. Later requests for the same image are served from disk, without a
fetch or a conversion. fetch or a conversion.
In this document, the upstream host is the server that holds an original image. This document calls the server that holds an original image the upstream host.
## Getting Started ## Getting Started
@@ -85,9 +85,9 @@ The reverse proxy must:
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That - wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
is the longest pixa takes to fetch, convert and send an image. is the longest pixa takes to fetch, convert and send an image.
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server. [`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy does
Caddy does everything in this list without further settings, and gets its own everything in this list without further settings, and gets its own TLS
TLS certificate. certificate.
The reverse proxy may also refuse `/metrics`, as the example does. Only a The reverse proxy may also refuse `/metrics`, as the example does. Only a
scraper that reaches pixa directly can then read it. pixa itself asks for scraper that reaches pixa directly can then read it. pixa itself asks for
@@ -112,9 +112,9 @@ database and the disk cache:
- In the container, the startup script gives the directory to the user pixa runs - In the container, the startup script gives the directory to the user pixa runs
as, uid 65532, and sets its mode to `750`. Outside the container, the user as, uid 65532, and sets its mode to `750`. Outside the container, the user
running pixa must be able to write it. running pixa must be able to write it.
- `cache_max_bytes` limits the cached images. The limit does not count the - `cache_max_bytes` limits the cached images. The database and files still being
database, or files still being written. Eviction runs in the background, so written come on top. Eviction runs in the background, so the cache can pass
the cache can pass the limit for a while. Leave room on the volume beyond it. the limit for a while. Leave room on the volume beyond it.
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at - Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
each start from the free space on the volume (see Configuration). Anything each start from the free space on the volume (see Configuration). Anything
else that fills or frees space on the volume moves that default. else that fills or frees space on the volume moves that default.
@@ -139,8 +139,8 @@ Outside Docker, pixa needs these at run time:
- libvips, the image library pixa converts images with, including its JPEG XL - libvips, the image library pixa converts images with, including its JPEG XL
support. pixa does not start without that support. On Alpine it is the support. pixa does not start without that support. On Alpine it is the
`vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do `vips-jxl` package. The nix and brew packages of libvips include it. So do the
the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on. apt packages from Debian 12 and Ubuntu 24.04 on.
- libheif, which libvips needs for AVIF. - libheif, which libvips needs for AVIF.
- the system's CA certificates, which HTTPS fetches need. - the system's CA certificates, which HTTPS fetches need.
@@ -401,9 +401,9 @@ An image URL has one of these forms. The second has no format:
`inside` or `outside` (below). It is `cover` when the URL has no `fit`. `inside` or `outside` (below). It is `cover` when the URL has no `fit`.
`q` and `fit` are part of what is cached, so each value of either is a separate `q` and `fit` are part of what is cached, so each value of either is a separate
cached image. A request is refused with 400 when its query string cannot be cached image. A request whose query string cannot be decoded, or gives any
decoded, or when it gives any parameter more than once. A request with a `q`, parameter more than once, is refused with 400. So is a `q`, `fit` or `exp` that
`fit` or `exp` that is empty or not valid is refused with 400 too. is empty or not valid.
pixa fetches sources over HTTPS only, and only from hosts with valid pixa fetches sources over HTTPS only, and only from hosts with valid
certificates. With `allow_http` set, it fetches every source over plain HTTP certificates. With `allow_http` set, it fetches every source over plain HTTP
@@ -522,11 +522,11 @@ of the variant key, in quotes.
### Signed URLs ### Signed URLs
A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
time (seconds since 1 January 1970) when it expires, and `sig`, its signature. time when it expires, and `sig`, its signature. The signature is an HMAC-SHA256:
The signature is an HMAC-SHA256: a hash of the request's details, keyed with a hash of the request's details, keyed with `signing_key`, that only someone who
`signing_key`, that only someone who knows the key can make. Signatures match knows the key can make. Signatures match exactly: every part of the signed data
exactly: every part of the signed data must be exactly what was signed. There is must be exactly what was signed. There is no suffix, wildcard or partial
no suffix, wildcard or partial matching. matching.
The signed data is one line of nine parts, joined with `:`: The signed data is one line of nine parts, joined with `:`:
@@ -552,15 +552,15 @@ HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit"
has the signature of the same URL ending in `.jxl`. `auto` is signed as has the signature of the same URL ending in `.jxl`. `auto` is signed as
`auto`, not as the format chosen for the request. `auto`, not as the format chosen for the request.
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature - `expiration`: the URL's `exp`, the Unix time in seconds when the signature
expires. A request whose `exp` is not a whole number is refused with 400. An expires. A request whose `exp` is not a whole number, an empty `exp=`
empty `exp=` is one example. included, is refused with 400.
- `quality`: the URL's `q`, or `85` when the URL has no `q`. - `quality`: the URL's `q`, or `85` when the URL has no `q`.
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`. - `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.
The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
compares it exactly. A signature encoded without padding is refused with 401. compares it exactly. A signature encoded without padding, as Node's `base64url`
Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding. and Go's `base64.RawURLEncoding` do, is refused with 401.
**Example.** With the signing key `example-signing-key-for-documentation`, **Example.** With the signing key `example-signing-key-for-documentation`,
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
@@ -614,16 +614,16 @@ The form's fields:
- The width and height. Both empty or `0` keep the source's size. When only one - The width and height. Both empty or `0` keep the source's size. When only one
of them is empty or `0`, that side is worked out from the source's proportions of them is empty or `0`, that side is worked out from the source's proportions
when the image is made (see Image URLs). when the image is made (see Image URLs).
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. - The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. It
`jxl` is selected unless another is chosen. A form sent with an empty format, is JPEG XL unless another is chosen. A form sent with an empty format, or
or none, also makes a JPEG XL URL. none, also makes a JPEG XL URL.
- The quality, a whole number from 1 to 100. The page offers five values: `25`, - The quality, a whole number from 1 to 100. The page offers five values, and
`50`, `70`, `85` and `100`. `85` is selected unless another is chosen. `85` unless another is chosen.
- The fit, one of the five fit modes (see Image URLs). `cover` is selected - The fit, one of the five fit modes (see Image URLs). It is `cover` unless
unless another is chosen. another is chosen.
- How long the URL lasts, in seconds, or `0` for never. The page offers seven - How long the URL lasts, in seconds, or `0` for never. The page offers never, 1
choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is minute, 1 hour, 1 day, 1 week, 30 days and 1 year, and never unless another is
selected unless another is chosen. chosen.
The page refuses a form with 400 when: The page refuses a form with 400 when:
@@ -682,9 +682,9 @@ dot. So an entry with a port, or a `*.` wildcard, aborts startup.
pixa lower-cases the host of a `/v1/image/` URL before anything else reads it. pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
So `CDN.Example.com` is matched, fetched, cached and counted against So `CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a `upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
port needs a signature even when the host is allowlisted. So does one whose host port, as in `cdn.example.com:8443`, needs a signature even when the host is
ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host allowlisted. So does one whose host ends in `:` with no port. The host must be
must be one of: one of:
- a host name or an IPv4 address: letters, digits, hyphens, underscores and - a host name or an IPv4 address: letters, digits, hyphens, underscores and
dots; dots;
@@ -701,7 +701,7 @@ the output. This cannot be turned off. The metadata removed is:
- EXIF, which can hold the GPS position, the camera make, model and serial - EXIF, which can hold the GPS position, the camera make, model and serial
number, the capture time and an embedded thumbnail; number, the capture time and an embedded thumbnail;
- XMP and IPTC, two other metadata formats; - XMP and IPTC;
- the ICC colour profile. - the ICC colour profile.
The rest of the rules: The rest of the rules:
@@ -710,9 +710,9 @@ The rest of the rules:
`orig` image is re-encoded and stripped like any other. `orig` image is re-encoded and stripped like any other.
- An image with an EXIF orientation is turned upright first, so it displays the - An image with an EXIF orientation is turned upright first, so it displays the
same without the tag. A requested size applies to the upright image. same without the tag. A requested size applies to the upright image.
- An image with an ICC profile is first converted to sRGB, the standard colour - An image with an ICC profile is converted to sRGB first, since clients show an
space, since clients show an image with no profile as sRGB. Colours outside image with no profile as sRGB. Colours outside sRGB are clipped: the most
sRGB are clipped: the most saturated ones in a Display P3 photo, for example. saturated ones in a Display P3 photo, for example.
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then - JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
writes an EXIF block of its own, which pixa cannot stop. The block holds the writes an EXIF block of its own, which pixa cannot stop. The block holds the
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi. image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
@@ -739,13 +739,13 @@ with `downstream_timeout`. Keep `downstream_timeout` longer than
`upstream_fetch_timeout` plus 20 seconds. `upstream_fetch_timeout` plus 20 seconds.
A conversion whose request's `downstream_timeout` has ended stops waiting for a A conversion whose request's `downstream_timeout` has ended stops waiting for a
processing slot. If its `downstream_timeout` ends after it has a slot but before processing slot. Once it has one, it gives the slot back without beginning to
it begins to encode, it gives the slot back. For a shared conversion, the first encode. For a shared conversion, the first request's deadline counts. When a
request's deadline counts. When a shared conversion stops this way, each request shared conversion stops this way, each request waiting for it is answered 503
waiting for it is answered 503 `server busy, try again later`, even if its own `server busy, try again later`, even if its own `downstream_timeout` has not
`downstream_timeout` has not ended. govips, the library pixa drives libvips ended. govips, the library pixa drives libvips through, gives no way to stop an
through, gives no way to stop an encode once it has begun. So an encode keeps encode once it has begun. So an encode keeps its slot until it finishes, which
its slot until it finishes, which can be after `downstream_timeout`. can be after `downstream_timeout`.
**Encode times.** The limits on image size bound how long a conversion can take. **Encode times.** The limits on image size bound how long a conversion can take.
They are 8192 on each side of the source and of the output, and 16,777,216 They are 8192 on each side of the source and of the output, and 16,777,216
@@ -762,20 +762,20 @@ it, encoding 8192x8192 random pixels takes up to about:
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
85 it would take about 96 seconds with an alpha channel, and 52 without. These 85 it would take about 96 seconds. These times hold at any quality and at 8 or
times hold at any quality and at 8 or 16 bits per sample. No encode within the 16 bits per sample. No encode within the limits takes more than about 40 seconds
limits takes more than about 40 seconds of the default `downstream_timeout` of the default `downstream_timeout` of 60.
of 60.
**Processing and connections.** `max_concurrent_processing` images are decoded **Processing and connections.** `max_concurrent_processing` images are decoded
and encoded at once. A request that finds every slot in use waits up to 10 and encoded at once. A request that finds every slot in use waits up to 10
seconds for one. If none frees up, it is answered 503 seconds for one. If none frees up, it is answered 503
`server busy, try again later`. `upstream_connections` bounds the fetches in `server busy, try again later`, unless `downstream_timeout` has ended first.
progress to all upstream hosts together, and `upstream_connections_per_host` `upstream_connections` bounds the fetches in progress to all upstream hosts
those to one host. A fetch holds its connection until its image has been together, and `upstream_connections_per_host` those to one host. A fetch holds
processed. A fetch that finds every shared connection in use waits up to 10 its connection until its image has been processed. A fetch that finds every
seconds the same way, then answers 503. A fetch that finds every connection to shared connection in use waits up to 10 seconds the same way, then answers 503.
its host in use waits for one until `downstream_timeout` ends. A fetch that finds every connection to its host in use waits for one until
`downstream_timeout` ends.
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute **Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
per client address. An attempt over the limit is refused with 429 and a per client address. An attempt over the limit is refused with 429 and a
@@ -789,17 +789,18 @@ limit off. Both limits count an IPv6 client by its `/64`.
from `X-Forwarded-For` only when the request arrives from an address in from `X-Forwarded-For` only when the request arrives from an address in
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The `trusted_proxies`. Otherwise all users behind the proxy count as one client. The
address the request arrives from is not always the proxy's own. A proxy on the address the request arrives from is not always the proxy's own. A proxy on the
Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of Docker host that connects to pixa over `127.0.0.1` is seen as the gateway of the
the container's Docker network, such as `172.17.0.1` on the default bridge. A container's Docker network, such as `172.17.0.1` on the default bridge. A proxy
proxy that connects through another of the host's addresses is seen with that that connects through another of the host's addresses is seen with that address.
address. To be sure, find the address as described under `trusted_proxies` (see To be sure, read the address as `remoteIP` in pixa's request log while it is not
Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the in `trusted_proxies` (see `trusted_proxies` under Configuration). The default
private address ranges. With that default, a client with a private address is `trusted_proxies` is the RFC 1918 ranges, the private address ranges. With that
itself trusted. It can then choose the address it is counted by, through its own default, a client with a private address can choose the address it is counted
`X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes by, through its own `X-Forwarded-For`, because its own address is trusted too.
this. Behind a content delivery network (CDN) or proxy whose addresses are not Setting `trusted_proxies` to only the proxy's address closes this. Behind a CDN
in `trusted_proxies`, all requests arriving from one address count as one or proxy whose addresses are not in `trusted_proxies`, all requests arriving
client. Such a deployment needs a higher `image_requests_per_minute`, or `0`. from one address count as one client. Such a deployment needs a higher
`image_requests_per_minute`, or `0`.
**Request limits.** pixa reads a request's headers within 10 seconds and the **Request limits.** pixa reads a request's headers within 10 seconds and the
whole request within 30 seconds. Headers are limited to 8 KiB. An idle whole request within 30 seconds. Headers are limited to 8 KiB. An idle
@@ -815,9 +816,8 @@ with 413.
`<ab>` and `<cd>` are the first and second pairs of characters of the file's `<ab>` and `<cd>` are the first and second pairs of characters of the file's
name. The database is one SQLite file. SQLite is the database library pixa name. The database is one SQLite file. SQLite is the database library pixa
embeds; nothing else needs to run. By default the database is in WAL mode, embeds; nothing else needs to run. In WAL mode, the default, SQLite keeps
SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal` `state.sqlite3-wal` and `state.sqlite3-shm` beside it while pixa runs.
and `state.sqlite3-shm` beside the file while pixa runs.
The image cache keeps these tables in the database: The image cache keeps these tables in the database:
@@ -855,23 +855,22 @@ naming the path and the error, and goes on with the rest of the check. A file
renamed or removed after pixa listed its directory gets no warning. renamed or removed after pixa listed its directory gets no warning.
A check reads the database once for every cached file, so the checks are spaced A check reads the database once for every cached file, so the checks are spaced
out. After a check ends, pixa waits at least ten times as long as that check out. After a check ends, the next one waits at least ten times as long as that
took. Eviction runs every five minutes (below). The next check starts at the check took. After a check that took one minute, the next one starts 10 to 15
first of those runs after that wait. After a check that took one minute, the minutes after that check ended.
next one starts 10 to 15 minutes after that check ended.
A cache hit opens the variant's file and nothing else. It makes no database A cache hit opens the variant's file and nothing else. It makes no database
call. The content type comes from the file's first bytes, which are those of the call. The content type comes from the file's first bytes, which are those of the
format pixa wrote it in. A file whose first bytes are not those of a format pixa format pixa wrote it in. A file whose first bytes are not those of a format pixa
writes is not served, and the image is converted again. A hit hands the time it writes is not served, and the image is converted again. A hit hands the time it
was served to a background writer and does not wait for the writer. Eviction was served to a background writer and does not wait for the writer. Eviction
orders by that time. The hit is counted in memory, for the orders by that time. The hit is counted in memory. A miss that uses a cached
`pixa_cache_hits_total` metric. A miss that uses a cached source also hands that source also hands that writer the time it used the source. Once a second the
writer the time it used the source. Once a second the writer writes the waiting writer writes the waiting times in one transaction. Up to 10,000 access times of
times in one transaction. Up to 10,000 access times of variants, and as many of variants, and as many of sources, can wait to be written. A time handed over
sources, can wait to be written. A time handed over beyond that is dropped. Its beyond that is dropped. Its image then only looks less recently used to
image then only looks less recently used to eviction. `script/bench` measures eviction. `script/bench` measures how many hits pixa serves per second (see Load
how many hits pixa serves per second (see Load Test). Test).
Eviction removes the least recently used sources and variants until the cache is Eviction removes the least recently used sources and variants until the cache is
within `cache_max_bytes`. It runs after each store, and every five minutes. It within `cache_max_bytes`. It runs after each store, and every five minutes. It
@@ -880,26 +879,25 @@ also removes the expired rows of the negative cache every five minutes, at most
### Configuration ### Configuration
Every setting can be given as an environment variable, in a config file Every setting can be given as an environment variable, in a YAML config file
(`--config`), or both. The file is YAML, a plain-text format. A variable present (`--config`), or both. A variable present in the environment wins over the file,
in the environment wins over the file, even when it is empty. The file wins over even when it is empty. The file wins over the built-in default. A variable's
the built-in default. pixa reads a variable's value as a quoted string in the value is read as the same text in quotes in the file would be. A setting that
file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts holds a list takes comma-separated entries, with the spaces around each trimmed.
startup. A setting that holds a list takes comma-separated entries, with the An empty variable is an empty list.
spaces around each trimmed. An empty variable is an empty list.
The file's `env:` section is the one exception to that order. pixa sets each The file's `env:` section is the one exception to that order. pixa sets each
variable named there while it loads the file. So that variable wins over the variable named there while it loads the file. So that variable wins over the
environment pixa started with, and over the file's key for the same setting. environment pixa started with, and over the file's key for the same setting.
pixa refuses to start when a value does not parse or is invalid. The error names pixa refuses to start when a value does not parse or is invalid. The error names
the setting. It also refuses to start when a key in the file is set to null. A the setting. It also refuses to start when a key in the file is set to null,
key written as `key:` with nothing after it is null. Omit the key to use the `key:` with nothing after it included. Omit the key to use the default. It
default. It refuses to start when the file has a key that is not in the list refuses to start when the file has a key that is not in the list below, and the
below, and the error names the key. It refuses to start when a variable starts error names the key. It refuses to start when a variable starts with `PIXA_` but
with `PIXA_` but is not in the list below, such as a misspelled one or is not in the list below, such as a misspelled one or `PIXA_PORT`. The error
`PIXA_PORT`. The error names the variable. The variables set by the file's names the variable. The variables set by the file's `env:` section are checked
`env:` section are checked the same way. the same way.
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
`--config`. `--config`.
@@ -918,14 +916,7 @@ cannot tell. With no file, pixa uses the environment and the defaults.
Durations are written as Go durations, such as `30s` or `2m`, and must be Durations are written as Go durations, such as `30s` or `2m`, and must be
positive. A bare number has no unit and aborts startup. Sizes are a whole number positive. A bare number has no unit and aborts startup. Sizes are a whole number
of bytes. A CIDR range is an address range written as an address and a prefix of bytes. A CIDR range is an address range written as an address and a prefix
length, such as `10.0.0.0/8`. An address matches a range that contains it. length, such as `10.0.0.0/8`.
`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain
IPv4 address. A range written in that form is read as the IPv4 range it covers:
`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers
more than IPv4 addresses and aborts startup.
Each item below is one setting, named by its config key: Each item below is one setting, named by its config key:
@@ -945,16 +936,16 @@ Each item below is one setting, named by its config key:
- `db_url`: variable `PIXA_DB_URL`, default - `db_url`: variable `PIXA_DB_URL`, default
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
database to open. The default keeps the database in WAL mode, SQLite's database to open. The default keeps the database in WAL mode, SQLite's
write-ahead log, which lets reads go on during a write. pixa adds write-ahead log, which lets reads go on during a write. In the default,
`_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another `state_dir` is percent-encoded: a `%`, `?` or `#` in the name is written as
program writing to the file then waits up to five seconds for it instead of `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one starting with `//`,
failing. pixa opens one connection, so its own reads and writes run one at a the default starts with `file:////`, so SQLite does not read the first name in
time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in the path as a host. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`. A
one you set. An empty `db_url` aborts startup: omit the key to use the write that finds another program writing to the file then waits up to five
default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in seconds for it instead of failing. pixa opens one connection, so its own reads
the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one and writes run one at a time. WAL mode comes only from the URL: keep
starting with `//`, the default starts with `file:////`, so SQLite does not `_pragma=journal_mode(WAL)` in one you set. An empty `db_url` aborts startup:
read the first name in the path as a host. omit the key to use the default.
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk - `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
cache limit in bytes. When the key is omitted, pixa works the limit out at cache limit in bytes. When the key is omitted, pixa works the limit out at
startup. It takes the free space on the filesystem holding startup. It takes the free space on the filesystem holding
@@ -969,19 +960,23 @@ Each item below is one setting, named by its config key:
you control. An entry that is neither a host name nor an IP address aborts you control. An entry that is neither a host name nor an IP address aborts
startup. startup.
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts - `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
whose pages may not show pixa's images, to stop other sites embedding them. whose pages may not show pixa's images, to stop other sites hotlinking them.
Entries are written and matched as for `allowlist_hosts`. A request whose Entries are written and matched as for `allowlist_hosts`. A request to
`Referer` header names a listed host is refused with 403. That applies to `/v1/image/` or `/v1/e/` whose `Referer` header names a listed host is refused
`/v1/image/` and `/v1/e/`. It happens before the signature or token is with 403. That happens before its signature or token is checked, and before
checked, and before the cache or the upstream host is used. The request is the cache or the upstream host is used. It is refused even when the image is
refused even when the image is cached. A request with no `Referer` is served, cached. A request with no `Referer`, or one that does not parse as a URL with
as many clients send none. So is one whose `Referer` does not parse as a URL a host, is served, as many clients send none. So a site whose pages send no
with a host. So a site whose pages send no `Referer`, for example with `Referer`, for example with `Referrer-Policy: no-referrer`, is not stopped.
`Referrer-Policy: no-referrer`, is not stopped. The login and generator pages The login and generator pages are not covered.
are not covered.
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR - `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
aborts startup. The built-in ranges always apply. They are: aborts startup. An IPv6 address in the IPv4-mapped form (`::ffff:a.b.c.d`) is
matched only as the IPv4 address it holds. So an IPv6 range such as `::/0`
refuses neither it nor a plain IPv4 address. An entry written in that form is
read as the IPv4 range it covers: `::ffff:10.0.0.0/104` refuses what
`10.0.0.0/8` refuses. Such an entry shorter than `/96` covers more than IPv4
addresses and aborts startup. The built-in ranges always apply. They are:
- `0.0.0.0/8`, this network, `0.0.0.0` included; - `0.0.0.0/8`, this network, `0.0.0.0` included;
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges; - `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
- `100.64.0.0/10`, carrier-grade NAT; - `100.64.0.0/10`, carrier-grade NAT;
@@ -1005,11 +1000,12 @@ Each item below is one setting, named by its config key:
came from is used. So a client connecting from outside these ranges cannot came from is used. So a client connecting from outside these ranges cannot
choose its address. An explicitly empty list (`[]`, or an empty variable) choose its address. An explicitly empty list (`[]`, or an empty variable)
trusts no one. A list you set replaces the default. An invalid CIDR aborts trusts no one. A list you set replaces the default. An invalid CIDR aborts
startup. Set this to the address pixa sees for requests that come through your startup. An IPv4-mapped entry or address is handled as under
proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or `blocked_networks`. Set this to the address pixa sees for requests that come
to trust nothing else (see Limits). To find that address, set this to `[]`, through your proxy, such as `172.17.0.1/32`. Do so when the default does not
send a request through the proxy, and read `remoteIP` in pixa's log line for cover it, or to trust nothing else (see Limits). To find that address, set
it. this to `[]`, send a request through the proxy, and read `remoteIP` in pixa's
log line for it.
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa - `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
fetches every source over plain HTTP, and follows redirects to plain HTTP. It fetches every source over plain HTTP, and follows redirects to plain HTTP. It
is for testing only. is for testing only.
@@ -1074,11 +1070,11 @@ Each item below is one setting, named by its config key:
### Architecture ### Architecture
- **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the - **Dependency injection**: Uber fx, a library that builds the program's parts
program's parts in dependency order and starts and stops them. `internal/app` in dependency order and starts and stops them. `internal/app` puts every part
puts every part of `pixad` in one `fx` graph. `internal/providers` builds the of `pixad` in one fx graph. `internal/providers` builds the image cache, the
image cache, the image proxy and the other parts that serve images and logins. image proxy and the other parts that serve images and logins. Each setting's
Each setting's default is declared once, in `internal/config`. default is declared once, in `internal/config`.
- **Packages**: these packages under `internal/` work with the concepts above: - **Packages**: these packages under `internal/` work with the concepts above:
- `handlers`: the routes, which read the image request from the URL, check - `handlers`: the routes, which read the image request from the URL, check
the signed or encrypted URL, and make the login session the signed or encrypted URL, and make the login session
@@ -1094,7 +1090,7 @@ Each item below is one setting, named by its config key:
- `seal`: the keys derived from `signing_key`, and the encryption of those - `seal`: the keys derived from `signing_key`, and the encryption of those
tokens tokens
- `session`: the login session - `session`: the login session
- **HTTP router**: `chi` (`github.com/go-chi/chi/v5`) - **HTTP router**: go-chi
- **Image processing**: govips, a Go wrapper for libvips - **Image processing**: govips, a Go wrapper for libvips
- **Database**: SQLite via `modernc.org/sqlite` - **Database**: SQLite via `modernc.org/sqlite`
- **Static assets**: embedded in the binary via `//go:embed` - **Static assets**: embedded in the binary via `//go:embed`
@@ -1116,9 +1112,9 @@ Each item below is one setting, named by its config key:
A request's `handler` label is the pattern of the route it matches, such as A request's `handler` label is the pattern of the route it matches, such as
`/v1/image/*`, never its path. Requests that match no route share the label `/v1/image/*`, never its path. Requests that match no route share the label
`unmatched`: those the router answers 404 or 405, and CORS preflights. A `unmatched`: those the router answers 404 or 405, and CORS preflights. A
request's `method` label is its method, for these nine: `GET`, `HEAD`, request's `method` label is its method when that is `GET`, `HEAD`, `POST`,
`POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for
other method is labelled `other`. any other method.
- **Logging**: the standard library's `slog`, as JSON on standard output, or as - **Logging**: the standard library's `slog`, as JSON on standard output, or as
text when standard output is a terminal. text when standard output is a terminal.
@@ -1128,25 +1124,25 @@ Each item below is one setting, named by its config key:
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The scripts in `script/` are the entrypoints for the development standard. The scripts in `script/` are the entrypoints for the development
workflow, and most `Makefile` targets are thin shims that call them. We provide: workflow, and most Makefile targets are thin shims that call them. We provide:
- `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and - `script/bootstrap` — install git, make, Go, Node, Yarn and prettier, and
download the Go modules. It can be run again at any time. With `--cgo`, it download the Go modules. It can be run again at any time. With `--cgo`, it
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker, libheif instead of Node, Yarn and prettier. Compiling pixa outside Docker, as
as `make build` and `script/bench` do, needs those. Docker itself must already `make build` and `script/bench` do, needs those. Docker itself must already be
be installed. installed.
- `script/setup` — make a fresh clone ready for development: `script/bootstrap`, - `script/setup` — make a fresh clone ready for development: bootstrap, then
then `script/install-precommit`. install-precommit.
- `script/projectname` — output the project name (`pixa`). - `script/projectname` — output the project name (`pixa`).
- `script/test` — run the test suite: build the `test` phase of the - `script/test` — run the test suite: build the `test` phase of the
`Dockerfile`, writing no image. `Dockerfile`, writing no image.
- `script/lint` — run `golangci-lint`: build the `lint` phase of the - `script/lint` — run `golangci-lint`: build the `lint` phase of the
`Dockerfile`, writing no image. The linter never runs on the host. `Dockerfile`, writing no image. The linter never runs on the host.
- `script/fmt` — format the Go code with `gofmt` and the markdown with - `script/fmt` — format the Go code with `gofmt` and the markdown with prettier
`prettier` (writes). (writes).
- `script/fmt-check` — check the same formatting (read-only), on the host. - `script/fmt-check` — check the same formatting (read-only), on the host.
- `script/check` — run `script/test`, `script/lint` and `script/fmt-check`. - `script/check` — run test, lint, and fmt-check.
- `script/docker` — build the Docker image tagged via `script/projectname`, with - `script/docker` — build the Docker image tagged via `script/projectname`, with
the version from `git describe`. The image's build stage depends on the `lint` the version from `git describe`. The image's build stage depends on the `lint`
and `test` phases, so this runs them too. and `test` phases, so this runs them too.
@@ -1195,7 +1191,7 @@ them:
- `script/install-precommit` - `script/install-precommit`
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt` `script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
step kept before `prettier`. `prettier` formats the markdown only, not the HTML step kept before prettier. prettier formats the markdown only, not the HTML
templates, as it cannot parse a Go template action inside a tag. templates, as it cannot parse a Go template action inside a tag.
## Load Test ## Load Test
@@ -1233,8 +1229,8 @@ For each scenario the script prints vegeta's report and two lines of its own:
- `Requests [total, rate, throughput]`: the requests sent, how many were sent - `Requests [total, rate, throughput]`: the requests sent, how many were sent
per second, and how many were answered successfully per second; per second, and how many were answered successfully per second;
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request - `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th to the end of its answer. `50`, `95` and `99` are the 50th, 95th and 99th
and 99th percentiles; percentiles;
- `Status Codes` and `Error Set`: anything other than `200` means the other - `Status Codes` and `Error Set`: anything other than `200` means the other
numbers are not for the scenario described, such as `503` when `pixad` was numbers are not for the scenario described, such as `503` when `pixad` was
busy; busy;
@@ -1250,22 +1246,22 @@ The numbers depend on the machine and on whatever else runs on it.
`script/bench` (or `make bench`) measures cache hits alone, without the network `script/bench` (or `make bench`) measures cache hits alone, without the network
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
cached image through the `/v1/image/` handler from several goroutines at once, cached image through the `/v1/image/` handler from as many goroutines at once as
one per CPU. A goroutine is a Go thread. The benchmark reports the hits served the machine has CPUs. It reports the hits served per second as `hits/s`. Each
per second as `hits/s`. Each run starts with an empty cache and an empty run starts with an empty cache and an empty database, a SQLite file of its own,
database, a SQLite file of its own, and puts the image in the cache before it and puts the image in the cache before it measures. The benchmark fails if
measures. The benchmark fails if serving the hits ran any statement on the serving the hits ran any statement on the database. Like the load test it is not
database. Like the load test it is not part of `script/check`. It runs Go on the part of `script/check`. It runs Go on the host, as the load test does, and
host, as the load test does, and compiles pixa there. So it needs the C compiler compiles pixa there. So it needs the C compiler and libraries that
and libraries that `script/bootstrap --cgo` installs, but not Docker. `script/bootstrap --cgo` installs, but not Docker.
## TODO ## TODO
See [`TODO.md`](TODO.md) for the full prioritized task list. See [TODO.md](TODO.md) for the full prioritized task list.
## License ## License
GPL-3.0. See [`LICENSE`](LICENSE). GPL-3.0. See [LICENSE](LICENSE).
## Author ## Author