Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
44f3192a03 |
@@ -9,7 +9,7 @@ original image, makes the requested image from it, and serves that. It keeps
|
|||||||
both on disk. Later requests for the same image are served from disk, without a
|
both on disk. Later requests for the same image are served from disk, without a
|
||||||
fetch or a conversion.
|
fetch or a conversion.
|
||||||
|
|
||||||
In this document, the upstream host is the server that holds an original image.
|
This document calls the server that holds an original image the upstream host.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -85,9 +85,9 @@ The reverse proxy must:
|
|||||||
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
|
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
|
||||||
is the longest pixa takes to fetch, convert and send an image.
|
is the longest pixa takes to fetch, convert and send an image.
|
||||||
|
|
||||||
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server.
|
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy does
|
||||||
Caddy does everything in this list without further settings, and gets its own
|
everything in this list without further settings, and gets its own TLS
|
||||||
TLS certificate.
|
certificate.
|
||||||
|
|
||||||
The reverse proxy may also refuse `/metrics`, as the example does. Only a
|
The reverse proxy may also refuse `/metrics`, as the example does. Only a
|
||||||
scraper that reaches pixa directly can then read it. pixa itself asks for
|
scraper that reaches pixa directly can then read it. pixa itself asks for
|
||||||
@@ -112,9 +112,9 @@ database and the disk cache:
|
|||||||
- In the container, the startup script gives the directory to the user pixa runs
|
- In the container, the startup script gives the directory to the user pixa runs
|
||||||
as, uid 65532, and sets its mode to `750`. Outside the container, the user
|
as, uid 65532, and sets its mode to `750`. Outside the container, the user
|
||||||
running pixa must be able to write it.
|
running pixa must be able to write it.
|
||||||
- `cache_max_bytes` limits the cached images. The limit does not count the
|
- `cache_max_bytes` limits the cached images. The database and files still being
|
||||||
database, or files still being written. Eviction runs in the background, so
|
written come on top. Eviction runs in the background, so the cache can pass
|
||||||
the cache can pass the limit for a while. Leave room on the volume beyond it.
|
the limit for a while. Leave room on the volume beyond it.
|
||||||
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
|
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
|
||||||
each start from the free space on the volume (see Configuration). Anything
|
each start from the free space on the volume (see Configuration). Anything
|
||||||
else that fills or frees space on the volume moves that default.
|
else that fills or frees space on the volume moves that default.
|
||||||
@@ -139,8 +139,8 @@ Outside Docker, pixa needs these at run time:
|
|||||||
|
|
||||||
- libvips, the image library pixa converts images with, including its JPEG XL
|
- libvips, the image library pixa converts images with, including its JPEG XL
|
||||||
support. pixa does not start without that support. On Alpine it is the
|
support. pixa does not start without that support. On Alpine it is the
|
||||||
`vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do
|
`vips-jxl` package. The nix and brew packages of libvips include it. So do the
|
||||||
the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on.
|
apt packages from Debian 12 and Ubuntu 24.04 on.
|
||||||
- libheif, which libvips needs for AVIF.
|
- libheif, which libvips needs for AVIF.
|
||||||
- the system's CA certificates, which HTTPS fetches need.
|
- the system's CA certificates, which HTTPS fetches need.
|
||||||
|
|
||||||
@@ -401,9 +401,9 @@ An image URL has one of these forms. The second has no format:
|
|||||||
`inside` or `outside` (below). It is `cover` when the URL has no `fit`.
|
`inside` or `outside` (below). It is `cover` when the URL has no `fit`.
|
||||||
|
|
||||||
`q` and `fit` are part of what is cached, so each value of either is a separate
|
`q` and `fit` are part of what is cached, so each value of either is a separate
|
||||||
cached image. A request is refused with 400 when its query string cannot be
|
cached image. A request whose query string cannot be decoded, or gives any
|
||||||
decoded, or when it gives any parameter more than once. A request with a `q`,
|
parameter more than once, is refused with 400. So is a `q`, `fit` or `exp` that
|
||||||
`fit` or `exp` that is empty or not valid is refused with 400 too.
|
is empty or not valid.
|
||||||
|
|
||||||
pixa fetches sources over HTTPS only, and only from hosts with valid
|
pixa fetches sources over HTTPS only, and only from hosts with valid
|
||||||
certificates. With `allow_http` set, it fetches every source over plain HTTP
|
certificates. With `allow_http` set, it fetches every source over plain HTTP
|
||||||
@@ -522,11 +522,11 @@ of the variant key, in quotes.
|
|||||||
### Signed URLs
|
### Signed URLs
|
||||||
|
|
||||||
A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
|
A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
|
||||||
time (seconds since 1 January 1970) when it expires, and `sig`, its signature.
|
time when it expires, and `sig`, its signature. The signature is an HMAC-SHA256:
|
||||||
The signature is an HMAC-SHA256: a hash of the request's details, keyed with
|
a hash of the request's details, keyed with `signing_key`, that only someone who
|
||||||
`signing_key`, that only someone who knows the key can make. Signatures match
|
knows the key can make. Signatures match exactly: every part of the signed data
|
||||||
exactly: every part of the signed data must be exactly what was signed. There is
|
must be exactly what was signed. There is no suffix, wildcard or partial
|
||||||
no suffix, wildcard or partial matching.
|
matching.
|
||||||
|
|
||||||
The signed data is one line of nine parts, joined with `:`:
|
The signed data is one line of nine parts, joined with `:`:
|
||||||
|
|
||||||
@@ -552,15 +552,15 @@ HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit"
|
|||||||
has the signature of the same URL ending in `.jxl`. `auto` is signed as
|
has the signature of the same URL ending in `.jxl`. `auto` is signed as
|
||||||
`auto`, not as the format chosen for the request.
|
`auto`, not as the format chosen for the request.
|
||||||
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature
|
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature
|
||||||
expires. A request whose `exp` is not a whole number is refused with 400. An
|
expires. A request whose `exp` is not a whole number, an empty `exp=`
|
||||||
empty `exp=` is one example.
|
included, is refused with 400.
|
||||||
- `quality`: the URL's `q`, or `85` when the URL has no `q`.
|
- `quality`: the URL's `q`, or `85` when the URL has no `q`.
|
||||||
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.
|
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.
|
||||||
|
|
||||||
The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
|
The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
|
||||||
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
|
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
|
||||||
compares it exactly. A signature encoded without padding is refused with 401.
|
compares it exactly. A signature encoded without padding, as Node's `base64url`
|
||||||
Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding.
|
and Go's `base64.RawURLEncoding` do, is refused with 401.
|
||||||
|
|
||||||
**Example.** With the signing key `example-signing-key-for-documentation`,
|
**Example.** With the signing key `example-signing-key-for-documentation`,
|
||||||
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
|
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
|
||||||
@@ -614,16 +614,16 @@ The form's fields:
|
|||||||
- The width and height. Both empty or `0` keep the source's size. When only one
|
- The width and height. Both empty or `0` keep the source's size. When only one
|
||||||
of them is empty or `0`, that side is worked out from the source's proportions
|
of them is empty or `0`, that side is worked out from the source's proportions
|
||||||
when the image is made (see Image URLs).
|
when the image is made (see Image URLs).
|
||||||
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`.
|
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. It
|
||||||
`jxl` is selected unless another is chosen. A form sent with an empty format,
|
is JPEG XL unless another is chosen. A form sent with an empty format, or
|
||||||
or none, also makes a JPEG XL URL.
|
none, also makes a JPEG XL URL.
|
||||||
- The quality, a whole number from 1 to 100. The page offers five values: `25`,
|
- The quality, a whole number from 1 to 100. The page offers five values, and
|
||||||
`50`, `70`, `85` and `100`. `85` is selected unless another is chosen.
|
`85` unless another is chosen.
|
||||||
- The fit, one of the five fit modes (see Image URLs). `cover` is selected
|
- The fit, one of the five fit modes (see Image URLs). It is `cover` unless
|
||||||
unless another is chosen.
|
another is chosen.
|
||||||
- How long the URL lasts, in seconds, or `0` for never. The page offers seven
|
- How long the URL lasts, in seconds, or `0` for never. The page offers never, 1
|
||||||
choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is
|
minute, 1 hour, 1 day, 1 week, 30 days and 1 year, and never unless another is
|
||||||
selected unless another is chosen.
|
chosen.
|
||||||
|
|
||||||
The page refuses a form with 400 when:
|
The page refuses a form with 400 when:
|
||||||
|
|
||||||
@@ -682,9 +682,9 @@ dot. So an entry with a port, or a `*.` wildcard, aborts startup.
|
|||||||
pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
|
pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
|
||||||
So `CDN.Example.com` is matched, fetched, cached and counted against
|
So `CDN.Example.com` is matched, fetched, cached and counted against
|
||||||
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
|
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
|
||||||
port needs a signature even when the host is allowlisted. So does one whose host
|
port, as in `cdn.example.com:8443`, needs a signature even when the host is
|
||||||
ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host
|
allowlisted. So does one whose host ends in `:` with no port. The host must be
|
||||||
must be one of:
|
one of:
|
||||||
|
|
||||||
- a host name or an IPv4 address: letters, digits, hyphens, underscores and
|
- a host name or an IPv4 address: letters, digits, hyphens, underscores and
|
||||||
dots;
|
dots;
|
||||||
@@ -701,7 +701,7 @@ the output. This cannot be turned off. The metadata removed is:
|
|||||||
|
|
||||||
- EXIF, which can hold the GPS position, the camera make, model and serial
|
- EXIF, which can hold the GPS position, the camera make, model and serial
|
||||||
number, the capture time and an embedded thumbnail;
|
number, the capture time and an embedded thumbnail;
|
||||||
- XMP and IPTC, two other metadata formats;
|
- XMP and IPTC;
|
||||||
- the ICC colour profile.
|
- the ICC colour profile.
|
||||||
|
|
||||||
The rest of the rules:
|
The rest of the rules:
|
||||||
@@ -710,9 +710,9 @@ The rest of the rules:
|
|||||||
`orig` image is re-encoded and stripped like any other.
|
`orig` image is re-encoded and stripped like any other.
|
||||||
- An image with an EXIF orientation is turned upright first, so it displays the
|
- An image with an EXIF orientation is turned upright first, so it displays the
|
||||||
same without the tag. A requested size applies to the upright image.
|
same without the tag. A requested size applies to the upright image.
|
||||||
- An image with an ICC profile is first converted to sRGB, the standard colour
|
- An image with an ICC profile is converted to sRGB first, since clients show an
|
||||||
space, since clients show an image with no profile as sRGB. Colours outside
|
image with no profile as sRGB. Colours outside sRGB are clipped: the most
|
||||||
sRGB are clipped: the most saturated ones in a Display P3 photo, for example.
|
saturated ones in a Display P3 photo, for example.
|
||||||
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
|
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
|
||||||
writes an EXIF block of its own, which pixa cannot stop. The block holds the
|
writes an EXIF block of its own, which pixa cannot stop. The block holds the
|
||||||
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
|
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
|
||||||
@@ -739,13 +739,13 @@ with `downstream_timeout`. Keep `downstream_timeout` longer than
|
|||||||
`upstream_fetch_timeout` plus 20 seconds.
|
`upstream_fetch_timeout` plus 20 seconds.
|
||||||
|
|
||||||
A conversion whose request's `downstream_timeout` has ended stops waiting for a
|
A conversion whose request's `downstream_timeout` has ended stops waiting for a
|
||||||
processing slot. If its `downstream_timeout` ends after it has a slot but before
|
processing slot. Once it has one, it gives the slot back without beginning to
|
||||||
it begins to encode, it gives the slot back. For a shared conversion, the first
|
encode. For a shared conversion, the first request's deadline counts. When a
|
||||||
request's deadline counts. When a shared conversion stops this way, each request
|
shared conversion stops this way, each request waiting for it is answered 503
|
||||||
waiting for it is answered 503 `server busy, try again later`, even if its own
|
`server busy, try again later`, even if its own `downstream_timeout` has not
|
||||||
`downstream_timeout` has not ended. govips, the library pixa drives libvips
|
ended. govips, the library pixa drives libvips through, gives no way to stop an
|
||||||
through, gives no way to stop an encode once it has begun. So an encode keeps
|
encode once it has begun. So an encode keeps its slot until it finishes, which
|
||||||
its slot until it finishes, which can be after `downstream_timeout`.
|
can be after `downstream_timeout`.
|
||||||
|
|
||||||
**Encode times.** The limits on image size bound how long a conversion can take.
|
**Encode times.** The limits on image size bound how long a conversion can take.
|
||||||
They are 8192 on each side of the source and of the output, and 16,777,216
|
They are 8192 on each side of the source and of the output, and 16,777,216
|
||||||
@@ -762,20 +762,20 @@ it, encoding 8192x8192 random pixels takes up to about:
|
|||||||
|
|
||||||
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
|
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
|
||||||
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
|
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
|
||||||
85 it would take about 96 seconds with an alpha channel, and 52 without. These
|
85 it would take about 96 seconds. These times hold at any quality and at 8 or
|
||||||
times hold at any quality and at 8 or 16 bits per sample. No encode within the
|
16 bits per sample. No encode within the limits takes more than about 40 seconds
|
||||||
limits takes more than about 40 seconds of the default `downstream_timeout`
|
of the default `downstream_timeout` of 60.
|
||||||
of 60.
|
|
||||||
|
|
||||||
**Processing and connections.** `max_concurrent_processing` images are decoded
|
**Processing and connections.** `max_concurrent_processing` images are decoded
|
||||||
and encoded at once. A request that finds every slot in use waits up to 10
|
and encoded at once. A request that finds every slot in use waits up to 10
|
||||||
seconds for one. If none frees up, it is answered 503
|
seconds for one. If none frees up, it is answered 503
|
||||||
`server busy, try again later`. `upstream_connections` bounds the fetches in
|
`server busy, try again later`, unless `downstream_timeout` has ended first.
|
||||||
progress to all upstream hosts together, and `upstream_connections_per_host`
|
`upstream_connections` bounds the fetches in progress to all upstream hosts
|
||||||
those to one host. A fetch holds its connection until its image has been
|
together, and `upstream_connections_per_host` those to one host. A fetch holds
|
||||||
processed. A fetch that finds every shared connection in use waits up to 10
|
its connection until its image has been processed. A fetch that finds every
|
||||||
seconds the same way, then answers 503. A fetch that finds every connection to
|
shared connection in use waits up to 10 seconds the same way, then answers 503.
|
||||||
its host in use waits for one until `downstream_timeout` ends.
|
A fetch that finds every connection to its host in use waits for one until
|
||||||
|
`downstream_timeout` ends.
|
||||||
|
|
||||||
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
|
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
|
||||||
per client address. An attempt over the limit is refused with 429 and a
|
per client address. An attempt over the limit is refused with 429 and a
|
||||||
@@ -789,17 +789,18 @@ limit off. Both limits count an IPv6 client by its `/64`.
|
|||||||
from `X-Forwarded-For` only when the request arrives from an address in
|
from `X-Forwarded-For` only when the request arrives from an address in
|
||||||
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The
|
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The
|
||||||
address the request arrives from is not always the proxy's own. A proxy on the
|
address the request arrives from is not always the proxy's own. A proxy on the
|
||||||
Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of
|
Docker host that connects to pixa over `127.0.0.1` is seen as the gateway of the
|
||||||
the container's Docker network, such as `172.17.0.1` on the default bridge. A
|
container's Docker network, such as `172.17.0.1` on the default bridge. A proxy
|
||||||
proxy that connects through another of the host's addresses is seen with that
|
that connects through another of the host's addresses is seen with that address.
|
||||||
address. To be sure, find the address as described under `trusted_proxies` (see
|
To be sure, read the address as `remoteIP` in pixa's request log while it is not
|
||||||
Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the
|
in `trusted_proxies` (see `trusted_proxies` under Configuration). The default
|
||||||
private address ranges. With that default, a client with a private address is
|
`trusted_proxies` is the RFC 1918 ranges, the private address ranges. With that
|
||||||
itself trusted. It can then choose the address it is counted by, through its own
|
default, a client with a private address can choose the address it is counted
|
||||||
`X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes
|
by, through its own `X-Forwarded-For`, because its own address is trusted too.
|
||||||
this. Behind a content delivery network (CDN) or proxy whose addresses are not
|
Setting `trusted_proxies` to only the proxy's address closes this. Behind a CDN
|
||||||
in `trusted_proxies`, all requests arriving from one address count as one
|
or proxy whose addresses are not in `trusted_proxies`, all requests arriving
|
||||||
client. Such a deployment needs a higher `image_requests_per_minute`, or `0`.
|
from one address count as one client. Such a deployment needs a higher
|
||||||
|
`image_requests_per_minute`, or `0`.
|
||||||
|
|
||||||
**Request limits.** pixa reads a request's headers within 10 seconds and the
|
**Request limits.** pixa reads a request's headers within 10 seconds and the
|
||||||
whole request within 30 seconds. Headers are limited to 8 KiB. An idle
|
whole request within 30 seconds. Headers are limited to 8 KiB. An idle
|
||||||
@@ -815,9 +816,8 @@ with 413.
|
|||||||
|
|
||||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||||
name. The database is one SQLite file. SQLite is the database library pixa
|
name. The database is one SQLite file. SQLite is the database library pixa
|
||||||
embeds; nothing else needs to run. By default the database is in WAL mode,
|
embeds; nothing else needs to run. In WAL mode, the default, SQLite keeps
|
||||||
SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal`
|
`state.sqlite3-wal` and `state.sqlite3-shm` beside it while pixa runs.
|
||||||
and `state.sqlite3-shm` beside the file while pixa runs.
|
|
||||||
|
|
||||||
The image cache keeps these tables in the database:
|
The image cache keeps these tables in the database:
|
||||||
|
|
||||||
@@ -855,23 +855,22 @@ naming the path and the error, and goes on with the rest of the check. A file
|
|||||||
renamed or removed after pixa listed its directory gets no warning.
|
renamed or removed after pixa listed its directory gets no warning.
|
||||||
|
|
||||||
A check reads the database once for every cached file, so the checks are spaced
|
A check reads the database once for every cached file, so the checks are spaced
|
||||||
out. After a check ends, pixa waits at least ten times as long as that check
|
out. After a check ends, the next one waits at least ten times as long as that
|
||||||
took. Eviction runs every five minutes (below). The next check starts at the
|
check took. After a check that took one minute, the next one starts 10 to 15
|
||||||
first of those runs after that wait. After a check that took one minute, the
|
minutes after that check ended.
|
||||||
next one starts 10 to 15 minutes after that check ended.
|
|
||||||
|
|
||||||
A cache hit opens the variant's file and nothing else. It makes no database
|
A cache hit opens the variant's file and nothing else. It makes no database
|
||||||
call. The content type comes from the file's first bytes, which are those of the
|
call. The content type comes from the file's first bytes, which are those of the
|
||||||
format pixa wrote it in. A file whose first bytes are not those of a format pixa
|
format pixa wrote it in. A file whose first bytes are not those of a format pixa
|
||||||
writes is not served, and the image is converted again. A hit hands the time it
|
writes is not served, and the image is converted again. A hit hands the time it
|
||||||
was served to a background writer and does not wait for the writer. Eviction
|
was served to a background writer and does not wait for the writer. Eviction
|
||||||
orders by that time. The hit is counted in memory, for the
|
orders by that time. The hit is counted in memory. A miss that uses a cached
|
||||||
`pixa_cache_hits_total` metric. A miss that uses a cached source also hands that
|
source also hands that writer the time it used the source. Once a second the
|
||||||
writer the time it used the source. Once a second the writer writes the waiting
|
writer writes the waiting times in one transaction. Up to 10,000 access times of
|
||||||
times in one transaction. Up to 10,000 access times of variants, and as many of
|
variants, and as many of sources, can wait to be written. A time handed over
|
||||||
sources, can wait to be written. A time handed over beyond that is dropped. Its
|
beyond that is dropped. Its image then only looks less recently used to
|
||||||
image then only looks less recently used to eviction. `script/bench` measures
|
eviction. `script/bench` measures how many hits pixa serves per second (see Load
|
||||||
how many hits pixa serves per second (see Load Test).
|
Test).
|
||||||
|
|
||||||
Eviction removes the least recently used sources and variants until the cache is
|
Eviction removes the least recently used sources and variants until the cache is
|
||||||
within `cache_max_bytes`. It runs after each store, and every five minutes. It
|
within `cache_max_bytes`. It runs after each store, and every five minutes. It
|
||||||
@@ -880,26 +879,25 @@ also removes the expired rows of the negative cache every five minutes, at most
|
|||||||
|
|
||||||
### Configuration
|
### Configuration
|
||||||
|
|
||||||
Every setting can be given as an environment variable, in a config file
|
Every setting can be given as an environment variable, in a YAML config file
|
||||||
(`--config`), or both. The file is YAML, a plain-text format. A variable present
|
(`--config`), or both. A variable present in the environment wins over the file,
|
||||||
in the environment wins over the file, even when it is empty. The file wins over
|
even when it is empty. The file wins over the built-in default. A variable's
|
||||||
the built-in default. pixa reads a variable's value as a quoted string in the
|
value is read as the same text in quotes in the file would be. A setting that
|
||||||
file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts
|
holds a list takes comma-separated entries, with the spaces around each trimmed.
|
||||||
startup. A setting that holds a list takes comma-separated entries, with the
|
An empty variable is an empty list.
|
||||||
spaces around each trimmed. An empty variable is an empty list.
|
|
||||||
|
|
||||||
The file's `env:` section is the one exception to that order. pixa sets each
|
The file's `env:` section is the one exception to that order. pixa sets each
|
||||||
variable named there while it loads the file. So that variable wins over the
|
variable named there while it loads the file. So that variable wins over the
|
||||||
environment pixa started with, and over the file's key for the same setting.
|
environment pixa started with, and over the file's key for the same setting.
|
||||||
|
|
||||||
pixa refuses to start when a value does not parse or is invalid. The error names
|
pixa refuses to start when a value does not parse or is invalid. The error names
|
||||||
the setting. It also refuses to start when a key in the file is set to null. A
|
the setting. It also refuses to start when a key in the file is set to null,
|
||||||
key written as `key:` with nothing after it is null. Omit the key to use the
|
`key:` with nothing after it included. Omit the key to use the default. It
|
||||||
default. It refuses to start when the file has a key that is not in the list
|
refuses to start when the file has a key that is not in the list below, and the
|
||||||
below, and the error names the key. It refuses to start when a variable starts
|
error names the key. It refuses to start when a variable starts with `PIXA_` but
|
||||||
with `PIXA_` but is not in the list below, such as a misspelled one or
|
is not in the list below, such as a misspelled one or `PIXA_PORT`. The error
|
||||||
`PIXA_PORT`. The error names the variable. The variables set by the file's
|
names the variable. The variables set by the file's `env:` section are checked
|
||||||
`env:` section are checked the same way.
|
the same way.
|
||||||
|
|
||||||
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
|
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
|
||||||
`--config`.
|
`--config`.
|
||||||
@@ -918,14 +916,7 @@ cannot tell. With no file, pixa uses the environment and the defaults.
|
|||||||
Durations are written as Go durations, such as `30s` or `2m`, and must be
|
Durations are written as Go durations, such as `30s` or `2m`, and must be
|
||||||
positive. A bare number has no unit and aborts startup. Sizes are a whole number
|
positive. A bare number has no unit and aborts startup. Sizes are a whole number
|
||||||
of bytes. A CIDR range is an address range written as an address and a prefix
|
of bytes. A CIDR range is an address range written as an address and a prefix
|
||||||
length, such as `10.0.0.0/8`. An address matches a range that contains it.
|
length, such as `10.0.0.0/8`.
|
||||||
|
|
||||||
`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6
|
|
||||||
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
|
||||||
address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain
|
|
||||||
IPv4 address. A range written in that form is read as the IPv4 range it covers:
|
|
||||||
`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers
|
|
||||||
more than IPv4 addresses and aborts startup.
|
|
||||||
|
|
||||||
Each item below is one setting, named by its config key:
|
Each item below is one setting, named by its config key:
|
||||||
|
|
||||||
@@ -945,16 +936,16 @@ Each item below is one setting, named by its config key:
|
|||||||
- `db_url`: variable `PIXA_DB_URL`, default
|
- `db_url`: variable `PIXA_DB_URL`, default
|
||||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
|
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
|
||||||
database to open. The default keeps the database in WAL mode, SQLite's
|
database to open. The default keeps the database in WAL mode, SQLite's
|
||||||
write-ahead log, which lets reads go on during a write. pixa adds
|
write-ahead log, which lets reads go on during a write. In the default,
|
||||||
`_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another
|
`state_dir` is percent-encoded: a `%`, `?` or `#` in the name is written as
|
||||||
program writing to the file then waits up to five seconds for it instead of
|
`%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one starting with `//`,
|
||||||
failing. pixa opens one connection, so its own reads and writes run one at a
|
the default starts with `file:////`, so SQLite does not read the first name in
|
||||||
time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
|
the path as a host. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`. A
|
||||||
one you set. An empty `db_url` aborts startup: omit the key to use the
|
write that finds another program writing to the file then waits up to five
|
||||||
default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in
|
seconds for it instead of failing. pixa opens one connection, so its own reads
|
||||||
the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one
|
and writes run one at a time. WAL mode comes only from the URL: keep
|
||||||
starting with `//`, the default starts with `file:////`, so SQLite does not
|
`_pragma=journal_mode(WAL)` in one you set. An empty `db_url` aborts startup:
|
||||||
read the first name in the path as a host.
|
omit the key to use the default.
|
||||||
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
|
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
|
||||||
cache limit in bytes. When the key is omitted, pixa works the limit out at
|
cache limit in bytes. When the key is omitted, pixa works the limit out at
|
||||||
startup. It takes the free space on the filesystem holding
|
startup. It takes the free space on the filesystem holding
|
||||||
@@ -969,19 +960,23 @@ Each item below is one setting, named by its config key:
|
|||||||
you control. An entry that is neither a host name nor an IP address aborts
|
you control. An entry that is neither a host name nor an IP address aborts
|
||||||
startup.
|
startup.
|
||||||
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
|
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
|
||||||
whose pages may not show pixa's images, to stop other sites embedding them.
|
whose pages may not show pixa's images, to stop other sites hotlinking them.
|
||||||
Entries are written and matched as for `allowlist_hosts`. A request whose
|
Entries are written and matched as for `allowlist_hosts`. A request to
|
||||||
`Referer` header names a listed host is refused with 403. That applies to
|
`/v1/image/` or `/v1/e/` whose `Referer` header names a listed host is refused
|
||||||
`/v1/image/` and `/v1/e/`. It happens before the signature or token is
|
with 403. That happens before its signature or token is checked, and before
|
||||||
checked, and before the cache or the upstream host is used. The request is
|
the cache or the upstream host is used. It is refused even when the image is
|
||||||
refused even when the image is cached. A request with no `Referer` is served,
|
cached. A request with no `Referer`, or one that does not parse as a URL with
|
||||||
as many clients send none. So is one whose `Referer` does not parse as a URL
|
a host, is served, as many clients send none. So a site whose pages send no
|
||||||
with a host. So a site whose pages send no `Referer`, for example with
|
`Referer`, for example with `Referrer-Policy: no-referrer`, is not stopped.
|
||||||
`Referrer-Policy: no-referrer`, is not stopped. The login and generator pages
|
The login and generator pages are not covered.
|
||||||
are not covered.
|
|
||||||
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
|
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
|
||||||
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
|
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
|
||||||
aborts startup. The built-in ranges always apply. They are:
|
aborts startup. An IPv6 address in the IPv4-mapped form (`::ffff:a.b.c.d`) is
|
||||||
|
matched only as the IPv4 address it holds. So an IPv6 range such as `::/0`
|
||||||
|
refuses neither it nor a plain IPv4 address. An entry written in that form is
|
||||||
|
read as the IPv4 range it covers: `::ffff:10.0.0.0/104` refuses what
|
||||||
|
`10.0.0.0/8` refuses. Such an entry shorter than `/96` covers more than IPv4
|
||||||
|
addresses and aborts startup. The built-in ranges always apply. They are:
|
||||||
- `0.0.0.0/8`, this network, `0.0.0.0` included;
|
- `0.0.0.0/8`, this network, `0.0.0.0` included;
|
||||||
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
|
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
|
||||||
- `100.64.0.0/10`, carrier-grade NAT;
|
- `100.64.0.0/10`, carrier-grade NAT;
|
||||||
@@ -1005,11 +1000,12 @@ Each item below is one setting, named by its config key:
|
|||||||
came from is used. So a client connecting from outside these ranges cannot
|
came from is used. So a client connecting from outside these ranges cannot
|
||||||
choose its address. An explicitly empty list (`[]`, or an empty variable)
|
choose its address. An explicitly empty list (`[]`, or an empty variable)
|
||||||
trusts no one. A list you set replaces the default. An invalid CIDR aborts
|
trusts no one. A list you set replaces the default. An invalid CIDR aborts
|
||||||
startup. Set this to the address pixa sees for requests that come through your
|
startup. An IPv4-mapped entry or address is handled as under
|
||||||
proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or
|
`blocked_networks`. Set this to the address pixa sees for requests that come
|
||||||
to trust nothing else (see Limits). To find that address, set this to `[]`,
|
through your proxy, such as `172.17.0.1/32`. Do so when the default does not
|
||||||
send a request through the proxy, and read `remoteIP` in pixa's log line for
|
cover it, or to trust nothing else (see Limits). To find that address, set
|
||||||
it.
|
this to `[]`, send a request through the proxy, and read `remoteIP` in pixa's
|
||||||
|
log line for it.
|
||||||
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
|
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
|
||||||
fetches every source over plain HTTP, and follows redirects to plain HTTP. It
|
fetches every source over plain HTTP, and follows redirects to plain HTTP. It
|
||||||
is for testing only.
|
is for testing only.
|
||||||
@@ -1074,11 +1070,11 @@ Each item below is one setting, named by its config key:
|
|||||||
|
|
||||||
### Architecture
|
### Architecture
|
||||||
|
|
||||||
- **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the
|
- **Dependency injection**: Uber fx, a library that builds the program's parts
|
||||||
program's parts in dependency order and starts and stops them. `internal/app`
|
in dependency order and starts and stops them. `internal/app` puts every part
|
||||||
puts every part of `pixad` in one `fx` graph. `internal/providers` builds the
|
of `pixad` in one fx graph. `internal/providers` builds the image cache, the
|
||||||
image cache, the image proxy and the other parts that serve images and logins.
|
image proxy and the other parts that serve images and logins. Each setting's
|
||||||
Each setting's default is declared once, in `internal/config`.
|
default is declared once, in `internal/config`.
|
||||||
- **Packages**: these packages under `internal/` work with the concepts above:
|
- **Packages**: these packages under `internal/` work with the concepts above:
|
||||||
- `handlers`: the routes, which read the image request from the URL, check
|
- `handlers`: the routes, which read the image request from the URL, check
|
||||||
the signed or encrypted URL, and make the login session
|
the signed or encrypted URL, and make the login session
|
||||||
@@ -1094,7 +1090,7 @@ Each item below is one setting, named by its config key:
|
|||||||
- `seal`: the keys derived from `signing_key`, and the encryption of those
|
- `seal`: the keys derived from `signing_key`, and the encryption of those
|
||||||
tokens
|
tokens
|
||||||
- `session`: the login session
|
- `session`: the login session
|
||||||
- **HTTP router**: `chi` (`github.com/go-chi/chi/v5`)
|
- **HTTP router**: go-chi
|
||||||
- **Image processing**: govips, a Go wrapper for libvips
|
- **Image processing**: govips, a Go wrapper for libvips
|
||||||
- **Database**: SQLite via `modernc.org/sqlite`
|
- **Database**: SQLite via `modernc.org/sqlite`
|
||||||
- **Static assets**: embedded in the binary via `//go:embed`
|
- **Static assets**: embedded in the binary via `//go:embed`
|
||||||
@@ -1116,9 +1112,9 @@ Each item below is one setting, named by its config key:
|
|||||||
A request's `handler` label is the pattern of the route it matches, such as
|
A request's `handler` label is the pattern of the route it matches, such as
|
||||||
`/v1/image/*`, never its path. Requests that match no route share the label
|
`/v1/image/*`, never its path. Requests that match no route share the label
|
||||||
`unmatched`: those the router answers 404 or 405, and CORS preflights. A
|
`unmatched`: those the router answers 404 or 405, and CORS preflights. A
|
||||||
request's `method` label is its method, for these nine: `GET`, `HEAD`,
|
request's `method` label is its method when that is `GET`, `HEAD`, `POST`,
|
||||||
`POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any
|
`PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for
|
||||||
other method is labelled `other`.
|
any other method.
|
||||||
|
|
||||||
- **Logging**: the standard library's `slog`, as JSON on standard output, or as
|
- **Logging**: the standard library's `slog`, as JSON on standard output, or as
|
||||||
text when standard output is a terminal.
|
text when standard output is a terminal.
|
||||||
@@ -1128,25 +1124,25 @@ Each item below is one setting, named by its config key:
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard. The scripts in `script/` are the entrypoints for the development
|
standard. The scripts in `script/` are the entrypoints for the development
|
||||||
workflow, and most `Makefile` targets are thin shims that call them. We provide:
|
workflow, and most Makefile targets are thin shims that call them. We provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and
|
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier, and
|
||||||
download the Go modules. It can be run again at any time. With `--cgo`, it
|
download the Go modules. It can be run again at any time. With `--cgo`, it
|
||||||
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
|
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
|
||||||
libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker,
|
libheif instead of Node, Yarn and prettier. Compiling pixa outside Docker, as
|
||||||
as `make build` and `script/bench` do, needs those. Docker itself must already
|
`make build` and `script/bench` do, needs those. Docker itself must already be
|
||||||
be installed.
|
installed.
|
||||||
- `script/setup` — make a fresh clone ready for development: `script/bootstrap`,
|
- `script/setup` — make a fresh clone ready for development: bootstrap, then
|
||||||
then `script/install-precommit`.
|
install-precommit.
|
||||||
- `script/projectname` — output the project name (`pixa`).
|
- `script/projectname` — output the project name (`pixa`).
|
||||||
- `script/test` — run the test suite: build the `test` phase of the
|
- `script/test` — run the test suite: build the `test` phase of the
|
||||||
`Dockerfile`, writing no image.
|
`Dockerfile`, writing no image.
|
||||||
- `script/lint` — run `golangci-lint`: build the `lint` phase of the
|
- `script/lint` — run `golangci-lint`: build the `lint` phase of the
|
||||||
`Dockerfile`, writing no image. The linter never runs on the host.
|
`Dockerfile`, writing no image. The linter never runs on the host.
|
||||||
- `script/fmt` — format the Go code with `gofmt` and the markdown with
|
- `script/fmt` — format the Go code with `gofmt` and the markdown with prettier
|
||||||
`prettier` (writes).
|
(writes).
|
||||||
- `script/fmt-check` — check the same formatting (read-only), on the host.
|
- `script/fmt-check` — check the same formatting (read-only), on the host.
|
||||||
- `script/check` — run `script/test`, `script/lint` and `script/fmt-check`.
|
- `script/check` — run test, lint, and fmt-check.
|
||||||
- `script/docker` — build the Docker image tagged via `script/projectname`, with
|
- `script/docker` — build the Docker image tagged via `script/projectname`, with
|
||||||
the version from `git describe`. The image's build stage depends on the `lint`
|
the version from `git describe`. The image's build stage depends on the `lint`
|
||||||
and `test` phases, so this runs them too.
|
and `test` phases, so this runs them too.
|
||||||
@@ -1195,7 +1191,7 @@ them:
|
|||||||
- `script/install-precommit`
|
- `script/install-precommit`
|
||||||
|
|
||||||
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
|
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
|
||||||
step kept before `prettier`. `prettier` formats the markdown only, not the HTML
|
step kept before prettier. prettier formats the markdown only, not the HTML
|
||||||
templates, as it cannot parse a Go template action inside a tag.
|
templates, as it cannot parse a Go template action inside a tag.
|
||||||
|
|
||||||
## Load Test
|
## Load Test
|
||||||
@@ -1233,8 +1229,8 @@ For each scenario the script prints vegeta's report and two lines of its own:
|
|||||||
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
|
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
|
||||||
per second, and how many were answered successfully per second;
|
per second, and how many were answered successfully per second;
|
||||||
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
|
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
|
||||||
to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th
|
to the end of its answer. `50`, `95` and `99` are the 50th, 95th and 99th
|
||||||
and 99th percentiles;
|
percentiles;
|
||||||
- `Status Codes` and `Error Set`: anything other than `200` means the other
|
- `Status Codes` and `Error Set`: anything other than `200` means the other
|
||||||
numbers are not for the scenario described, such as `503` when `pixad` was
|
numbers are not for the scenario described, such as `503` when `pixad` was
|
||||||
busy;
|
busy;
|
||||||
@@ -1250,22 +1246,22 @@ The numbers depend on the machine and on whatever else runs on it.
|
|||||||
|
|
||||||
`script/bench` (or `make bench`) measures cache hits alone, without the network
|
`script/bench` (or `make bench`) measures cache hits alone, without the network
|
||||||
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
|
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
|
||||||
cached image through the `/v1/image/` handler from several goroutines at once,
|
cached image through the `/v1/image/` handler from as many goroutines at once as
|
||||||
one per CPU. A goroutine is a Go thread. The benchmark reports the hits served
|
the machine has CPUs. It reports the hits served per second as `hits/s`. Each
|
||||||
per second as `hits/s`. Each run starts with an empty cache and an empty
|
run starts with an empty cache and an empty database, a SQLite file of its own,
|
||||||
database, a SQLite file of its own, and puts the image in the cache before it
|
and puts the image in the cache before it measures. The benchmark fails if
|
||||||
measures. The benchmark fails if serving the hits ran any statement on the
|
serving the hits ran any statement on the database. Like the load test it is not
|
||||||
database. Like the load test it is not part of `script/check`. It runs Go on the
|
part of `script/check`. It runs Go on the host, as the load test does, and
|
||||||
host, as the load test does, and compiles pixa there. So it needs the C compiler
|
compiles pixa there. So it needs the C compiler and libraries that
|
||||||
and libraries that `script/bootstrap --cgo` installs, but not Docker.
|
`script/bootstrap --cgo` installs, but not Docker.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
See [`TODO.md`](TODO.md) for the full prioritized task list.
|
See [TODO.md](TODO.md) for the full prioritized task list.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
GPL-3.0. See [`LICENSE`](LICENSE).
|
GPL-3.0. See [LICENSE](LICENSE).
|
||||||
|
|
||||||
## Author
|
## Author
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user