Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8669b5cf04 |
@@ -9,7 +9,7 @@ original image, makes the requested image from it, and serves that. It keeps
|
||||
both on disk. Later requests for the same image are served from disk, without a
|
||||
fetch or a conversion.
|
||||
|
||||
This document calls the server that holds an original image the upstream host.
|
||||
In this document, the upstream host is the server that holds an original image.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -85,9 +85,9 @@ The reverse proxy must:
|
||||
- wait for pixa's answer for at least `downstream_timeout` (default `60s`). That
|
||||
is the longest pixa takes to fetch, convert and send an image.
|
||||
|
||||
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy. Caddy does
|
||||
everything in this list without further settings, and gets its own TLS
|
||||
certificate.
|
||||
[`configs/Caddyfile`](configs/Caddyfile) is an example for Caddy, a web server.
|
||||
Caddy does everything in this list without further settings, and gets its own
|
||||
TLS certificate.
|
||||
|
||||
The reverse proxy may also refuse `/metrics`, as the example does. Only a
|
||||
scraper that reaches pixa directly can then read it. pixa itself asks for
|
||||
@@ -112,9 +112,9 @@ database and the disk cache:
|
||||
- In the container, the startup script gives the directory to the user pixa runs
|
||||
as, uid 65532, and sets its mode to `750`. Outside the container, the user
|
||||
running pixa must be able to write it.
|
||||
- `cache_max_bytes` limits the cached images. The database and files still being
|
||||
written come on top. Eviction runs in the background, so the cache can pass
|
||||
the limit for a while. Leave room on the volume beyond it.
|
||||
- `cache_max_bytes` limits the cached images. The limit does not count the
|
||||
database, or files still being written. Eviction runs in the background, so
|
||||
the cache can pass the limit for a while. Leave room on the volume beyond it.
|
||||
- Set `cache_max_bytes` for a lasting deployment. Its default is worked out at
|
||||
each start from the free space on the volume (see Configuration). Anything
|
||||
else that fills or frees space on the volume moves that default.
|
||||
@@ -139,8 +139,8 @@ Outside Docker, pixa needs these at run time:
|
||||
|
||||
- libvips, the image library pixa converts images with, including its JPEG XL
|
||||
support. pixa does not start without that support. On Alpine it is the
|
||||
`vips-jxl` package. The nix and brew packages of libvips include it. So do the
|
||||
apt packages from Debian 12 and Ubuntu 24.04 on.
|
||||
`vips-jxl` package. The Nix and Homebrew packages of libvips include it. So do
|
||||
the Debian and Ubuntu packages, from Debian 12 and Ubuntu 24.04 on.
|
||||
- libheif, which libvips needs for AVIF.
|
||||
- the system's CA certificates, which HTTPS fetches need.
|
||||
|
||||
@@ -401,9 +401,9 @@ An image URL has one of these forms. The second has no format:
|
||||
`inside` or `outside` (below). It is `cover` when the URL has no `fit`.
|
||||
|
||||
`q` and `fit` are part of what is cached, so each value of either is a separate
|
||||
cached image. A request whose query string cannot be decoded, or gives any
|
||||
parameter more than once, is refused with 400. So is a `q`, `fit` or `exp` that
|
||||
is empty or not valid.
|
||||
cached image. A request is refused with 400 when its query string cannot be
|
||||
decoded, or when it gives any parameter more than once. A request with a `q`,
|
||||
`fit` or `exp` that is empty or not valid is refused with 400 too.
|
||||
|
||||
pixa fetches sources over HTTPS only, and only from hosts with valid
|
||||
certificates. With `allow_http` set, it fetches every source over plain HTTP
|
||||
@@ -522,11 +522,11 @@ of the variant key, in quotes.
|
||||
### Signed URLs
|
||||
|
||||
A signed URL is a `/v1/image/` URL with two query parameters: `exp`, the Unix
|
||||
time when it expires, and `sig`, its signature. The signature is an HMAC-SHA256:
|
||||
a hash of the request's details, keyed with `signing_key`, that only someone who
|
||||
knows the key can make. Signatures match exactly: every part of the signed data
|
||||
must be exactly what was signed. There is no suffix, wildcard or partial
|
||||
matching.
|
||||
time (seconds since 1 January 1970) when it expires, and `sig`, its signature.
|
||||
The signature is an HMAC-SHA256: a hash of the request's details, keyed with
|
||||
`signing_key`, that only someone who knows the key can make. Signatures match
|
||||
exactly: every part of the signed data must be exactly what was signed. There is
|
||||
no suffix, wildcard or partial matching.
|
||||
|
||||
The signed data is one line of nine parts, joined with `:`:
|
||||
|
||||
@@ -552,15 +552,15 @@ HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit"
|
||||
has the signature of the same URL ending in `.jxl`. `auto` is signed as
|
||||
`auto`, not as the format chosen for the request.
|
||||
- `expiration`: the URL's `exp`, the Unix time in seconds when the signature
|
||||
expires. A request whose `exp` is not a whole number, an empty `exp=`
|
||||
included, is refused with 400.
|
||||
expires. A request whose `exp` is not a whole number is refused with 400. An
|
||||
empty `exp=` is one example.
|
||||
- `quality`: the URL's `q`, or `85` when the URL has no `q`.
|
||||
- `fit`: the URL's `fit`, or `cover` when the URL has no `fit`.
|
||||
|
||||
The URL's `sig` is the HMAC-SHA256 result in base64url, the URL-safe alphabet of
|
||||
RFC 4648, with the trailing `=` padding kept: 44 characters in all. pixa
|
||||
compares it exactly. A signature encoded without padding, as Node's `base64url`
|
||||
and Go's `base64.RawURLEncoding` do, is refused with 401.
|
||||
compares it exactly. A signature encoded without padding is refused with 401.
|
||||
Node's `base64url` and Go's `base64.RawURLEncoding` encode without padding.
|
||||
|
||||
**Example.** With the signing key `example-signing-key-for-documentation`,
|
||||
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP, expiring at
|
||||
@@ -614,16 +614,16 @@ The form's fields:
|
||||
- The width and height. Both empty or `0` keep the source's size. When only one
|
||||
of them is empty or `0`, that side is worked out from the source's proportions
|
||||
when the image is made (see Image URLs).
|
||||
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`. It
|
||||
is JPEG XL unless another is chosen. A form sent with an empty format, or
|
||||
none, also makes a JPEG XL URL.
|
||||
- The quality, a whole number from 1 to 100. The page offers five values, and
|
||||
`85` unless another is chosen.
|
||||
- The fit, one of the five fit modes (see Image URLs). It is `cover` unless
|
||||
another is chosen.
|
||||
- How long the URL lasts, in seconds, or `0` for never. The page offers never, 1
|
||||
minute, 1 hour, 1 day, 1 week, 30 days and 1 year, and never unless another is
|
||||
chosen.
|
||||
- The format: `orig`, `auto`, `jpeg`, `png`, `webp`, `avif`, `jxl` or `gif`.
|
||||
`jxl` is selected unless another is chosen. A form sent with an empty format,
|
||||
or none, also makes a JPEG XL URL.
|
||||
- The quality, a whole number from 1 to 100. The page offers five values: `25`,
|
||||
`50`, `70`, `85` and `100`. `85` is selected unless another is chosen.
|
||||
- The fit, one of the five fit modes (see Image URLs). `cover` is selected
|
||||
unless another is chosen.
|
||||
- How long the URL lasts, in seconds, or `0` for never. The page offers seven
|
||||
choices: never, 1 minute, 1 hour, 1 day, 1 week, 30 days and 1 year. Never is
|
||||
selected unless another is chosen.
|
||||
|
||||
The page refuses a form with 400 when:
|
||||
|
||||
@@ -682,9 +682,9 @@ dot. So an entry with a port, or a `*.` wildcard, aborts startup.
|
||||
pixa lower-cases the host of a `/v1/image/` URL before anything else reads it.
|
||||
So `CDN.Example.com` is matched, fetched, cached and counted against
|
||||
`upstream_connections_per_host` as `cdn.example.com`. A URL whose host has a
|
||||
port, as in `cdn.example.com:8443`, needs a signature even when the host is
|
||||
allowlisted. So does one whose host ends in `:` with no port. The host must be
|
||||
one of:
|
||||
port needs a signature even when the host is allowlisted. So does one whose host
|
||||
ends in `:` with no port. `cdn.example.com:8443` is a host with a port. The host
|
||||
must be one of:
|
||||
|
||||
- a host name or an IPv4 address: letters, digits, hyphens, underscores and
|
||||
dots;
|
||||
@@ -701,7 +701,7 @@ the output. This cannot be turned off. The metadata removed is:
|
||||
|
||||
- EXIF, which can hold the GPS position, the camera make, model and serial
|
||||
number, the capture time and an embedded thumbnail;
|
||||
- XMP and IPTC;
|
||||
- XMP and IPTC, two other metadata formats;
|
||||
- the ICC colour profile.
|
||||
|
||||
The rest of the rules:
|
||||
@@ -710,9 +710,9 @@ The rest of the rules:
|
||||
`orig` image is re-encoded and stripped like any other.
|
||||
- An image with an EXIF orientation is turned upright first, so it displays the
|
||||
same without the tag. A requested size applies to the upright image.
|
||||
- An image with an ICC profile is converted to sRGB first, since clients show an
|
||||
image with no profile as sRGB. Colours outside sRGB are clipped: the most
|
||||
saturated ones in a Display P3 photo, for example.
|
||||
- An image with an ICC profile is first converted to sRGB, the standard colour
|
||||
space, since clients show an image with no profile as sRGB. Colours outside
|
||||
sRGB are clipped: the most saturated ones in a Display P3 photo, for example.
|
||||
- JPEG XL output is the one exception, with libvips 8.16 and later. libvips then
|
||||
writes an EXIF block of its own, which pixa cannot stop. The block holds the
|
||||
image's size and fixed values: an orientation of 1 and a resolution of 72 dpi.
|
||||
@@ -739,13 +739,13 @@ with `downstream_timeout`. Keep `downstream_timeout` longer than
|
||||
`upstream_fetch_timeout` plus 20 seconds.
|
||||
|
||||
A conversion whose request's `downstream_timeout` has ended stops waiting for a
|
||||
processing slot. Once it has one, it gives the slot back without beginning to
|
||||
encode. For a shared conversion, the first request's deadline counts. When a
|
||||
shared conversion stops this way, each request waiting for it is answered 503
|
||||
`server busy, try again later`, even if its own `downstream_timeout` has not
|
||||
ended. govips, the library pixa drives libvips through, gives no way to stop an
|
||||
encode once it has begun. So an encode keeps its slot until it finishes, which
|
||||
can be after `downstream_timeout`.
|
||||
processing slot. If its `downstream_timeout` ends after it has a slot but before
|
||||
it begins to encode, it gives the slot back. For a shared conversion, the first
|
||||
request's deadline counts. When a shared conversion stops this way, each request
|
||||
waiting for it is answered 503 `server busy, try again later`, even if its own
|
||||
`downstream_timeout` has not ended. govips, the library pixa drives libvips
|
||||
through, gives no way to stop an encode once it has begun. So an encode keeps
|
||||
its slot until it finishes, which can be after `downstream_timeout`.
|
||||
|
||||
**Encode times.** The limits on image size bound how long a conversion can take.
|
||||
They are 8192 on each side of the source and of the output, and 16,777,216
|
||||
@@ -762,20 +762,20 @@ it, encoding 8192x8192 random pixels takes up to about:
|
||||
|
||||
An AVIF output of 4096x4096, 8192x2048 or 2048x8192 random pixels takes up to
|
||||
about 34 seconds with an alpha channel, and 22 without. At 8192x8192 and quality
|
||||
85 it would take about 96 seconds. These times hold at any quality and at 8 or
|
||||
16 bits per sample. No encode within the limits takes more than about 40 seconds
|
||||
of the default `downstream_timeout` of 60.
|
||||
85 it would take about 96 seconds with an alpha channel, and 52 without. These
|
||||
times hold at any quality and at 8 or 16 bits per sample. No encode within the
|
||||
limits takes more than about 40 seconds of the default `downstream_timeout`
|
||||
of 60.
|
||||
|
||||
**Processing and connections.** `max_concurrent_processing` images are decoded
|
||||
and encoded at once. A request that finds every slot in use waits up to 10
|
||||
seconds for one. If none frees up, it is answered 503
|
||||
`server busy, try again later`, unless `downstream_timeout` has ended first.
|
||||
`upstream_connections` bounds the fetches in progress to all upstream hosts
|
||||
together, and `upstream_connections_per_host` those to one host. A fetch holds
|
||||
its connection until its image has been processed. A fetch that finds every
|
||||
shared connection in use waits up to 10 seconds the same way, then answers 503.
|
||||
A fetch that finds every connection to its host in use waits for one until
|
||||
`downstream_timeout` ends.
|
||||
`server busy, try again later`. `upstream_connections` bounds the fetches in
|
||||
progress to all upstream hosts together, and `upstream_connections_per_host`
|
||||
those to one host. A fetch holds its connection until its image has been
|
||||
processed. A fetch that finds every shared connection in use waits up to 10
|
||||
seconds the same way, then answers 503. A fetch that finds every connection to
|
||||
its host in use waits for one until `downstream_timeout` ends.
|
||||
|
||||
**Client limits.** The login form (`POST /`) is limited to 5 attempts per minute
|
||||
per client address. An attempt over the limit is refused with 429 and a
|
||||
@@ -789,18 +789,17 @@ limit off. Both limits count an IPv6 client by its `/64`.
|
||||
from `X-Forwarded-For` only when the request arrives from an address in
|
||||
`trusted_proxies`. Otherwise all users behind the proxy count as one client. The
|
||||
address the request arrives from is not always the proxy's own. A proxy on the
|
||||
Docker host that connects to pixa over `127.0.0.1` is seen as the gateway of the
|
||||
container's Docker network, such as `172.17.0.1` on the default bridge. A proxy
|
||||
that connects through another of the host's addresses is seen with that address.
|
||||
To be sure, read the address as `remoteIP` in pixa's request log while it is not
|
||||
in `trusted_proxies` (see `trusted_proxies` under Configuration). The default
|
||||
`trusted_proxies` is the RFC 1918 ranges, the private address ranges. With that
|
||||
default, a client with a private address can choose the address it is counted
|
||||
by, through its own `X-Forwarded-For`, because its own address is trusted too.
|
||||
Setting `trusted_proxies` to only the proxy's address closes this. Behind a CDN
|
||||
or proxy whose addresses are not in `trusted_proxies`, all requests arriving
|
||||
from one address count as one client. Such a deployment needs a higher
|
||||
`image_requests_per_minute`, or `0`.
|
||||
Docker host may connect to pixa over `127.0.0.1`. pixa then sees the gateway of
|
||||
the container's Docker network, such as `172.17.0.1` on the default bridge. A
|
||||
proxy that connects through another of the host's addresses is seen with that
|
||||
address. To be sure, find the address as described under `trusted_proxies` (see
|
||||
Configuration). The default `trusted_proxies` is the RFC 1918 ranges, the
|
||||
private address ranges. With that default, a client with a private address is
|
||||
itself trusted. It can then choose the address it is counted by, through its own
|
||||
`X-Forwarded-For`. Setting `trusted_proxies` to only the proxy's address closes
|
||||
this. Behind a content delivery network (CDN) or proxy whose addresses are not
|
||||
in `trusted_proxies`, all requests arriving from one address count as one
|
||||
client. Such a deployment needs a higher `image_requests_per_minute`, or `0`.
|
||||
|
||||
**Request limits.** pixa reads a request's headers within 10 seconds and the
|
||||
whole request within 30 seconds. Headers are limited to 8 KiB. An idle
|
||||
@@ -816,8 +815,9 @@ with 413.
|
||||
|
||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||
name. The database is one SQLite file. SQLite is the database library pixa
|
||||
embeds; nothing else needs to run. In WAL mode, the default, SQLite keeps
|
||||
`state.sqlite3-wal` and `state.sqlite3-shm` beside it while pixa runs.
|
||||
embeds; nothing else needs to run. By default the database is in WAL mode,
|
||||
SQLite's write-ahead log (see `db_url`). SQLite then keeps `state.sqlite3-wal`
|
||||
and `state.sqlite3-shm` beside the file while pixa runs.
|
||||
|
||||
The image cache keeps these tables in the database:
|
||||
|
||||
@@ -855,22 +855,23 @@ naming the path and the error, and goes on with the rest of the check. A file
|
||||
renamed or removed after pixa listed its directory gets no warning.
|
||||
|
||||
A check reads the database once for every cached file, so the checks are spaced
|
||||
out. After a check ends, the next one waits at least ten times as long as that
|
||||
check took. After a check that took one minute, the next one starts 10 to 15
|
||||
minutes after that check ended.
|
||||
out. After a check ends, pixa waits at least ten times as long as that check
|
||||
took. Eviction runs every five minutes (below). The next check starts at the
|
||||
first of those runs after that wait. After a check that took one minute, the
|
||||
next one starts 10 to 15 minutes after that check ended.
|
||||
|
||||
A cache hit opens the variant's file and nothing else. It makes no database
|
||||
call. The content type comes from the file's first bytes, which are those of the
|
||||
format pixa wrote it in. A file whose first bytes are not those of a format pixa
|
||||
writes is not served, and the image is converted again. A hit hands the time it
|
||||
was served to a background writer and does not wait for the writer. Eviction
|
||||
orders by that time. The hit is counted in memory. A miss that uses a cached
|
||||
source also hands that writer the time it used the source. Once a second the
|
||||
writer writes the waiting times in one transaction. Up to 10,000 access times of
|
||||
variants, and as many of sources, can wait to be written. A time handed over
|
||||
beyond that is dropped. Its image then only looks less recently used to
|
||||
eviction. `script/bench` measures how many hits pixa serves per second (see Load
|
||||
Test).
|
||||
orders by that time. The hit is counted in memory, for the
|
||||
`pixa_cache_hits_total` metric. A miss that uses a cached source also hands that
|
||||
writer the time it used the source. Once a second the writer writes the waiting
|
||||
times in one transaction. Up to 10,000 access times of variants, and as many of
|
||||
sources, can wait to be written. A time handed over beyond that is dropped. Its
|
||||
image then only looks less recently used to eviction. `script/bench` measures
|
||||
how many hits pixa serves per second (see Load Test).
|
||||
|
||||
Eviction removes the least recently used sources and variants until the cache is
|
||||
within `cache_max_bytes`. It runs after each store, and every five minutes. It
|
||||
@@ -879,25 +880,26 @@ also removes the expired rows of the negative cache every five minutes, at most
|
||||
|
||||
### Configuration
|
||||
|
||||
Every setting can be given as an environment variable, in a YAML config file
|
||||
(`--config`), or both. A variable present in the environment wins over the file,
|
||||
even when it is empty. The file wins over the built-in default. A variable's
|
||||
value is read as the same text in quotes in the file would be. A setting that
|
||||
holds a list takes comma-separated entries, with the spaces around each trimmed.
|
||||
An empty variable is an empty list.
|
||||
Every setting can be given as an environment variable, in a config file
|
||||
(`--config`), or both. The file is YAML, a plain-text format. A variable present
|
||||
in the environment wins over the file, even when it is empty. The file wins over
|
||||
the built-in default. pixa reads a variable's value as a quoted string in the
|
||||
file. So `PIXA_DEBUG=1` turns debug on, but `debug: 1` in the file aborts
|
||||
startup. A setting that holds a list takes comma-separated entries, with the
|
||||
spaces around each trimmed. An empty variable is an empty list.
|
||||
|
||||
The file's `env:` section is the one exception to that order. pixa sets each
|
||||
variable named there while it loads the file. So that variable wins over the
|
||||
environment pixa started with, and over the file's key for the same setting.
|
||||
|
||||
pixa refuses to start when a value does not parse or is invalid. The error names
|
||||
the setting. It also refuses to start when a key in the file is set to null,
|
||||
`key:` with nothing after it included. Omit the key to use the default. It
|
||||
refuses to start when the file has a key that is not in the list below, and the
|
||||
error names the key. It refuses to start when a variable starts with `PIXA_` but
|
||||
is not in the list below, such as a misspelled one or `PIXA_PORT`. The error
|
||||
names the variable. The variables set by the file's `env:` section are checked
|
||||
the same way.
|
||||
the setting. It also refuses to start when a key in the file is set to null. A
|
||||
key written as `key:` with nothing after it is null. Omit the key to use the
|
||||
default. It refuses to start when the file has a key that is not in the list
|
||||
below, and the error names the key. It refuses to start when a variable starts
|
||||
with `PIXA_` but is not in the list below, such as a misspelled one or
|
||||
`PIXA_PORT`. The error names the variable. The variables set by the file's
|
||||
`env:` section are checked the same way.
|
||||
|
||||
pixa also accepts `PIXA_CONFIG_PATH`, which names the config file, like
|
||||
`--config`.
|
||||
@@ -916,7 +918,14 @@ cannot tell. With no file, pixa uses the environment and the defaults.
|
||||
Durations are written as Go durations, such as `30s` or `2m`, and must be
|
||||
positive. A bare number has no unit and aborts startup. Sizes are a whole number
|
||||
of bytes. A CIDR range is an address range written as an address and a prefix
|
||||
length, such as `10.0.0.0/8`.
|
||||
length, such as `10.0.0.0/8`. An address matches a range that contains it.
|
||||
|
||||
`blocked_networks` and `trusted_proxies` hold CIDR ranges. For both, an IPv6
|
||||
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
||||
address it holds. So an IPv6 range such as `::/0` matches neither it nor a plain
|
||||
IPv4 address. A range written in that form is read as the IPv4 range it covers:
|
||||
`::ffff:10.0.0.0/104` is `10.0.0.0/8`. Such a range shorter than `/96` covers
|
||||
more than IPv4 addresses and aborts startup.
|
||||
|
||||
Each item below is one setting, named by its config key:
|
||||
|
||||
@@ -936,16 +945,16 @@ Each item below is one setting, named by its config key:
|
||||
- `db_url`: variable `PIXA_DB_URL`, default
|
||||
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`. The SQLite
|
||||
database to open. The default keeps the database in WAL mode, SQLite's
|
||||
write-ahead log, which lets reads go on during a write. In the default,
|
||||
`state_dir` is percent-encoded: a `%`, `?` or `#` in the name is written as
|
||||
`%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one starting with `//`,
|
||||
the default starts with `file:////`, so SQLite does not read the first name in
|
||||
the path as a host. pixa adds `_pragma=busy_timeout(5000)` to any `db_url`. A
|
||||
write that finds another program writing to the file then waits up to five
|
||||
seconds for it instead of failing. pixa opens one connection, so its own reads
|
||||
and writes run one at a time. WAL mode comes only from the URL: keep
|
||||
`_pragma=journal_mode(WAL)` in one you set. An empty `db_url` aborts startup:
|
||||
omit the key to use the default.
|
||||
write-ahead log, which lets reads go on during a write. pixa adds
|
||||
`_pragma=busy_timeout(5000)` to any `db_url`. A write that finds another
|
||||
program writing to the file then waits up to five seconds for it instead of
|
||||
failing. pixa opens one connection, so its own reads and writes run one at a
|
||||
time. WAL mode comes only from the URL: keep `_pragma=journal_mode(WAL)` in
|
||||
one you set. An empty `db_url` aborts startup: omit the key to use the
|
||||
default. In the default, `state_dir` is percent-encoded: a `%`, `?` or `#` in
|
||||
the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`, or one
|
||||
starting with `//`, the default starts with `file:////`, so SQLite does not
|
||||
read the first name in the path as a host.
|
||||
- `cache_max_bytes`: variable `PIXA_CACHE_MAX_BYTES`, default below. The disk
|
||||
cache limit in bytes. When the key is omitted, pixa works the limit out at
|
||||
startup. It takes the free space on the filesystem holding
|
||||
@@ -960,23 +969,19 @@ Each item below is one setting, named by its config key:
|
||||
you control. An entry that is neither a host name nor an IP address aborts
|
||||
startup.
|
||||
- `referer_blocklist`: variable `PIXA_REFERER_BLOCKLIST`, default none. Hosts
|
||||
whose pages may not show pixa's images, to stop other sites hotlinking them.
|
||||
Entries are written and matched as for `allowlist_hosts`. A request to
|
||||
`/v1/image/` or `/v1/e/` whose `Referer` header names a listed host is refused
|
||||
with 403. That happens before its signature or token is checked, and before
|
||||
the cache or the upstream host is used. It is refused even when the image is
|
||||
cached. A request with no `Referer`, or one that does not parse as a URL with
|
||||
a host, is served, as many clients send none. So a site whose pages send no
|
||||
`Referer`, for example with `Referrer-Policy: no-referrer`, is not stopped.
|
||||
The login and generator pages are not covered.
|
||||
whose pages may not show pixa's images, to stop other sites embedding them.
|
||||
Entries are written and matched as for `allowlist_hosts`. A request whose
|
||||
`Referer` header names a listed host is refused with 403. That applies to
|
||||
`/v1/image/` and `/v1/e/`. It happens before the signature or token is
|
||||
checked, and before the cache or the upstream host is used. The request is
|
||||
refused even when the image is cached. A request with no `Referer` is served,
|
||||
as many clients send none. So is one whose `Referer` does not parse as a URL
|
||||
with a host. So a site whose pages send no `Referer`, for example with
|
||||
`Referrer-Policy: no-referrer`, is not stopped. The login and generator pages
|
||||
are not covered.
|
||||
- `blocked_networks`: variable `PIXA_BLOCKED_NETWORKS`, default none. CIDR
|
||||
ranges pixa never fetches from, on top of the built-in ones. An invalid CIDR
|
||||
aborts startup. An IPv6 address in the IPv4-mapped form (`::ffff:a.b.c.d`) is
|
||||
matched only as the IPv4 address it holds. So an IPv6 range such as `::/0`
|
||||
refuses neither it nor a plain IPv4 address. An entry written in that form is
|
||||
read as the IPv4 range it covers: `::ffff:10.0.0.0/104` refuses what
|
||||
`10.0.0.0/8` refuses. Such an entry shorter than `/96` covers more than IPv4
|
||||
addresses and aborts startup. The built-in ranges always apply. They are:
|
||||
aborts startup. The built-in ranges always apply. They are:
|
||||
- `0.0.0.0/8`, this network, `0.0.0.0` included;
|
||||
- `10.0.0.0/8`, `172.16.0.0/12` and `192.168.0.0/16`, the private ranges;
|
||||
- `100.64.0.0/10`, carrier-grade NAT;
|
||||
@@ -1000,12 +1005,11 @@ Each item below is one setting, named by its config key:
|
||||
came from is used. So a client connecting from outside these ranges cannot
|
||||
choose its address. An explicitly empty list (`[]`, or an empty variable)
|
||||
trusts no one. A list you set replaces the default. An invalid CIDR aborts
|
||||
startup. An IPv4-mapped entry or address is handled as under
|
||||
`blocked_networks`. Set this to the address pixa sees for requests that come
|
||||
through your proxy, such as `172.17.0.1/32`. Do so when the default does not
|
||||
cover it, or to trust nothing else (see Limits). To find that address, set
|
||||
this to `[]`, send a request through the proxy, and read `remoteIP` in pixa's
|
||||
log line for it.
|
||||
startup. Set this to the address pixa sees for requests that come through your
|
||||
proxy, such as `172.17.0.1/32`. Do so when the default does not cover it, or
|
||||
to trust nothing else (see Limits). To find that address, set this to `[]`,
|
||||
send a request through the proxy, and read `remoteIP` in pixa's log line for
|
||||
it.
|
||||
- `allow_http`: variable `PIXA_ALLOW_HTTP`, default `false`. While `true`, pixa
|
||||
fetches every source over plain HTTP, and follows redirects to plain HTTP. It
|
||||
is for testing only.
|
||||
@@ -1070,11 +1074,11 @@ Each item below is one setting, named by its config key:
|
||||
|
||||
### Architecture
|
||||
|
||||
- **Dependency injection**: Uber fx, a library that builds the program's parts
|
||||
in dependency order and starts and stops them. `internal/app` puts every part
|
||||
of `pixad` in one fx graph. `internal/providers` builds the image cache, the
|
||||
image proxy and the other parts that serve images and logins. Each setting's
|
||||
default is declared once, in `internal/config`.
|
||||
- **Dependency injection**: `fx` (`go.uber.org/fx`), a library that builds the
|
||||
program's parts in dependency order and starts and stops them. `internal/app`
|
||||
puts every part of `pixad` in one `fx` graph. `internal/providers` builds the
|
||||
image cache, the image proxy and the other parts that serve images and logins.
|
||||
Each setting's default is declared once, in `internal/config`.
|
||||
- **Packages**: these packages under `internal/` work with the concepts above:
|
||||
- `handlers`: the routes, which read the image request from the URL, check
|
||||
the signed or encrypted URL, and make the login session
|
||||
@@ -1090,7 +1094,7 @@ Each item below is one setting, named by its config key:
|
||||
- `seal`: the keys derived from `signing_key`, and the encryption of those
|
||||
tokens
|
||||
- `session`: the login session
|
||||
- **HTTP router**: go-chi
|
||||
- **HTTP router**: `chi` (`github.com/go-chi/chi/v5`)
|
||||
- **Image processing**: govips, a Go wrapper for libvips
|
||||
- **Database**: SQLite via `modernc.org/sqlite`
|
||||
- **Static assets**: embedded in the binary via `//go:embed`
|
||||
@@ -1112,9 +1116,9 @@ Each item below is one setting, named by its config key:
|
||||
A request's `handler` label is the pattern of the route it matches, such as
|
||||
`/v1/image/*`, never its path. Requests that match no route share the label
|
||||
`unmatched`: those the router answers 404 or 405, and CORS preflights. A
|
||||
request's `method` label is its method when that is `GET`, `HEAD`, `POST`,
|
||||
`PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` or `TRACE`, and `other` for
|
||||
any other method.
|
||||
request's `method` label is its method, for these nine: `GET`, `HEAD`,
|
||||
`POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. Any
|
||||
other method is labelled `other`.
|
||||
|
||||
- **Logging**: the standard library's `slog`, as JSON on standard output, or as
|
||||
text when standard output is a terminal.
|
||||
@@ -1124,25 +1128,25 @@ Each item below is one setting, named by its config key:
|
||||
This repository adheres to the
|
||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||
standard. The scripts in `script/` are the entrypoints for the development
|
||||
workflow, and most Makefile targets are thin shims that call them. We provide:
|
||||
workflow, and most `Makefile` targets are thin shims that call them. We provide:
|
||||
|
||||
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier, and
|
||||
- `script/bootstrap` — install `git`, `make`, Go, Node, Yarn and `prettier`, and
|
||||
download the Go modules. It can be run again at any time. With `--cgo`, it
|
||||
installs a C compiler, `pkg-config`, libvips (with its JPEG XL support) and
|
||||
libheif instead of Node, Yarn and prettier. Compiling pixa outside Docker, as
|
||||
`make build` and `script/bench` do, needs those. Docker itself must already be
|
||||
installed.
|
||||
- `script/setup` — make a fresh clone ready for development: bootstrap, then
|
||||
install-precommit.
|
||||
libheif instead of Node, Yarn and `prettier`. Compiling pixa outside Docker,
|
||||
as `make build` and `script/bench` do, needs those. Docker itself must already
|
||||
be installed.
|
||||
- `script/setup` — make a fresh clone ready for development: `script/bootstrap`,
|
||||
then `script/install-precommit`.
|
||||
- `script/projectname` — output the project name (`pixa`).
|
||||
- `script/test` — run the test suite: build the `test` phase of the
|
||||
`Dockerfile`, writing no image.
|
||||
- `script/lint` — run `golangci-lint`: build the `lint` phase of the
|
||||
`Dockerfile`, writing no image. The linter never runs on the host.
|
||||
- `script/fmt` — format the Go code with `gofmt` and the markdown with prettier
|
||||
(writes).
|
||||
- `script/fmt` — format the Go code with `gofmt` and the markdown with
|
||||
`prettier` (writes).
|
||||
- `script/fmt-check` — check the same formatting (read-only), on the host.
|
||||
- `script/check` — run test, lint, and fmt-check.
|
||||
- `script/check` — run `script/test`, `script/lint` and `script/fmt-check`.
|
||||
- `script/docker` — build the Docker image tagged via `script/projectname`, with
|
||||
the version from `git describe`. The image's build stage depends on the `lint`
|
||||
and `test` phases, so this runs them too.
|
||||
@@ -1191,7 +1195,7 @@ them:
|
||||
- `script/install-precommit`
|
||||
|
||||
`script/fmt` and `script/fmt-check` are the standard copies with pixa's `gofmt`
|
||||
step kept before prettier. prettier formats the markdown only, not the HTML
|
||||
step kept before `prettier`. `prettier` formats the markdown only, not the HTML
|
||||
templates, as it cannot parse a Go template action inside a tag.
|
||||
|
||||
## Load Test
|
||||
@@ -1229,8 +1233,8 @@ For each scenario the script prints vegeta's report and two lines of its own:
|
||||
- `Requests [total, rate, throughput]`: the requests sent, how many were sent
|
||||
per second, and how many were answered successfully per second;
|
||||
- `Latencies [min, mean, 50, 90, 95, 99, max]`: the time from sending a request
|
||||
to the end of its answer. `50`, `95` and `99` are the 50th, 95th and 99th
|
||||
percentiles;
|
||||
to the end of its answer. `50`, `90`, `95` and `99` are the 50th, 90th, 95th
|
||||
and 99th percentiles;
|
||||
- `Status Codes` and `Error Set`: anything other than `200` means the other
|
||||
numbers are not for the scenario described, such as `503` when `pixad` was
|
||||
busy;
|
||||
@@ -1246,22 +1250,22 @@ The numbers depend on the machine and on whatever else runs on it.
|
||||
|
||||
`script/bench` (or `make bench`) measures cache hits alone, without the network
|
||||
and with no `pixad` process. A Go benchmark in `internal/handlers` serves one
|
||||
cached image through the `/v1/image/` handler from as many goroutines at once as
|
||||
the machine has CPUs. It reports the hits served per second as `hits/s`. Each
|
||||
run starts with an empty cache and an empty database, a SQLite file of its own,
|
||||
and puts the image in the cache before it measures. The benchmark fails if
|
||||
serving the hits ran any statement on the database. Like the load test it is not
|
||||
part of `script/check`. It runs Go on the host, as the load test does, and
|
||||
compiles pixa there. So it needs the C compiler and libraries that
|
||||
`script/bootstrap --cgo` installs, but not Docker.
|
||||
cached image through the `/v1/image/` handler from several goroutines at once,
|
||||
one per CPU. A goroutine is a Go thread. The benchmark reports the hits served
|
||||
per second as `hits/s`. Each run starts with an empty cache and an empty
|
||||
database, a SQLite file of its own, and puts the image in the cache before it
|
||||
measures. The benchmark fails if serving the hits ran any statement on the
|
||||
database. Like the load test it is not part of `script/check`. It runs Go on the
|
||||
host, as the load test does, and compiles pixa there. So it needs the C compiler
|
||||
and libraries that `script/bootstrap --cgo` installs, but not Docker.
|
||||
|
||||
## TODO
|
||||
|
||||
See [TODO.md](TODO.md) for the full prioritized task list.
|
||||
See [`TODO.md`](TODO.md) for the full prioritized task list.
|
||||
|
||||
## License
|
||||
|
||||
GPL-3.0. See [LICENSE](LICENSE).
|
||||
GPL-3.0. See [`LICENSE`](LICENSE).
|
||||
|
||||
## Author
|
||||
|
||||
|
||||
Reference in New Issue
Block a user