Format the markdown with prettier in script/fmt and script/fmt-check (closes #100) #219
@@ -4,73 +4,68 @@ Last Updated 2026-01-08
|
|||||||
|
|
||||||
These rules MUST be followed at all times, it is very important.
|
These rules MUST be followed at all times, it is very important.
|
||||||
|
|
||||||
* Never use `git add -A` - add specific changes to a deliberate commit. A
|
- Never use `git add -A` - add specific changes to a deliberate commit. A commit
|
||||||
commit should contain one change. After each change, make a commit with a
|
should contain one change. After each change, make a commit with a good
|
||||||
good one-line summary.
|
one-line summary.
|
||||||
|
|
||||||
* NEVER modify the linter config without asking first.
|
- NEVER modify the linter config without asking first.
|
||||||
|
|
||||||
* NEVER modify tests to exclude special cases or otherwise get them to pass
|
- NEVER modify tests to exclude special cases or otherwise get them to pass
|
||||||
without asking first. In almost all cases, the code should be changed,
|
without asking first. In almost all cases, the code should be changed, NOT the
|
||||||
NOT the tests. If you think the test needs to be changed, make your case
|
tests. If you think the test needs to be changed, make your case for that and
|
||||||
for that and ask for permission to proceed, then stop. You need explicit
|
ask for permission to proceed, then stop. You need explicit user approval to
|
||||||
user approval to modify existing tests. (You do not need user approval
|
modify existing tests. (You do not need user approval for writing NEW tests.)
|
||||||
for writing NEW tests.)
|
|
||||||
|
|
||||||
* When linting, assume the linter config is CORRECT, and that each item
|
- When linting, assume the linter config is CORRECT, and that each item output
|
||||||
output by the linter is something that legitimately needs fixing in the
|
by the linter is something that legitimately needs fixing in the code.
|
||||||
code.
|
|
||||||
|
|
||||||
* When running tests, use `make test`.
|
- When running tests, use `make test`.
|
||||||
|
|
||||||
* Before commits, run `make check`. This runs `make lint` and `make test`
|
- Before commits, run `make check`. This runs `make lint` and `make test` and
|
||||||
and `make check-fmt`. Any issues discovered MUST be resolved before
|
`make check-fmt`. Any issues discovered MUST be resolved before committing
|
||||||
committing unless explicitly told otherwise.
|
unless explicitly told otherwise.
|
||||||
|
|
||||||
* When fixing a bug, write a failing test for the bug FIRST. Add
|
- When fixing a bug, write a failing test for the bug FIRST. Add appropriate
|
||||||
appropriate logging to the test to ensure it is written correctly. Commit
|
logging to the test to ensure it is written correctly. Commit that. Then go
|
||||||
that. Then go about fixing the bug until the test passes (without
|
about fixing the bug until the test passes (without modifying the test
|
||||||
modifying the test further). Then commit that.
|
further). Then commit that.
|
||||||
|
|
||||||
* When adding a new feature, do the same - implement a test first (TDD). It
|
- When adding a new feature, do the same - implement a test first (TDD). It
|
||||||
doesn't have to be super complex. Commit the test, then commit the
|
doesn't have to be super complex. Commit the test, then commit the feature.
|
||||||
feature.
|
|
||||||
|
|
||||||
* When adding a new feature, use a feature branch. When the feature is
|
- When adding a new feature, use a feature branch. When the feature is
|
||||||
completely finished and the code is up to standards (passes `make check`)
|
completely finished and the code is up to standards (passes `make check`) then
|
||||||
then and only then can the feature branch be merged into `main` and the
|
and only then can the feature branch be merged into `main` and the branch
|
||||||
branch deleted.
|
deleted.
|
||||||
|
|
||||||
* Write godoc documentation comments for all exported types and functions as
|
- Write godoc documentation comments for all exported types and functions as you
|
||||||
you go along.
|
go along.
|
||||||
|
|
||||||
* ALWAYS be consistent in naming. If you name something one thing in one
|
- ALWAYS be consistent in naming. If you name something one thing in one place,
|
||||||
place, name it the EXACT SAME THING in another place.
|
name it the EXACT SAME THING in another place.
|
||||||
|
|
||||||
* Be descriptive and specific in naming. `wl` is bad;
|
- Be descriptive and specific in naming. `wl` is bad; `SourceHostWhitelist` is
|
||||||
`SourceHostWhitelist` is good. `ConnsPerHost` is bad;
|
good. `ConnsPerHost` is bad; `MaxConnectionsPerHost` is good.
|
||||||
`MaxConnectionsPerHost` is good.
|
|
||||||
|
|
||||||
* This is not prototype or teaching code - this is designed for production.
|
- This is not prototype or teaching code - this is designed for production. Any
|
||||||
Any security issues (such as denial of service) or other web
|
security issues (such as denial of service) or other web vulnerabilities are
|
||||||
vulnerabilities are P1 bugs and must be added to TODO.md at the top.
|
P1 bugs and must be added to TODO.md at the top.
|
||||||
|
|
||||||
* As this is production code, no stubbing of implementations unless
|
- As this is production code, no stubbing of implementations unless specifically
|
||||||
specifically instructed. We need working implementations.
|
instructed. We need working implementations.
|
||||||
|
|
||||||
* NEVER silently fall back to a different setting when a user's parameter
|
- NEVER silently fall back to a different setting when a user's parameter
|
||||||
explicitly specifies a value. If a user requests format=webp and WebP
|
explicitly specifies a value. If a user requests format=webp and WebP encoding
|
||||||
encoding is not supported, return an error - do NOT silently output PNG
|
is not supported, return an error - do NOT silently output PNG instead. If a
|
||||||
instead. If a user specifies fit=invalid and that fit mode doesn't exist,
|
user specifies fit=invalid and that fit mode doesn't exist, return an error -
|
||||||
return an error - do NOT silently default to "cover". Silent fallbacks
|
do NOT silently default to "cover". Silent fallbacks violate the principle of
|
||||||
violate the principle of least surprise and mask bugs. The only acceptable
|
least surprise and mask bugs. The only acceptable defaults are for OMITTED
|
||||||
defaults are for OMITTED parameters, never for INVALID explicit values.
|
parameters, never for INVALID explicit values.
|
||||||
|
|
||||||
* Avoid vendoring deps unless specifically instructed to. NEVER commit
|
- Avoid vendoring deps unless specifically instructed to. NEVER commit the
|
||||||
the vendor directory, NEVER commit compiled binaries. If these
|
vendor directory, NEVER commit compiled binaries. If these directories or
|
||||||
directories or files exist, add them to .gitignore (and commit the
|
files exist, add them to .gitignore (and commit the .gitignore) if they are
|
||||||
.gitignore) if they are not already in there. Keep the entire git
|
not already in there. Keep the entire git repository (with history) small -
|
||||||
repository (with history) small - under 20MiB, unless you specifically
|
under 20MiB, unless you specifically must commit larger files (e.g. test
|
||||||
must commit larger files (e.g. test fixture example media files). Only
|
fixture example media files). Only OUR source code and immediately supporting
|
||||||
OUR source code and immediately supporting files (such as test examples)
|
files (such as test examples) goes into the repo/history.
|
||||||
goes into the repo/history.
|
|
||||||
|
|||||||
@@ -1,10 +1,9 @@
|
|||||||
# pixa
|
# pixa
|
||||||
|
|
||||||
pixa is a GPL-3.0-licensed Go web server by
|
pixa is a GPL-3.0-licensed Go web server by [@sneak](https://sneak.berlin) that
|
||||||
[@sneak](https://sneak.berlin) that proxies images from upstream
|
proxies images from upstream sources, optionally resizing or transforming them,
|
||||||
sources, optionally resizing or transforming them, and serves the
|
and serves the results. Both source and transformed images are cached to disk so
|
||||||
results. Both source and transformed images are cached to disk so that
|
that subsequent requests are served without origin fetches or additional
|
||||||
subsequent requests are served without origin fetches or additional
|
|
||||||
processing.
|
processing.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
@@ -27,12 +26,11 @@ make docker
|
|||||||
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
|
docker run -p 8080:8080 -e PIXA_SIGNING_KEY="$(openssl rand -base64 32)" pixa:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
A container takes its settings from environment variables (see
|
A container takes its settings from environment variables (see Configuration
|
||||||
Configuration below for the list). Only `PIXA_SIGNING_KEY` is required; if
|
below for the list). Only `PIXA_SIGNING_KEY` is required; if it is unset the
|
||||||
it is unset the container exits at startup naming the variable. Everything
|
container exits at startup naming the variable. Everything else has a built-in
|
||||||
else has a built-in default. A config file mounted at `/etc/pixa/config.yml`
|
default. A config file mounted at `/etc/pixa/config.yml` is optional: it is read
|
||||||
is optional: it is read when present, and an environment variable wins over
|
when present, and an environment variable wins over the same setting in it.
|
||||||
the same setting in it.
|
|
||||||
|
|
||||||
## Deployment
|
## Deployment
|
||||||
|
|
||||||
@@ -104,32 +102,30 @@ already be installed.
|
|||||||
What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
||||||
|
|
||||||
- **Port:** pixa listens on container port `8080`.
|
- **Port:** pixa listens on container port `8080`.
|
||||||
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its
|
- **Volume:** container path `/var/lib/pixa`, where pixa keeps its database and
|
||||||
database and cache. Creating the host directory when it is missing is
|
cache. Creating the host directory when it is missing is upaas's job, tracked
|
||||||
upaas's job, tracked in https://git.eeqj.de/sneak/upaas/issues/235.
|
in https://git.eeqj.de/sneak/upaas/issues/235.
|
||||||
- **Environment variables:**
|
- **Environment variables:**
|
||||||
- `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs
|
- `PIXA_SIGNING_KEY` (required): secret for signed and encrypted URLs and
|
||||||
and login, 32+ characters, for example from
|
login, 32+ characters, for example from `openssl rand -base64 32`
|
||||||
`openssl rand -base64 32`
|
|
||||||
- `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
|
- `PIXA_ALLOWLIST_HOSTS`: upstream hosts served without a signature,
|
||||||
comma-separated
|
comma-separated
|
||||||
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
|
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
|
||||||
default 75% of (free space + what the cache holds)
|
default 75% of (free space + what the cache holds)
|
||||||
- the rest are in the table under Configuration below
|
- the rest are in the table under Configuration below
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json`. upaas reads the container's health 60
|
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
||||||
seconds after a deploy and marks the deploy failed unless it is
|
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
||||||
`healthy`. The probe uses the port from `PORT` (default `8080`), so a
|
uses the port from `PORT` (default `8080`), so a port changed only in a
|
||||||
port changed only in a mounted config file is not seen by it: change
|
mounted config file is not seen by it: change the port with `PORT`.
|
||||||
the port with `PORT`.
|
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
Image-heavy web applications need a fast, caching reverse proxy that
|
Image-heavy web applications need a fast, caching reverse proxy that can resize
|
||||||
can resize and transcode images on the fly. pixa fills that role as a
|
and transcode images on the fly. pixa fills that role as a single,
|
||||||
single, self-contained binary with no external runtime dependencies
|
self-contained binary with no external runtime dependencies beyond libvips. It
|
||||||
beyond libvips. It supports HMAC-SHA256 signed URLs with expiration to
|
supports HMAC-SHA256 signed URLs with expiration to prevent abuse, and
|
||||||
prevent abuse, and allowlisted source hosts for open access.
|
allowlisted source hosts for open access.
|
||||||
|
|
||||||
## Design
|
## Design
|
||||||
|
|
||||||
@@ -138,8 +134,8 @@ prevent abuse, and allowlisted source hosts for open access.
|
|||||||
- **Source content**:
|
- **Source content**:
|
||||||
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
`<state_dir>/cache/sources/<ab>/<cd>/<sha256 of source content>`
|
||||||
- **Source metadata**:
|
- **Source metadata**:
|
||||||
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json`
|
`<state_dir>/cache/metadata/<hostname>/<sha256 of path and query>.json` (host,
|
||||||
(host, path and query, content hash, upstream status and headers, fetch time)
|
path and query, content hash, upstream status and headers, fetch time)
|
||||||
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
- **Database**: `<state_dir>/state.sqlite3` (SQLite)
|
||||||
- **Transformed images**:
|
- **Transformed images**:
|
||||||
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
`<state_dir>/cache/variants/<ab>/<cd>/<sha256 of host, path, query, size, format, quality and fit>`,
|
||||||
@@ -148,8 +144,8 @@ prevent abuse, and allowlisted source hosts for open access.
|
|||||||
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
`<ab>` and `<cd>` are the first and second pairs of characters of the file's
|
||||||
name.
|
name.
|
||||||
|
|
||||||
Multiple source paths may reference the same content blob; the
|
Multiple source paths may reference the same content blob; the database tracks
|
||||||
database tracks references rather than using filesystem refcounting.
|
references rather than using filesystem refcounting.
|
||||||
|
|
||||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||||
@@ -164,8 +160,8 @@ answers any method as it answers `GET`. A browser's CORS preflight request
|
|||||||
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
|
(`OPTIONS` with `Origin` and `Access-Control-Request-Method` headers) to any
|
||||||
path under `/v1/` answers 200, in maintenance mode too.
|
path under `/v1/` answers 200, in maintenance mode too.
|
||||||
|
|
||||||
- `GET /` — the login page, or the URL generator page with a login session
|
- `GET /` — the login page, or the URL generator page with a login session (see
|
||||||
(see Encrypted URLs). Needs: nothing. Answers: 200.
|
Encrypted URLs). Needs: nothing. Answers: 200.
|
||||||
- `POST /` — log in with the signing key typed into the login page. Needs: the
|
- `POST /` — log in with the signing key typed into the login page. Needs: the
|
||||||
login page's form (below). Answers: 303 to `/` with a login session cookie
|
login page's form (below). Answers: 303 to `/` with a login session cookie
|
||||||
that lasts 30 days for the right key; 200 with the login page and an error for
|
that lasts 30 days for the right key; 200 with the login page and an error for
|
||||||
@@ -198,11 +194,10 @@ path under `/v1/` answers 200, in maintenance mode too.
|
|||||||
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
|
||||||
nothing. Answers: 200.
|
nothing. Answers: 200.
|
||||||
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
|
||||||
`uptime_seconds`, `uptime_human`, `version`, `appname` and
|
`uptime_seconds`, `uptime_human`, `version`, `appname` and `maintenance_mode`.
|
||||||
`maintenance_mode`. Needs: nothing. Answers: 200, always.
|
Needs: nothing. Answers: 200, always.
|
||||||
- `GET /static/<file>` — the stylesheet and script the login and generator
|
- `GET /static/<file>` — the stylesheet and script the login and generator pages
|
||||||
pages load. Needs: nothing. Answers: 200, or 404 for a file that does not
|
load. Needs: nothing. Answers: 200, or 404 for a file that does not exist.
|
||||||
exist.
|
|
||||||
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
|
- `GET /metrics` — Prometheus metrics (see Architecture). Needs: HTTP basic
|
||||||
authentication with `metrics.username` and `metrics.password`. Answers: 200;
|
authentication with `metrics.username` and `metrics.password`. Answers: 200;
|
||||||
401 without them; 404 when they are not set, as the route then does not exist.
|
401 without them; 404 when they are not set, as the route then does not exist.
|
||||||
@@ -226,9 +221,9 @@ HTTP is for development on the browser's own machine: the login session cookie
|
|||||||
is always marked `Secure`, and over plain HTTP a browser keeps such a cookie
|
is always marked `Secure`, and over plain HTTP a browser keeps such a cookie
|
||||||
only for its own machine (`localhost`), if at all. A form is also refused with
|
only for its own machine (`localhost`), if at all. A form is also refused with
|
||||||
403 when the page's host is not the `Host` header pixa receives, so a reverse
|
403 when the page's host is not the `Host` header pixa receives, so a reverse
|
||||||
proxy in front of pixa must pass that header on unchanged. A form body over
|
proxy in front of pixa must pass that header on unchanged. A form body over 1
|
||||||
1 MiB is refused with 413. The image routes answer the errors listed for them
|
MiB is refused with 413. The image routes answer the errors listed for them with
|
||||||
with JSON holding `error`, `status` and `timestamp`.
|
JSON holding `error`, `status` and `timestamp`.
|
||||||
|
|
||||||
An image URL has this form:
|
An image URL has this form:
|
||||||
|
|
||||||
@@ -240,8 +235,8 @@ Images are only fetched from origins using TLS with valid certificates, unless
|
|||||||
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
|
`allow_http` is set: then pixa fetches every image over plain HTTP, which is for
|
||||||
testing only.
|
testing only.
|
||||||
|
|
||||||
A request whose query string cannot be decoded, or gives any parameter more
|
A request whose query string cannot be decoded, or gives any parameter more than
|
||||||
than once, is refused with 400.
|
once, is refused with 400.
|
||||||
|
|
||||||
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
- `<format>`: one of `orig` (or `original`), `jpeg` (or `jpg`), `png`, `webp`,
|
||||||
`avif`, `gif`
|
`avif`, `gif`
|
||||||
@@ -249,39 +244,37 @@ than once, is refused with 400.
|
|||||||
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
- `sig` and `exp`: the signature and its expiry, needed unless the host is
|
||||||
allowlisted (see Signature Specification)
|
allowlisted (see Signature Specification)
|
||||||
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
|
- `q` and `fit`: the output quality and how the image is fitted to `<size>`,
|
||||||
both optional (values under Signature Specification). Both are part of what
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
is cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
An image is served with `Cache-Control: public, max-age=<seconds>, immutable`.
|
||||||
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL),
|
When the URL has an expiry (an `exp`, or the TTL of an encrypted URL), `max-age`
|
||||||
`max-age` is the whole seconds left until then, at most one year, so no browser
|
is the whole seconds left until then, at most one year, so no browser or proxy
|
||||||
or proxy cache keeps the image after pixa would refuse the URL. A URL with no
|
cache keeps the image after pixa would refuse the URL. A URL with no expiry gets
|
||||||
expiry gets one year. `immutable` only stops a client revalidating while its
|
one year. `immutable` only stops a client revalidating while its copy is fresh.
|
||||||
copy is fresh.
|
|
||||||
|
|
||||||
When several requests for the same image, size, format, quality and fit miss
|
When several requests for the same image, size, format, quality and fit miss the
|
||||||
the cache at once, they share one upstream fetch (or one read of the cached
|
cache at once, they share one upstream fetch (or one read of the cached source)
|
||||||
source) and one transcode: the first request does the work, and the others wait
|
and one transcode: the first request does the work, and the others wait for its
|
||||||
for its image or its error, holding no upstream connection or processing slot
|
image or its error, holding no upstream connection or processing slot of their
|
||||||
of their own. A waiting request stops waiting when its own client goes away.
|
own. A waiting request stops waiting when its own client goes away. The work
|
||||||
The work goes on for the others even if the first request's client goes away,
|
goes on for the others even if the first request's client goes away, until that
|
||||||
until that request's `downstream_timeout` ends. The shared fetch sends the first
|
request's `downstream_timeout` ends. The shared fetch sends the first request's
|
||||||
request's ID upstream, and the lines logged for the fetch and the transcode
|
ID upstream, and the lines logged for the fetch and the transcode carry that ID.
|
||||||
carry that ID.
|
|
||||||
|
|
||||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||||
refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
|
refused with 429 and a `Retry-After` header. Behind a reverse proxy the client
|
||||||
address comes from `X-Forwarded-For` only when the address pixa sees for
|
address comes from `X-Forwarded-For` only when the address pixa sees for
|
||||||
requests that come through the proxy is in `trusted_proxies`; otherwise all
|
requests that come through the proxy is in `trusted_proxies`; otherwise all
|
||||||
users behind the proxy are counted as one client. That address is not always
|
users behind the proxy are counted as one client. That address is not always the
|
||||||
the proxy's own: a proxy on the Docker host that connects to pixa over
|
proxy's own: a proxy on the Docker host that connects to pixa over `127.0.0.1`
|
||||||
`127.0.0.1` is seen as the gateway of the container's Docker network, such as
|
is seen as the gateway of the container's Docker network, such as `172.17.0.1`
|
||||||
`172.17.0.1` on the default bridge, and one that connects through another of the
|
on the default bridge, and one that connects through another of the host's
|
||||||
host's addresses is seen with that address. To be sure, read it as `remoteIP` in
|
addresses is seen with that address. To be sure, read it as `remoteIP` in pixa's
|
||||||
pixa's request log while it is not in `trusted_proxies` (see `trusted_proxies`
|
request log while it is not in `trusted_proxies` (see `trusted_proxies` under
|
||||||
under Configuration). With the default `trusted_proxies` (the RFC 1918 ranges),
|
Configuration). With the default `trusted_proxies` (the RFC 1918 ranges), a
|
||||||
a client with a private address can choose the address it is counted by through
|
client with a private address can choose the address it is counted by through
|
||||||
its own `X-Forwarded-For`, whether it connects directly or through the proxy,
|
its own `X-Forwarded-For`, whether it connects directly or through the proxy,
|
||||||
because its own address is trusted too. Setting `trusted_proxies` to only the
|
because its own address is trusted too. Setting `trusted_proxies` to only the
|
||||||
address pixa sees for requests that come through the proxy closes this.
|
address pixa sees for requests that come through the proxy closes this.
|
||||||
@@ -306,18 +299,18 @@ nor change what it asks for.
|
|||||||
with `http` instead while `debug` is on. The name after the token is ignored
|
with `http` instead while `debug` is on. The name after the token is ignored
|
||||||
and only gives the URL a file extension, `jpg` for `orig`.
|
and only gives the URL a file extension, `jpg` for `orig`.
|
||||||
|
|
||||||
The token holds the source's host, path and query and the size, format,
|
The token holds the source's host, path and query and the size, format, quality,
|
||||||
quality, fit and expiry, encrypted with a key derived from `signing_key`. The
|
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
||||||
source URL's scheme is not kept: the image is fetched like any other (see
|
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
||||||
Routes), and the blocked networks still apply.
|
the blocked networks still apply.
|
||||||
|
|
||||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or
|
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||||
never. The expiry is fixed in the token when the URL is made and cannot be
|
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||||
changed or revoked afterwards. Until then the image is served with a `max-age`
|
revoked afterwards. Until then the image is served with a `max-age` that ends at
|
||||||
that ends at the expiry (see Routes); after it the URL answers 410
|
the expiry (see Routes); after it the URL answers 410 `URL has expired`. A URL
|
||||||
`URL has expired`. A URL made to last forever stops working only when
|
made to last forever stops working only when `signing_key` changes: changing it
|
||||||
`signing_key` changes: changing it makes every encrypted URL already handed out
|
makes every encrypted URL already handed out answer 400, and ends every login
|
||||||
answer 400, and ends every login session.
|
session.
|
||||||
|
|
||||||
### Image Metadata
|
### Image Metadata
|
||||||
|
|
||||||
@@ -336,16 +329,15 @@ turned off.
|
|||||||
|
|
||||||
### Source Hosts
|
### Source Hosts
|
||||||
|
|
||||||
Source hosts may be allowlisted in the configuration. Non-allowlisted
|
Source hosts may be allowlisted in the configuration. Non-allowlisted hosts
|
||||||
hosts require an HMAC-SHA256 signature.
|
require an HMAC-SHA256 signature.
|
||||||
|
|
||||||
#### Signature Specification
|
#### Signature Specification
|
||||||
|
|
||||||
Signatures use HMAC-SHA256 and include an expiration timestamp to
|
Signatures use HMAC-SHA256 and include an expiration timestamp to prevent replay
|
||||||
prevent replay attacks. Signatures are **exact match only**: every
|
attacks. Signatures are **exact match only**: every component (host, path,
|
||||||
component (host, path, query, dimensions, format, expiration, quality,
|
query, dimensions, format, expiration, quality, fit) must match exactly what was
|
||||||
fit) must match exactly what was signed. No suffix matching, wildcard
|
signed. No suffix matching, wildcard matching, or partial matching is supported.
|
||||||
matching, or partial matching is supported.
|
|
||||||
|
|
||||||
**Signed data format** (colon-separated):
|
**Signed data format** (colon-separated):
|
||||||
|
|
||||||
@@ -362,24 +354,24 @@ Where:
|
|||||||
- `height` — requested height in pixels, `0` for original
|
- `height` — requested height in pixels, `0` for original
|
||||||
- `format` — output format, one of those listed under Routes, with `original`
|
- `format` — output format, one of those listed under Routes, with `original`
|
||||||
signed as `orig` and `jpg` as `jpeg`
|
signed as `orig` and `jpg` as `jpeg`
|
||||||
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when
|
- `expiration` — the URL's `exp` query parameter, the Unix timestamp when the
|
||||||
the signature expires; a request whose `exp` is not a whole number, an
|
signature expires; a request whose `exp` is not a whole number, an empty
|
||||||
empty `exp=` included, is refused with 400
|
`exp=` included, is refused with 400
|
||||||
- `quality` — the URL's `q` query parameter, a whole number from 1 to 100,
|
- `quality` — the URL's `q` query parameter, a whole number from 1 to 100, or
|
||||||
or `85` when the URL has no `q`; a request whose `q` is anything else is
|
`85` when the URL has no `q`; a request whose `q` is anything else is refused
|
||||||
refused with 400
|
with 400
|
||||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||||
anything else, an empty `fit=` included, is refused with 400
|
anything else, an empty `fit=` included, is refused with 400
|
||||||
|
|
||||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet
|
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||||
of RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||||
compares it exactly, so a signature encoded without padding, as Node's
|
compares it exactly, so a signature encoded without padding, as Node's
|
||||||
`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401.
|
`base64url` and Go's `base64.RawURLEncoding` do, is refused with 401.
|
||||||
|
|
||||||
**Example:** with the signing key `example-signing-key-for-documentation`,
|
**Example:** with the signing key `example-signing-key-for-documentation`,
|
||||||
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with
|
resize `https://cdn.example.com/photos/cat.jpg` to 800x600 WebP with expiration
|
||||||
expiration 1704067200, default quality and fit:
|
1704067200, default quality and fit:
|
||||||
|
|
||||||
1. Build input:
|
1. Build input:
|
||||||
`cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
|
`cdn.example.com:/photos/cat.jpg::800:600:webp:1704067200:85:cover`
|
||||||
@@ -407,29 +399,29 @@ or a `*.` wildcard, aborts startup.
|
|||||||
|
|
||||||
### Configuration
|
### Configuration
|
||||||
|
|
||||||
Every setting can be given as an environment variable, in a YAML config
|
Every setting can be given as an environment variable, in a YAML config file
|
||||||
file (`--config`), or both. A variable present in the environment wins over
|
(`--config`), or both. A variable present in the environment wins over the file,
|
||||||
the file, even when it is empty, and the file wins over the built-in
|
even when it is empty, and the file wins over the built-in default. The one
|
||||||
default. The one exception is a variable named in the file's `env:` section:
|
exception is a variable named in the file's `env:` section: it is set while the
|
||||||
it is set while the file loads, so it overrides both the environment the
|
file loads, so it overrides both the environment the process was started with
|
||||||
process was started with and the file's own key. A variable's value is
|
and the file's own key. A variable's value is parsed as the same text in the
|
||||||
parsed as the same text in the file would be. The three lists take
|
file would be. The three lists take comma-separated entries, with the spaces
|
||||||
comma-separated entries, with the spaces around each trimmed; an empty
|
around each trimmed; an empty variable is an empty list. A value that does not
|
||||||
variable is an empty list. A value that does not parse or is invalid aborts
|
parse or is invalid aborts startup, naming the variable. A variable whose name
|
||||||
startup, naming the variable. A variable whose name starts with `PIXA_` but
|
starts with `PIXA_` but is not in the table below, such as a misspelled one or
|
||||||
is not in the table below, such as a misspelled one or `PIXA_PORT`, aborts
|
`PIXA_PORT`, aborts startup naming it, as an unknown config key does. The one
|
||||||
startup naming it, as an unknown config key does. The one other accepted
|
other accepted name is `PIXA_CONFIG_PATH`, the config file's path (like
|
||||||
name is `PIXA_CONFIG_PATH`, the config file's path (like `--config`). The
|
`--config`). The variables set by the file's `env:` section are checked the same
|
||||||
variables set by the file's `env:` section are checked the same way.
|
way.
|
||||||
|
|
||||||
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
pixa reads at most one config file: the one given with `--config` (or `-c`),
|
||||||
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
otherwise the one `PIXA_CONFIG_PATH` names, otherwise the first of these that
|
||||||
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
pixa finds: `/etc/pixa/config.yml`, `/etc/pixa/config.yaml`,
|
||||||
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml`
|
`~/.config/pixa/config.yml`, `~/.config/pixa/config.yaml`, then `config.yml` and
|
||||||
and `config.yaml` in the working directory. A named file that does not exist,
|
`config.yaml` in the working directory. A named file that does not exist, cannot
|
||||||
cannot be read or does not parse aborts startup. Of the files pixa looks for on
|
be read or does not parse aborts startup. Of the files pixa looks for on its
|
||||||
its own, only one that does not exist is passed over, without a message. One
|
own, only one that does not exist is passed over, without a message. One that
|
||||||
that pixa cannot read or parse aborts startup, naming the file. So does one in a
|
pixa cannot read or parse aborts startup, naming the file. So does one in a
|
||||||
directory pixa may not enter, whether or not it is there, since pixa cannot
|
directory pixa may not enter, whether or not it is there, since pixa cannot
|
||||||
tell. With no file, pixa uses the environment and the defaults.
|
tell. With no file, pixa uses the environment and the defaults.
|
||||||
|
|
||||||
@@ -464,11 +456,11 @@ Key settings in more detail:
|
|||||||
of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
|
of the image routes, `/v1/image/` and `/v1/e/`, sent as the CORS
|
||||||
`Access-Control-Allow-Origin` header; no other route sends it. `*`, the
|
`Access-Control-Allow-Origin` header; no other route sends it. `*`, the
|
||||||
default, is any site; otherwise one `http` or `https` origin such as
|
default, is any site; otherwise one `http` or `https` origin such as
|
||||||
`https://example.com`, whose host is a lowercase host name (letters,
|
`https://example.com`, whose host is a lowercase host name (letters, digits,
|
||||||
digits, hyphens and dots, with a letter in its last part) or an IP address
|
hyphens and dots, with a letter in its last part) or an IP address (IPv6 in
|
||||||
(IPv6 in brackets, in its shortest form), with an optional port 1-65535
|
brackets, in its shortest form), with an optional port 1-65535 that has no
|
||||||
that has no leading zero and is not the scheme's default. Any other value,
|
leading zero and is not the scheme's default. Any other value, including
|
||||||
including another scheme such as a browser extension's, aborts startup
|
another scheme such as a browser extension's, aborts startup
|
||||||
- `allowlist_hosts` — list of allowed upstream hosts
|
- `allowlist_hosts` — list of allowed upstream hosts
|
||||||
- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
|
- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
|
||||||
stop other sites hotlinking them. Entries are written and matched as for
|
stop other sites hotlinking them. Entries are written and matched as for
|
||||||
@@ -477,40 +469,37 @@ Key settings in more detail:
|
|||||||
`/v1/e/` whose `Referer` header names a listed host is refused with 403 before
|
`/v1/e/` whose `Referer` header names a listed host is refused with 403 before
|
||||||
its signature or token is checked and before the cache or the upstream host is
|
its signature or token is checked and before the cache or the upstream host is
|
||||||
used, so it fetches nothing, and it is refused even when the image is cached.
|
used, so it fetches nothing, and it is refused even when the image is cached.
|
||||||
A request with no `Referer`, or one that does not parse as a URL
|
A request with no `Referer`, or one that does not parse as a URL with a host,
|
||||||
with a host, is served, as many clients send none. So this is easily got
|
is served, as many clients send none. So this is easily got around: a site
|
||||||
around: a site whose pages send no `Referer` (for example with
|
whose pages send no `Referer` (for example with
|
||||||
`Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
|
`Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
|
||||||
and generator pages. Default: empty
|
and generator pages. Default: empty
|
||||||
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection,
|
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
||||||
added to the always-enforced built-in ranges (loopback, private,
|
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
||||||
link-local, CGNAT, benchmark, NAT64, and the like); an invalid CIDR
|
benchmark, NAT64, and the like); an invalid CIDR aborts startup
|
||||||
aborts startup
|
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
||||||
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front
|
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
||||||
of pixa. `X-Forwarded-For` is believed only when the direct peer falls
|
of these ranges; the logged and login-recorded client address is then the
|
||||||
inside one of these ranges; the logged and login-recorded client
|
rightmost forwarded entry that is not itself a trusted proxy. Otherwise the
|
||||||
address is then the rightmost forwarded entry that is not itself a
|
direct peer address is used and the header is ignored, so a client connecting
|
||||||
trusted proxy. Otherwise the direct peer address is used and the header
|
directly from an address outside these ranges cannot spoof its address. An
|
||||||
is ignored, so a client connecting directly from an address outside
|
omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
|
||||||
these ranges cannot spoof its address.
|
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a
|
||||||
An omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
|
private network; an explicitly empty list (`[]`) trusts no one, and an
|
||||||
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a
|
explicit list replaces the default. An invalid CIDR aborts startup. Set this
|
||||||
proxy on a private network; an explicitly empty list (`[]`) trusts no
|
to the address pixa sees for requests that come through your proxy, such as
|
||||||
one, and an explicit list replaces the default. An invalid CIDR aborts
|
`172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else
|
||||||
startup. Set this to the address pixa sees for requests that come through
|
(see the login limit under Routes). For a proxy on the Docker host that
|
||||||
your proxy, such as `172.17.0.1/32`, when the defaults do not cover it, or
|
connects to pixa over `127.0.0.1`, that address is the gateway of the
|
||||||
to trust nothing else (see the login limit under Routes). For a proxy on
|
container's Docker network (`172.17.0.1` on the default bridge), not the
|
||||||
the Docker host that connects to pixa over `127.0.0.1`, that address is the
|
proxy's own address; a proxy that connects through another of the host's
|
||||||
gateway of the container's Docker network (`172.17.0.1` on the default
|
addresses is seen with that address. To be sure which address it is, set this
|
||||||
bridge), not the proxy's own address; a proxy that connects through another of
|
to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request through the
|
||||||
the host's addresses is seen with that address. To be sure which address it
|
proxy, and read `remoteIP` in pixa's request log line for it
|
||||||
is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
|
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
|
||||||
through the proxy, and read `remoteIP` in pixa's request log line for it
|
as a duration such as `30s` (the default) or `2m`
|
||||||
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream
|
- `upstream_max_response_size` — largest upstream response accepted, in bytes;
|
||||||
host, as a duration such as `30s` (the default) or `2m`
|
default `52428800` (50 MiB). It also limits the image data pixa decodes
|
||||||
- `upstream_max_response_size` — largest upstream response accepted, in
|
|
||||||
bytes; default `52428800` (50 MiB). It also limits the image data pixa
|
|
||||||
decodes
|
|
||||||
- `downstream_timeout` — time allowed for answering one client request, as a
|
- `downstream_timeout` — time allowed for answering one client request, as a
|
||||||
duration; default `60s`. The upstream fetch counts toward it, and so do the
|
duration; default `60s`. The upstream fetch counts toward it, and so do the
|
||||||
waits for an upstream connection and for a processing slot (up to 10 seconds
|
waits for an upstream connection and for a processing slot (up to 10 seconds
|
||||||
@@ -522,11 +511,11 @@ Key settings in more detail:
|
|||||||
so a write that finds another in progress waits up to five seconds for it
|
so a write that finds another in progress waits up to five seconds for it
|
||||||
instead of failing. WAL mode comes only from the URL: keep
|
instead of failing. WAL mode comes only from the URL: keep
|
||||||
`_pragma=journal_mode(WAL)` in one you set
|
`_pragma=journal_mode(WAL)` in one you set
|
||||||
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the
|
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
|
||||||
disk cache entirely; omitted defaults to 75% of the sum of the free space on
|
cache entirely; omitted defaults to 75% of the sum of the free space on the
|
||||||
the filesystem containing `<state_dir>/cache/` and the bytes of source and
|
filesystem containing `<state_dir>/cache/` and the bytes of source and
|
||||||
transformed images the cache already holds, worked out at startup (minimum
|
transformed images the cache already holds, worked out at startup (minimum 500
|
||||||
500 MiB)
|
MiB)
|
||||||
- `upstream_connections` — the most connections to upstream hosts at once, all
|
- `upstream_connections` — the most connections to upstream hosts at once, all
|
||||||
hosts together, on top of `upstream_connections_per_host`; default `64`. A
|
hosts together, on top of `upstream_connections_per_host`; default `64`. A
|
||||||
fetch holds its connection until its image has been processed. A fetch that
|
fetch holds its connection until its image has been processed. A fetch that
|
||||||
@@ -541,10 +530,10 @@ Key settings in more detail:
|
|||||||
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
|
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
|
||||||
`/v1/e/`) answer every request for an image with 503, a `Retry-After` header
|
`/v1/e/`) answer every request for an image with 503, a `Retry-After` header
|
||||||
and a JSON error body. The health check (`/.well-known/healthcheck.json`)
|
and a JSON error body. The health check (`/.well-known/healthcheck.json`)
|
||||||
still answers 200 and reports `"maintenance_mode": true`. It stays 200
|
still answers 200 and reports `"maintenance_mode": true`. It stays 200 because
|
||||||
because the image's Docker `HEALTHCHECK` requests it: a 503 there would make
|
the image's Docker `HEALTHCHECK` requests it: a 503 there would make the
|
||||||
the container unhealthy, and upaas marks a deploy failed when its container
|
container unhealthy, and upaas marks a deploy failed when its container is
|
||||||
is unhealthy. The login and URL generator pages and `/metrics` keep working
|
unhealthy. The login and URL generator pages and `/metrics` keep working
|
||||||
|
|
||||||
See `configs/config.example.yml` for all options with defaults.
|
See `configs/config.example.yml` for all options with defaults.
|
||||||
|
|
||||||
@@ -566,15 +555,15 @@ See `configs/config.example.yml` for all options with defaults.
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and the Makefile targets are thin shims that call
|
development workflow, and the Makefile targets are thin shims that call them. We
|
||||||
them. We provide:
|
provide:
|
||||||
|
|
||||||
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and
|
- `script/bootstrap` — install git, make, Go, Node, Yarn and prettier and
|
||||||
download the Go modules (idempotent); with `--cgo`, the C compiler and the
|
download the Go modules (idempotent); with `--cgo`, the C compiler and the
|
||||||
libvips and libheif libraries that compiling pixa needs instead of Node, Yarn
|
libvips and libheif libraries that compiling pixa needs instead of Node, Yarn
|
||||||
and prettier
|
and prettier
|
||||||
- `script/setup` — make a fresh clone ready for development
|
- `script/setup` — make a fresh clone ready for development (bootstrap, then
|
||||||
(bootstrap, then install-precommit)
|
install-precommit)
|
||||||
- `script/projectname` — output the project name ("pixa")
|
- `script/projectname` — output the project name ("pixa")
|
||||||
- `script/test` — run the test suite: build the `test` phase of the
|
- `script/test` — run the test suite: build the `test` phase of the
|
||||||
`Dockerfile`, tagged `pixa-test`
|
`Dockerfile`, tagged `pixa-test`
|
||||||
@@ -594,8 +583,8 @@ them. We provide:
|
|||||||
then `script/check`, then build the image as `script/docker` does
|
then `script/check`, then build the image as `script/docker` does
|
||||||
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
- `script/precommit` — pre-commit checks (`go mod tidy` guard, then
|
||||||
`script/check`)
|
`script/check`)
|
||||||
- `script/install-precommit` — install the git pre-commit hook that
|
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||||
runs `script/precommit`
|
`script/precommit`
|
||||||
|
|
||||||
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
|
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
|
||||||
phases run on every build instead of coming from the build cache.
|
phases run on every build instead of coming from the build cache.
|
||||||
|
|||||||
@@ -1,28 +1,27 @@
|
|||||||
# Workflow
|
# Workflow
|
||||||
|
|
||||||
* branch per issue from `next`
|
- branch per issue from `next`
|
||||||
* do the work in Next Step
|
- do the work in Next Step
|
||||||
* move Next Step to the top of Completed Steps
|
- move Next Step to the top of Completed Steps
|
||||||
* `TODO.md` merges with git's union merge (`.gitattributes`), which never
|
- `TODO.md` merges with git's union merge (`.gitattributes`), which never
|
||||||
reports a conflict: read the merged entries after every merge or rebase
|
reports a conflict: read the merged entries after every merge or rebase
|
||||||
* move the top item of Future Steps into Next Step
|
- move the top item of Future Steps into Next Step
|
||||||
* commit (`TODO.md` changes in the same commit as the work)
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
* open a PR based on `next`
|
- open a PR based on `next`
|
||||||
* an independent reviewer who did not write the change gates it
|
- an independent reviewer who did not write the change gates it
|
||||||
* the manager squash-merges the PR into `next` once review passes
|
- the manager squash-merges the PR into `next` once review passes
|
||||||
* `next` stays green and mergeable to `main` at any time; only the owner
|
- `next` stays green and mergeable to `main` at any time; only the owner merges
|
||||||
merges `next` into `main`, via the single milestone PR
|
`next` into `main`, via the single milestone PR
|
||||||
* push
|
- push
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work
|
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
|
||||||
lands on `next`, and `main` receives only the milestone PR that the
|
`next`, and `main` receives only the milestone PR that the owner merges. `next`
|
||||||
owner merges. `next` is at the canonical `golangci-lint` v2.12.2 config
|
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
|
||||||
and is green. Recent work extracted the internal/magic,
|
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
|
||||||
internal/allowlist, internal/httpfetcher, and internal/signature
|
internal/signature packages. The gosec findings from the 2026-07-06 survey are
|
||||||
packages. The gosec findings from the 2026-07-06 survey are resolved.
|
resolved. The disk cache is now size-bounded with LRU eviction
|
||||||
The disk cache is now size-bounded with LRU eviction
|
|
||||||
(`cache_max_bytes`), closing the unbounded disk growth DoS vector.
|
(`cache_max_bytes`), closing the unbounded disk growth DoS vector.
|
||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
@@ -31,16 +30,16 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- 2026-10-05 the markdown is formatted with prettier (closes #100):
|
- 2026-10-05 the markdown is formatted with prettier (closes #100): `script/fmt`
|
||||||
`script/fmt` and `script/fmt-check` run prettier 3.8.1, pinned in
|
and `script/fmt-check` run prettier 3.8.1, pinned in `package.json` and
|
||||||
`package.json` and `yarn.lock`, on `**/*.md` after `gofmt`, with four-space
|
`yarn.lock`, on `**/*.md` after `gofmt`, with four-space tabs and
|
||||||
tabs and `proseWrap: always` as `.prettierrc` says; `.prettierignore` keeps
|
`proseWrap: always` as `.prettierrc` says; `.prettierignore` keeps it off
|
||||||
it off `REPO_POLICIES.md`, the copy from `sneak/prompts`, and `vendor/`.
|
`REPO_POLICIES.md`, the copy from `sneak/prompts`, and `vendor/`. Plain
|
||||||
Plain `script/bootstrap` installs Node and Yarn as the one in `sneak/prompts`
|
`script/bootstrap` installs Node and Yarn as the one in `sneak/prompts` does
|
||||||
does and then prettier; `script/bootstrap --cgo` does not, as the
|
and then prettier; `script/bootstrap --cgo` does not, as the `Dockerfile`
|
||||||
`Dockerfile` stages that run it format nothing. The HTML templates stay
|
stages that run it format nothing. The HTML templates stay unformatted:
|
||||||
unformatted: prettier cannot parse a Go template action inside a tag. The
|
prettier cannot parse a Go template action inside a tag. The markdown was
|
||||||
markdown was reflowed in a commit of its own.
|
reflowed in a commit of its own.
|
||||||
- 2026-10-05 lint and tests run as the `lint` and `test` phases of the
|
- 2026-10-05 lint and tests run as the `lint` and `test` phases of the
|
||||||
`Dockerfile`, built with `--no-cache` (closes #202): `script/check`,
|
`Dockerfile`, built with `--no-cache` (closes #202): `script/check`,
|
||||||
`script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
`script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||||
@@ -113,13 +112,12 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
(the workflow's `script/docker-smoke` step),
|
(the workflow's `script/docker-smoke` step),
|
||||||
https://git.eeqj.de/sneak/pixa/issues/204 (`.claude/` in `.gitignore`),
|
https://git.eeqj.de/sneak/pixa/issues/204 (`.claude/` in `.gitignore`),
|
||||||
https://git.eeqj.de/sneak/pixa/issues/205 (`.dockerignore` patterns at every
|
https://git.eeqj.de/sneak/pixa/issues/205 (`.dockerignore` patterns at every
|
||||||
depth), https://git.eeqj.de/sneak/pixa/issues/206 (a thin
|
depth), https://git.eeqj.de/sneak/pixa/issues/206 (a thin `cmd/pixad/main.go`)
|
||||||
`cmd/pixad/main.go`) and https://git.eeqj.de/sneak/pixa/issues/208
|
and https://git.eeqj.de/sneak/pixa/issues/208 (`fetch-depth: 0` on the CI
|
||||||
(`fetch-depth: 0` on the CI checkout, so the build sees the tags). Its rule
|
checkout, so the build sees the tags). Its rule that no build stage runs
|
||||||
that no build stage runs `git describe` is not followed: pixa takes the
|
`git describe` is not followed: pixa takes the version from the `.git` in the
|
||||||
version from the `.git` in the build context, per
|
build context, per https://git.eeqj.de/sneak/pixa/issues/166, as the copy on
|
||||||
https://git.eeqj.de/sneak/pixa/issues/166, as the copy on `sneak/prompts`
|
`sneak/prompts` `next` already says.
|
||||||
`next` already says.
|
|
||||||
- 2026-10-04 an integration test of the image proxy flow (closes #80):
|
- 2026-10-04 an integration test of the image proxy flow (closes #80):
|
||||||
`TestImageProxyFlow` in `internal/server` starts the database, handlers and
|
`TestImageProxyFlow` in `internal/server` starts the database, handlers and
|
||||||
middleware from the constructors `pixad` uses, with a fresh state directory,
|
middleware from the constructors `pixad` uses, with a fresh state directory,
|
||||||
@@ -135,13 +133,12 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
`httpfetcher.Config.DialContext` connects in place of the dialer that refuses
|
`httpfetcher.Config.DialContext` connects in place of the dialer that refuses
|
||||||
internal addresses, the URL and redirect checks still running, and
|
internal addresses, the URL and redirect checks still running, and
|
||||||
`handlers.Params.Fetcher` replaces the fetcher the handlers build.
|
`handlers.Params.Fetcher` replaces the fetcher the handlers build.
|
||||||
- 2026-10-04 a URL made on the generator page with a `ttl` is tested to
|
- 2026-10-04 a URL made on the generator page with a `ttl` is tested to expire
|
||||||
expire (closes #199): a new test in `internal/handlers` makes a URL on the
|
(closes #199): a new test in `internal/handlers` makes a URL on the generator
|
||||||
generator page with a `ttl` of one second, checks that `/v1/e/` serves it at
|
page with a `ttl` of one second, checks that `/v1/e/` serves it at once, waits
|
||||||
once, waits two seconds and checks that it then answers 410. The test waits
|
two seconds and checks that it then answers 410. The test waits for real, as
|
||||||
for real, as pixa reads the clock directly when it makes and checks a URL; it
|
pixa reads the clock directly when it makes and checks a URL; it waits two
|
||||||
waits two seconds because the time a URL expires is kept in whole seconds.
|
seconds because the time a URL expires is kept in whole seconds. Test only.
|
||||||
Test only.
|
|
||||||
- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
|
- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
|
||||||
(`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
|
(`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
|
||||||
`allowlist_hosts` with the same matcher; an entry of either list that is
|
`allowlist_hosts` with the same matcher; an entry of either list that is
|
||||||
@@ -159,30 +156,29 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
`README.md`, the comments in `internal/config/config.go` and the startup error
|
`README.md`, the comments in `internal/config/config.go` and the startup error
|
||||||
for the placeholder signing key name the new path; `scripts/manual-test.sh`
|
for the placeholder signing key name the new path; `scripts/manual-test.sh`
|
||||||
and its directory are deleted, as the handler tests in `internal/handlers`
|
and its directory are deleted, as the handler tests in `internal/handlers`
|
||||||
cover every check it made except two: fetching a real image from the
|
cover every check it made except two: fetching a real image from the internet,
|
||||||
internet, and a URL made on the generator page with a `ttl` answering 410 once
|
and a URL made on the generator page with a `ttl` answering 410 once the `ttl`
|
||||||
the `ttl` has passed (https://git.eeqj.de/sneak/pixa/issues/199);
|
has passed (https://git.eeqj.de/sneak/pixa/issues/199); `CONVENTIONS.md` is
|
||||||
`CONVENTIONS.md` is deleted, as `REPO_POLICIES.md` links the canonical Go HTTP
|
deleted, as `REPO_POLICIES.md` links the canonical Go HTTP server conventions.
|
||||||
server conventions.
|
|
||||||
- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes
|
- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes
|
||||||
#198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write
|
#198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write
|
||||||
that finds another in progress on another connection waits up to five seconds
|
that finds another in progress on another connection waits up to five seconds
|
||||||
for it, and the default `db_url` turns on WAL mode with
|
for it, and the default `db_url` turns on WAL mode with
|
||||||
`_pragma=journal_mode(WAL)`. The old default's `_journal_mode=WAL` is not a
|
`_pragma=journal_mode(WAL)`. The old default's `_journal_mode=WAL` is not a
|
||||||
parameter the driver reads, so the database was never in WAL mode.
|
parameter the driver reads, so the database was never in WAL mode.
|
||||||
- 2026-10-04 `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup`
|
- 2026-10-04 `TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup` only
|
||||||
only passes through a periodic pass (closes #189): it slept for three
|
passes through a periodic pass (closes #189): it slept for three eviction
|
||||||
eviction intervals before writing its file, and a startup pass still running
|
intervals before writing its file, and a startup pass still running then could
|
||||||
then could adopt the file itself. It now holds the test database's only
|
adopt the file itself. It now holds the test database's only connection until
|
||||||
connection until the startup pass waits for it after walking the empty
|
the startup pass waits for it after walking the empty variant directory,
|
||||||
variant directory, writes the file and lets the connection go, as
|
writes the file and lets the connection go, as
|
||||||
`TestEvictionRunsOnPeriodicSchedule` does, so only a periodic reconciliation
|
`TestEvictionRunsOnPeriodicSchedule` does, so only a periodic reconciliation
|
||||||
pass can adopt the file. Test only.
|
pass can adopt the file. Test only.
|
||||||
- 2026-10-04 logging in, logging out, the URL generator and `/v1/e/` have
|
- 2026-10-04 logging in, logging out, the URL generator and `/v1/e/` have
|
||||||
handler tests (closes #77): new tests in `internal/handlers`, with no
|
handler tests (closes #77): new tests in `internal/handlers`, with no network,
|
||||||
network, check that `GET /` without a login session shows the login form; a
|
check that `GET /` without a login session shows the login form; a wrong key
|
||||||
wrong key shows it again with an error and sets no session cookie; the right
|
shows it again with an error and sets no session cookie; the right key answers
|
||||||
key answers 303 to `/` with a session cookie marked `Secure`, `HttpOnly` and
|
303 to `/` with a session cookie marked `Secure`, `HttpOnly` and
|
||||||
`SameSite=Strict`, with which `GET /` shows the generator page; `GET /logout`
|
`SameSite=Strict`, with which `GET /` shows the generator page; `GET /logout`
|
||||||
answers 303 to `/` with an empty session cookie sent with `Max-Age=0`;
|
answers 303 to `/` with an empty session cookie sent with `Max-Age=0`;
|
||||||
`POST /generate` without a login session answers 303 to `/`; `/v1/e/` serves
|
`POST /generate` without a login session answers 303 to `/`; `/v1/e/` serves
|
||||||
@@ -192,11 +188,11 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
- 2026-10-04 `TODO.md` merges with git's union merge (closes #190): a root
|
- 2026-10-04 `TODO.md` merges with git's union merge (closes #190): a root
|
||||||
`.gitattributes`, copied from `sneak/prompts`, marks it `merge=union`, so two
|
`.gitattributes`, copied from `sneak/prompts`, marks it `merge=union`, so two
|
||||||
branches that each add an entry at the top of Completed Steps merge without a
|
branches that each add an entry at the top of Completed Steps merge without a
|
||||||
conflict and keep both entries. Git now never reports a conflict in
|
conflict and keep both entries. Git now never reports a conflict in `TODO.md`:
|
||||||
`TODO.md`: a real one keeps both versions of the lines, and two entries that
|
a real one keeps both versions of the lines, and two entries that share an
|
||||||
share an identical line can end up one inside the other, which a rebase can
|
identical line can end up one inside the other, which a rebase can do to an
|
||||||
do to an entry already on `next`. The Workflow above says to read the merged
|
entry already on `next`. The Workflow above says to read the merged entries
|
||||||
entries after every merge or rebase.
|
after every merge or rebase.
|
||||||
- 2026-10-04 the default `cache_max_bytes` no longer shrinks as the cache fills
|
- 2026-10-04 the default `cache_max_bytes` no longer shrinks as the cache fills
|
||||||
(closes #184): for an omitted key, the cache works out the limit when it
|
(closes #184): for an omitted key, the cache works out the limit when it
|
||||||
opens, after the database is open, as 75% of the sum of the free space on the
|
opens, after the database is open, as 75% of the sum of the free space on the
|
||||||
@@ -220,10 +216,10 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
that pixa may not enter, aborts startup naming the file, as a file that does
|
that pixa may not enter, aborts startup naming the file, as a file that does
|
||||||
not parse already did.
|
not parse already did.
|
||||||
- 2026-10-04 `.golangci.yml` re-vendored from the canonical copy (closes #57):
|
- 2026-10-04 `.golangci.yml` re-vendored from the canonical copy (closes #57):
|
||||||
the deprecated `gomodguard` is switched off, so lint runs print no
|
the deprecated `gomodguard` is switched off, so lint runs print no deprecation
|
||||||
deprecation warning; its successor `gomodguard_v2` runs with the shared
|
warning; its successor `gomodguard_v2` runs with the shared module block list,
|
||||||
module block list, and `depguard` keeps `net/http/httptest` out of files that
|
and `depguard` keeps `net/http/httptest` out of files that are not tests. The
|
||||||
are not tests. The tree needed no code changes.
|
tree needed no code changes.
|
||||||
- 2026-10-04 the Content-Security-Policy allows no inline script or style
|
- 2026-10-04 the Content-Security-Policy allows no inline script or style
|
||||||
(closes #125): `script-src` and `style-src` are `'self'` only. The generator
|
(closes #125): `script-src` and `style-src` are `'self'` only. The generator
|
||||||
page's two inline `onclick` handlers moved into
|
page's two inline `onclick` handlers moved into
|
||||||
@@ -242,30 +238,30 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
counts, the health check for a load balancer, what a stop does and its exit
|
counts, the health check for a load balancer, what a stop does and its exit
|
||||||
codes, and what running outside Docker needs; `configs/Caddyfile` is the
|
codes, and what running outside Docker needs; `configs/Caddyfile` is the
|
||||||
example, checked with `caddy validate`.
|
example, checked with `caddy validate`.
|
||||||
- 2026-10-04 the metrics basic auth, CORS preflight, request logging and
|
- 2026-10-04 the metrics basic auth, CORS preflight, request logging and metrics
|
||||||
metrics recording have tests (closes #79): `MetricsAuth` on its own answers
|
recording have tests (closes #79): `MetricsAuth` on its own answers 401 with a
|
||||||
401 with a challenge without credentials or with a wrong username or password
|
challenge without credentials or with a wrong username or password and lets
|
||||||
and lets the configured ones through; a preflight request gets `*` for any
|
the configured ones through; a preflight request gets `*` for any origin when
|
||||||
origin when `access_control_allow_origin` is `*` and no
|
`access_control_allow_origin` is `*` and no `Access-Control-Allow-Origin` from
|
||||||
`Access-Control-Allow-Origin` from another origin than the configured one; a
|
another origin than the configured one; a `POST /` carrying the signing key
|
||||||
`POST /` carrying the signing key leaves no trace of it in the request log
|
leaves no trace of it in the request log line, and the login handler's own log
|
||||||
line, and the login handler's own log lines leave out the submitted key; the
|
lines leave out the submitted key; the metrics middleware on its own records a
|
||||||
metrics middleware on its own records a request it served, and the router
|
request it served, and the router records nothing while no metrics username is
|
||||||
records nothing while no metrics username is set. Not tested: that the router
|
set. Not tested: that the router puts the basic auth in front of `/metrics`
|
||||||
puts the basic auth in front of `/metrics` and records requests when a
|
and records requests when a metrics username is set. Only one test per package
|
||||||
metrics username is set. Only one test per package can set up `/metrics`, and
|
can set up `/metrics`, and in `internal/server` that is
|
||||||
in `internal/server` that is `TestMaintenanceModeKeepsOtherRoutes`, which
|
`TestMaintenanceModeKeepsOtherRoutes`, which needs the owner's approval to
|
||||||
needs the owner's approval to change; #180 holds it. Tests only; the basic
|
change; #180 holds it. Tests only; the basic auth library already compares the
|
||||||
auth library already compares the password in constant time.
|
password in constant time.
|
||||||
- 2026-10-04 the image route's signature check and error answers are tested
|
- 2026-10-04 the image route's signature check and error answers are tested
|
||||||
(closes #76): new tests in `internal/handlers`, with no network, check the
|
(closes #76): new tests in `internal/handlers`, with no network, check the
|
||||||
status and JSON error body for a missing, wrong, unpadded, upper-case or
|
status and JSON error body for a missing, wrong, unpadded, upper-case or
|
||||||
expired signature on a host not on the allowlist, or a valid one sent for
|
expired signature on a host not on the allowlist, or a valid one sent for its
|
||||||
its parent domain, a sibling host, a subdomain or the host with another
|
parent domain, a sibling host, a subdomain or the host with another domain
|
||||||
domain appended (401), an unparseable path (400), `localhost` as the
|
appended (401), an unparseable path (400), `localhost` as the upstream host
|
||||||
upstream host (403) and an upstream error (502); that an allowlisted host is
|
(403) and an upstream error (502); that an allowlisted host is served without
|
||||||
served without a signature, another host only with a valid one; and the
|
a signature, another host only with a valid one; and the answers of
|
||||||
answers of `/robots.txt` and the health check. No code changes.
|
`/robots.txt` and the health check. No code changes.
|
||||||
- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates
|
- 2026-10-04 request IDs returned and passed on, and `/v1/e/` revalidates
|
||||||
(closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each
|
(closes #84): pixa's own `RequestID` middleware, in place of chi's, gives each
|
||||||
request an ID, its own `X-Request-ID` when that is at most 64 letters, digits,
|
request an ID, its own `X-Request-ID` when that is at most 64 letters, digits,
|
||||||
@@ -282,26 +278,26 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
`Vary: Accept` is left to #88.
|
`Vary: Accept` is left to #88.
|
||||||
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
|
- 2026-10-04 routes, encrypted URLs and config file documented (closes #75):
|
||||||
"Routes" in `README.md` lists every route with its method, purpose, what it
|
"Routes" in `README.md` lists every route with its method, purpose, what it
|
||||||
needs and the status codes it answers with, and says `q` and `fit` are part
|
needs and the status codes it answers with, and says `q` and `fit` are part of
|
||||||
of what is cached; "Encrypted URLs" covers logging in, making one on the
|
what is cached; "Encrypted URLs" covers logging in, making one on the
|
||||||
generator page, how long it lasts and the 410 once it has expired;
|
generator page, how long it lasts and the 410 once it has expired;
|
||||||
"Configuration" gives the order in which pixa looks for its config file;
|
"Configuration" gives the order in which pixa looks for its config file;
|
||||||
`config.example.yml` lists `db_url` and `env` and gives every key's default;
|
`config.example.yml` lists `db_url` and `env` and gives every key's default;
|
||||||
`scripts/manual-test.sh` is left to #97.
|
`scripts/manual-test.sh` is left to #97.
|
||||||
- 2026-10-04 shutdown stops cache eviction in progress (closes #102):
|
- 2026-10-04 shutdown stops cache eviction in progress (closes #102):
|
||||||
`StartEviction` runs the eviction goroutine with its own context, which
|
`StartEviction` runs the eviction goroutine with its own context, which
|
||||||
`StopEviction` cancels, so a pass in progress stops at its next database
|
`StopEviction` cancels, so a pass in progress stops at its next database call,
|
||||||
call, file, row or eviction candidate instead of running to completion, and
|
file, row or eviction candidate instead of running to completion, and no pass
|
||||||
no pass starts after it, so a stop logs at most one warning;
|
starts after it, so a stop logs at most one warning; `StopEviction` takes a
|
||||||
`StopEviction` takes a context and, when that context ends before the
|
context and, when that context ends before the goroutine exits, stops waiting
|
||||||
goroutine exits, stops waiting and returns its error; the handlers' stop hook
|
and returns its error; the handlers' stop hook passes fx's stop context, so an
|
||||||
passes fx's stop context, so an eviction still running when fx's stop
|
eviction still running when fx's stop deadline ends fails the stop and makes
|
||||||
deadline ends fails the stop and makes the exit code 1.
|
the exit code 1.
|
||||||
- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`,
|
- 2026-10-04 dead code in `internal/imgcache` is gone (closes #73): `Purge`,
|
||||||
which only returned an error and which nothing called, is no longer part of
|
which only returned an error and which nothing called, is no longer part of
|
||||||
the `ImageCache` interface or `Service`; the `SignatureValidator`,
|
the `ImageCache` interface or `Service`; the `SignatureValidator`, `Allowlist`
|
||||||
`Allowlist` and `Storage` interfaces, which nothing implemented or used, are
|
and `Storage` interfaces, which nothing implemented or used, are deleted.
|
||||||
deleted. Nothing else changes.
|
Nothing else changes.
|
||||||
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
|
- 2026-10-04 upstream host semaphores and variant `.meta` files no longer
|
||||||
outlive their use (closes #87): the fetcher counts the fetches holding or
|
outlive their use (closes #87): the fetcher counts the fetches holding or
|
||||||
waiting for a slot of each upstream host's semaphore and removes the host's
|
waiting for a slot of each upstream host's semaphore and removes the host's
|
||||||
@@ -313,21 +309,20 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
cache directories pixa uses (`cache/sources`, `cache/metadata`,
|
cache directories pixa uses (`cache/sources`, `cache/metadata`,
|
||||||
`cache/variants`) and how files are named in each, and the comments in
|
`cache/variants`) and how files are named in each, and the comments in
|
||||||
`001_schema.sql` name the same paths; the routes and the signature section
|
`001_schema.sql` name the same paths; the routes and the signature section
|
||||||
list the same output formats, `jpg` and `original` included; the TLS
|
list the same output formats, `jpg` and `original` included; the TLS sentence
|
||||||
sentence names `allow_http` as its exception; "Metrics" says only generic
|
names `allow_http` as its exception; "Metrics" says only generic HTTP and Go
|
||||||
HTTP and Go runtime metrics exist, measured and served only when the metrics
|
runtime metrics exist, measured and served only when the metrics username and
|
||||||
username and password are set.
|
password are set.
|
||||||
- 2026-10-03 shutdown sets the exit code and waits for image processing
|
- 2026-10-03 shutdown sets the exit code and waits for image processing (closes
|
||||||
(closes #86): fx alone handles SIGINT and SIGTERM, and the server's own
|
#86): fx alone handles SIGINT and SIGTERM, and the server's own signal handler
|
||||||
signal handler is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's
|
is gone; fx's `Run` in `cmd/pixad` exits with the shutdown's code: 0 for a
|
||||||
code: 0 for a signal, 1 when the HTTP server cannot listen or the app fails
|
signal, 1 when the HTTP server cannot listen or the app fails to start or to
|
||||||
to start or to stop; the server's stop hook, which fx waits for, stops the
|
stop; the server's stop hook, which fx waits for, stops the HTTP server, waits
|
||||||
HTTP server, waits for the images still being processed, both within 5
|
for the images still being processed, both within 5 seconds, then flushes
|
||||||
seconds, then flushes Sentry; images still being processed after that are
|
Sentry; images still being processed after that are logged with their count
|
||||||
logged with their count and make the exit code 1; a Sentry DSN that cannot be
|
and make the exit code 1; a Sentry DSN that cannot be used fails startup, so
|
||||||
used fails startup, so the stop hooks of what had already started run,
|
the stop hooks of what had already started run, instead of exiting the process
|
||||||
instead of exiting the process from a goroutine; the eviction loop is left to
|
from a goroutine; the eviction loop is left to #102.
|
||||||
#102.
|
|
||||||
- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks
|
- 2026-10-03 every `script/cibuild` and `script/docker` run executes the checks
|
||||||
(closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check`
|
(closes #101): the `Dockerfile` declares `CHECK_EPOCH` above `make fmt-check`
|
||||||
and `make lint` in the lint stage and above `make test` in the build stage,
|
and `make lint` in the lint stage and above `make test` in the build stage,
|
||||||
@@ -342,11 +337,11 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
upstream fetch or cached source read and one transcode through
|
upstream fetch or cached source read and one transcode through
|
||||||
`golang.org/x/sync/singleflight`; the first request's processing ignores its
|
`golang.org/x/sync/singleflight`; the first request's processing ignores its
|
||||||
cancellation but keeps its deadline, and the others wait for its image or
|
cancellation but keeps its deadline, and the others wait for its image or
|
||||||
error holding no upstream connection or processing slot, and stop waiting
|
error holding no upstream connection or processing slot, and stop waiting when
|
||||||
when their own context ends; the request doing the processing waits for it
|
their own context ends; the request doing the processing waits for it even
|
||||||
even then, up to its deadline; a request whose context has already ended
|
then, up to its deadline; a request whose context has already ended starts
|
||||||
starts nothing; each request counts one miss, and the processing counts its
|
nothing; each request counts one miss, and the processing counts its fetch and
|
||||||
fetch and transcode once; a panic while processing is reported to Sentry when
|
transcode once; a panic while processing is reported to Sentry when
|
||||||
`sentry_dsn` is set and becomes an error for every waiting request instead of
|
`sentry_dsn` is set and becomes an error for every waiting request instead of
|
||||||
stopping pixad; documented in `README.md`.
|
stopping pixad; documented in `README.md`.
|
||||||
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
|
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
|
||||||
@@ -355,20 +350,20 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
still answers a preflight `OPTIONS` request; the login and URL generator
|
still answers a preflight `OPTIONS` request; the login and URL generator
|
||||||
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
|
pages, `/metrics` and the other routes send no `Access-Control-Allow-Origin`;
|
||||||
documented in `README.md` and `config.example.yml`.
|
documented in `README.md` and `config.example.yml`.
|
||||||
- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not
|
- 2026-10-02 a plain `docker build .` stamps the tag or short commit, not `dev`
|
||||||
`dev` (closes #166): `.dockerignore` lets `.git` into the build context,
|
(closes #166): `.dockerignore` lets `.git` into the build context, without
|
||||||
without `.git/config`; with no `VERSION` build argument the `Dockerfile`
|
`.git/config`; with no `VERSION` build argument the `Dockerfile` takes the
|
||||||
takes the version from `git describe --tags --always`, and fails the build if
|
version from `git describe --tags --always`, and fails the build if the
|
||||||
the context carries `.git` and no version comes out; `ARG VERSION` has no
|
context carries `.git` and no version comes out; `ARG VERSION` has no default;
|
||||||
default; pixad logs its version, with its name and architecture, as its first
|
pixad logs its version, with its name and architecture, as its first log line
|
||||||
log line at startup.
|
at startup.
|
||||||
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes
|
- 2026-09-29 the container makes `/var/lib/pixa` usable by itself (closes #159):
|
||||||
#159): `deploy/docker-entrypoint.sh` creates the directory if it is missing,
|
`deploy/docker-entrypoint.sh` creates the directory if it is missing, gives
|
||||||
gives the directory and everything in it to `pixad` when the directory or one
|
the directory and everything in it to `pixad` when the directory or one of its
|
||||||
of its top-level entries belongs to another user or group, sets its mode to
|
top-level entries belongs to another user or group, sets its mode to `750`,
|
||||||
`750`, then runs the server as `pixad`; data left by an earlier run under
|
then runs the server as `pixad`; data left by an earlier run under another uid
|
||||||
another uid is taken over this way; "Running under upaas" in `README.md` no
|
is taken over this way; "Running under upaas" in `README.md` no longer tells
|
||||||
longer tells the operator to create or chown the host directory.
|
the operator to create or chown the host directory.
|
||||||
- 2026-09-29 variant content types kept in memory (closes #70):
|
- 2026-09-29 variant content types kept in memory (closes #70):
|
||||||
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
|
`Cache.metaCache` holds the content types of up to 10,000 variants in an LRU
|
||||||
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
|
(`github.com/hashicorp/golang-lru/v2`), filled by `StoreVariant` and by
|
||||||
@@ -407,8 +402,8 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
`ARG VERSION` sits just above the build, so a new version reruns neither
|
`ARG VERSION` sits just above the build, so a new version reruns neither
|
||||||
`script/bootstrap` nor the tests.
|
`script/bootstrap` nor the tests.
|
||||||
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
|
- 2026-09-29 migrations at the path `REPO_POLICIES.md` sets (closes #96): the
|
||||||
migration files moved, contents unchanged, from `internal/database/schema/`
|
migration files moved, contents unchanged, from `internal/database/schema/` to
|
||||||
to `internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
|
`internal/db/migrations/` as `000_migration.sql` and `001_schema.sql`; the
|
||||||
`internal/db/migrations` package embeds them and `internal/database` reads
|
`internal/db/migrations` package embeds them and `internal/database` reads
|
||||||
them through its `FS()`; the `internal/database` package itself stays; the
|
them through its `FS()`; the `internal/database` package itself stays; the
|
||||||
version still comes from the filename prefix, so a database that has recorded
|
version still comes from the filename prefix, so a database that has recorded
|
||||||
@@ -423,8 +418,8 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
`=` padding kept, and gives the example's `sig` for a stated signing key.
|
`=` padding kept, and gives the example's `sig` for a stated signing key.
|
||||||
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
|
- 2026-09-29 fixed uid and gid for `pixad` (closes #151): the image creates the
|
||||||
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
|
`pixad` group with gid 65532 and the `pixad` user with uid 65532, instead of
|
||||||
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad`
|
the first free uid 1000, so a bind-mounted `/var/lib/pixa` given to `pixad` is
|
||||||
is not owned on the host by a person's login account; the first-run step of
|
not owned on the host by a person's login account; the first-run step of
|
||||||
"Running under upaas" in `README.md` names the uid and gid.
|
"Running under upaas" in `README.md` names the uid and gid.
|
||||||
- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image
|
- 2026-09-29 `max-age` never outlives an expiring URL (closes #63): both image
|
||||||
routes build `Cache-Control` from the request's `Expires`, which an encrypted
|
routes build `Cache-Control` from the request's `Expires`, which an encrypted
|
||||||
@@ -433,29 +428,27 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
that is sooner, never negative; an allowlisted host's URL that has an `exp`
|
that is sooner, never negative; an allowlisted host's URL that has an `exp`
|
||||||
follows it too; `immutable` stays, as freshness now ends at the expiry;
|
follows it too; `immutable` stays, as freshness now ends at the expiry;
|
||||||
documented in `README.md`.
|
documented in `README.md`.
|
||||||
- 2026-09-28 add the four settings `README.md` documented but pixa did not
|
- 2026-09-28 add the four settings `README.md` documented but pixa did not have,
|
||||||
have, which aborted startup as unknown keys (closes #61):
|
which aborted startup as unknown keys (closes #61):
|
||||||
`access_control_allow_origin` (default `*`, the CORS origin),
|
`access_control_allow_origin` (default `*`, the CORS origin),
|
||||||
`upstream_fetch_timeout` (default `30s`), `upstream_max_response_size`
|
`upstream_fetch_timeout` (default `30s`), `upstream_max_response_size`
|
||||||
(default 50 MiB) and `downstream_timeout` (default `60s`, both the
|
(default 50 MiB) and `downstream_timeout` (default `60s`, both the server's
|
||||||
server's write timeout and the per-request timeout); each has a
|
write timeout and the per-request timeout); each has a `PIXA_` variable;
|
||||||
`PIXA_` variable; durations are positive Go duration strings, the size a
|
durations are positive Go duration strings, the size a whole number of bytes
|
||||||
whole number of bytes up to 1 GiB, the origin `*` or one `http` or
|
up to 1 GiB, the origin `*` or one `http` or `https` origin as `README.md`
|
||||||
`https` origin as `README.md` describes it; an invalid value
|
describes it; an invalid value aborts startup naming the key and the value;
|
||||||
aborts startup naming the key and the value; documented in
|
documented in `config.example.yml` and `README.md`.
|
||||||
`config.example.yml` and `README.md`.
|
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats` counts
|
||||||
- 2026-09-28 cache stats report real numbers (closes #56): `Cache.Stats`
|
the cached source images and processed variants (`source_content` plus
|
||||||
counts the cached source images and processed variants (`source_content`
|
`variant_content`) and takes their size from `Cache.UsageBytes`, instead of
|
||||||
plus `variant_content`) and takes their size from `Cache.UsageBytes`,
|
reading `request_cache` and `output_content`, which nothing writes; those two
|
||||||
instead of reading `request_cache` and `output_content`, which nothing
|
tables are left in the schema; a disabled disk cache reports no items and no
|
||||||
writes; those two tables are left in the schema; a disabled disk cache
|
size. A hit is counted even when the request context has ended. A miss is
|
||||||
reports no items and no size. A hit is counted even when the request
|
counted after it is served or fails, also when the request context has ended
|
||||||
context has ended. A miss is counted after it is served or fails, also
|
by then, with the bytes it read from upstream, so `upstream_fetch_count` and
|
||||||
when the request context has ended by then, with the bytes it read from
|
`upstream_fetch_bytes` move, including for an upstream body that fails partway
|
||||||
upstream, so `upstream_fetch_count` and `upstream_fetch_bytes` move,
|
or a fetched source that then fails the magic byte check; `transform_count`
|
||||||
including for an upstream body that fails partway or a fetched source
|
counts each image the image processor transcodes.
|
||||||
that then fails the magic byte check; `transform_count` counts each image
|
|
||||||
the image processor transcodes.
|
|
||||||
- 2026-09-28 strip metadata from processed images (closes #82): every output is
|
- 2026-09-28 strip metadata from processed images (closes #82): every output is
|
||||||
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
|
exported with govips' `StripMetadata`, so it carries no EXIF, XMP, IPTC or ICC
|
||||||
profile; the image is first turned upright with `AutoRotate` (before sizes are
|
profile; the image is first turned upright with `AutoRotate` (before sizes are
|
||||||
@@ -466,220 +459,201 @@ P2: security: per-IP rate limiting on the image routes
|
|||||||
attempts per minute per client address, and an attempt over the limit is
|
attempts per minute per client address, and an attempt over the limit is
|
||||||
refused with 429 and a `Retry-After` header; the address is the one
|
refused with 429 and a `Retry-After` header; the address is the one
|
||||||
`internal/clientip` resolves through `trusted_proxies`, an IPv6 client is
|
`internal/clientip` resolves through `trusted_proxies`, an IPv6 client is
|
||||||
counted by its /64, and an IPv4-mapped address as the IPv4 address it
|
counted by its /64, and an IPv4-mapped address as the IPv4 address it carries;
|
||||||
carries; the limit is a `RateLimit` middleware in `internal/middleware` on
|
the limit is a `RateLimit` middleware in `internal/middleware` on
|
||||||
`github.com/go-chi/httprate`, which the image routes can reuse; the library
|
`github.com/go-chi/httprate`, which the image routes can reuse; the library
|
||||||
keeps counts for the current and the previous minute only; documented in
|
keeps counts for the current and the previous minute only; documented in
|
||||||
`README.md`.
|
`README.md`.
|
||||||
- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed
|
- 2026-09-28 refuse an unparseable `exp` on `/v1/image/` and log swallowed cache
|
||||||
cache errors (closes #72): an `exp` in the URL that is not a whole
|
errors (closes #72): an `exp` in the URL that is not a whole number, an empty
|
||||||
number, an empty `exp=` included, is a 400 naming `exp` and the value,
|
`exp=` included, is a 400 naming `exp` and the value, instead of being ignored
|
||||||
instead of being ignored and answered with 401 as if the URL had no
|
and answered with 401 as if the URL had no `exp`; only an `exp` missing from
|
||||||
`exp`; only an `exp` missing from the URL is unchanged; `README.md` says
|
the URL is unchanged; `README.md` says so where it documents `exp`. A failed
|
||||||
so where it documents `exp`. A failed variant `.meta` write, source
|
variant `.meta` write, source metadata JSON write, `Stats` count query, stats
|
||||||
metadata JSON write, `Stats` count query, stats counter update, negative
|
counter update, negative cache write or expired negative cache delete is now
|
||||||
cache write or expired negative cache delete is now logged at `warn`
|
logged at `warn` with the path or key and the error, and stays non-fatal.
|
||||||
with the path or key and the error, and stays non-fatal.
|
- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a `fit` in the
|
||||||
- 2026-09-28 refuse an empty `fit` on `/v1/image/` (closes #139): a
|
URL with an empty value (`fit=`) is a 400 naming `fit`, instead of being
|
||||||
`fit` in the URL with an empty value (`fit=`) is a 400 naming `fit`,
|
served as `cover` and verified against a signature made for `cover`; only a
|
||||||
instead of being served as `cover` and verified against a signature
|
`fit` missing from the URL is still `cover`; any other value still goes
|
||||||
made for `cover`; only a `fit` missing from the URL is still `cover`;
|
through the existing fit-mode check; `README.md` says so where it documents
|
||||||
any other value still goes through the existing fit-mode check;
|
`fit`.
|
||||||
`README.md` says so where it documents `fit`.
|
- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q` that is
|
||||||
- 2026-09-28 refuse an invalid `q` on `/v1/image/` (closes #134): a `q`
|
not a whole number from 1 to 100, an empty `q` included, is a 400 naming `q`
|
||||||
that is not a whole number from 1 to 100, an empty `q` included, is a
|
and the value, instead of being served at the default 85; the route reads `q`
|
||||||
400 naming `q` and the value, instead of being served at the default
|
with the generator's quality check (`parseFormInt` with `minQuality` and
|
||||||
85; the route reads `q` with the generator's quality check
|
`maxQuality`); only a `q` missing from the URL is still 85; a query string
|
||||||
(`parseFormInt` with `minQuality` and `maxQuality`); only a `q` missing
|
that cannot be decoded, such as `q=80%`, is a 400 showing it; any query
|
||||||
from the URL is still 85; a query string that cannot be decoded, such
|
parameter given more than once (`q`, `fit`, `sig`, `exp` alike) is a 400
|
||||||
as `q=80%`, is a 400 showing it; any query parameter given more than
|
naming it, so none is read from its first value only; `README.md` states the
|
||||||
once (`q`, `fit`, `sig`, `exp` alike) is a 400 naming it, so none is
|
range and both query-string rules.
|
||||||
read from its first value only; `README.md` states the range and both
|
- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes #133):
|
||||||
query-string rules.
|
a variable whose name starts with `PIXA_` but is neither a setting's variable
|
||||||
- 2026-09-28 unknown `PIXA_` environment variables abort startup (closes
|
nor `PIXA_CONFIG_PATH` aborts startup naming it, as an unknown config key
|
||||||
#133): a variable whose name starts with `PIXA_` but is neither a
|
does, and `PIXA_PORT` is named with a pointer to `PORT`; the check runs after
|
||||||
setting's variable nor `PIXA_CONFIG_PATH` aborts startup naming it, as
|
the config file loads, so the variables the file's `env:` section sets are
|
||||||
an unknown config key does, and `PIXA_PORT` is named with a pointer to
|
checked too; documented in `README.md`.
|
||||||
`PORT`; the check runs after the config file loads, so the variables
|
- 2026-09-28 start on a fresh upaas volume (closes #129): the image starts as
|
||||||
the file's `env:` section sets are checked too; documented in
|
root only to give `/var/lib/pixa` to `pixad` when `pixad` does not own it
|
||||||
`README.md`.
|
(`deploy/docker-entrypoint.sh`), then runs the server as `pixad` through
|
||||||
- 2026-09-28 start on a fresh upaas volume (closes #129): the image
|
`su-exec`, so a root-owned host directory bind-mounted there no longer stops
|
||||||
starts as root only to give `/var/lib/pixa` to `pixad` when `pixad`
|
the container at startup; `README.md` gains a "Running under upaas" section.
|
||||||
does not own it (`deploy/docker-entrypoint.sh`), then runs the server
|
- 2026-09-28 run all linting in Docker via `Dockerfile.lint` + `script/lint`
|
||||||
as `pixad` through `su-exec`, so a root-owned host directory
|
(closes #104): `make lint` calls `script/lint`, the only way the linter is
|
||||||
bind-mounted there no longer stops the container at startup;
|
run; inside a container (both Dockerfiles set `container=docker`) it runs
|
||||||
`README.md` gains a "Running under upaas" section.
|
`golangci-lint`, anywhere else it builds the hash-pinned `Dockerfile.lint`,
|
||||||
- 2026-09-28 run all linting in Docker via `Dockerfile.lint` +
|
whose last step runs `script/lint` again; the `Dockerfile` lint stage runs
|
||||||
`script/lint` (closes #104): `make lint` calls `script/lint`, the only
|
`make lint`; no host or nix-shell `golangci-lint` path remains
|
||||||
way the linter is run; inside a container (both Dockerfiles set
|
(`script/bootstrap` installs no linter); a per-run `CACHEBUST` build-arg keeps
|
||||||
`container=docker`) it runs `golangci-lint`, anywhere else it builds the
|
the lint step from being served from cache, and a tmpfs mount on that step
|
||||||
hash-pinned `Dockerfile.lint`, whose last step runs `script/lint` again;
|
keeps Go's and golangci-lint's caches out of its layer, so a run leaves no
|
||||||
the `Dockerfile` lint stage runs `make lint`; no host or nix-shell
|
large build cache behind; `golangci-lint config verify` stays out, as it
|
||||||
`golangci-lint` path remains (`script/bootstrap` installs no linter);
|
fetches its schema over an unpinned live HTTPS call
|
||||||
a per-run `CACHEBUST` build-arg keeps the lint step from being served
|
- 2026-09-28 every setting as an environment variable (closes #128, also covers
|
||||||
from cache, and a tmpfs mount on that step keeps Go's and
|
#99): each config key can be set by `PIXA_` plus the key in upper case (`.`
|
||||||
golangci-lint's caches out of its layer, so a run leaves no large build
|
written as `_`), and the port by `PORT`; a variable present in the
|
||||||
cache behind; `golangci-lint config verify` stays out, as it fetches its
|
environment, even empty, wins over the config file, which wins over the
|
||||||
schema over an unpinned live HTTPS call
|
default; the typed getters read the variable first, so every existing check
|
||||||
- 2026-09-28 every setting as an environment variable (closes #128, also
|
applies to it and a bad value aborts startup naming the variable; lists are
|
||||||
covers #99): each config key can be set by `PIXA_` plus the key in upper
|
comma-separated, and an empty variable (or `""` in the file) is an empty list;
|
||||||
case (`.` written as `_`), and the port by `PORT`; a variable present in
|
the Docker image no longer bakes in `config.docker.yml` or passes `--config`,
|
||||||
the environment, even empty, wins over the config file, which wins over
|
and its `HEALTHCHECK` probes `PORT` (default `8080`); the config file is
|
||||||
the default; the typed getters read the variable first, so every existing
|
looked for under `/etc/pixa` and `~/.config/pixa` instead of the daemon name
|
||||||
check applies to it and a bad value aborts startup naming the variable;
|
`pixad`; documented in `README.md` and `config.example.yml`.
|
||||||
lists are comma-separated, and an empty variable (or `""` in the file) is
|
- 2026-09-28 quality and fit in the URL signature (closes #60): the signed data
|
||||||
an empty list; the Docker image no longer bakes in `config.docker.yml` or
|
is now `host:path:query:width:height:format:expiration:quality:fit`, using
|
||||||
passes `--config`, and its `HEALTHCHECK` probes `PORT` (default `8080`);
|
`85` and `cover` when the URL has no `q` or `fit`, so one signed URL can no
|
||||||
the config file is looked for under `/etc/pixa` and `~/.config/pixa`
|
longer be replayed across other quality and fit values to create unauthorized
|
||||||
instead of the daemon name `pixad`; documented in `README.md` and
|
cache entries and transcodes; the known-answer vectors in
|
||||||
`config.example.yml`.
|
`internal/signature/golden_test.go` and the README signature specification
|
||||||
- 2026-09-28 quality and fit in the URL signature (closes #60): the signed
|
describe the new format.
|
||||||
data is now `host:path:query:width:height:format:expiration:quality:fit`,
|
- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in the
|
||||||
using `85` and `cover` when the URL has no `q` or `fit`, so one signed
|
runtime stage probing `/.well-known/healthcheck.json` with busybox `wget`;
|
||||||
URL can no longer be replayed across other quality and fit values to
|
`script/docker-smoke` (`make docker-smoke`) builds the image, starts it with a
|
||||||
create unauthorized cache entries and transcodes; the known-answer
|
throwaway `PIXA_SIGNING_KEY`, and passes only once Docker reports it healthy
|
||||||
vectors in `internal/signature/golden_test.go` and the README signature
|
within 30 seconds, removing the container on exit; the Gitea workflow runs it
|
||||||
specification describe the new format.
|
after `script/cibuild`.
|
||||||
- 2026-09-28 Docker image healthcheck (closes #111): a `HEALTHCHECK` in
|
|
||||||
the runtime stage probing `/.well-known/healthcheck.json` with busybox
|
|
||||||
`wget`; `script/docker-smoke` (`make docker-smoke`) builds the image,
|
|
||||||
starts it with a throwaway `PIXA_SIGNING_KEY`, and passes only once
|
|
||||||
Docker reports it healthy within 30 seconds, removing the container on
|
|
||||||
exit; the Gitea workflow runs it after `script/cibuild`.
|
|
||||||
- 2026-09-21 trusted-proxy client IP resolution (closes #94): a
|
- 2026-09-21 trusted-proxy client IP resolution (closes #94): a
|
||||||
`trusted_proxies` config key taking a list of CIDRs, parsed by the same
|
`trusted_proxies` config key taking a list of CIDRs, parsed by the same
|
||||||
`net/netip` list parser as `blocked_networks` (an invalid entry aborts
|
`net/netip` list parser as `blocked_networks` (an invalid entry aborts startup
|
||||||
startup naming the key and value; an omitted key defaults to the RFC 1918
|
naming the key and value; an omitted key defaults to the RFC 1918 private
|
||||||
private ranges, an explicitly empty list trusts no one, and an explicit
|
ranges, an explicitly empty list trusts no one, and an explicit list replaces
|
||||||
list replaces the default); a new
|
the default); a new `internal/clientip` package resolves the client address by
|
||||||
`internal/clientip` package resolves the client address by honoring
|
honoring `X-Forwarded-For` only when the direct peer is a trusted proxy,
|
||||||
`X-Forwarded-For` only when the direct peer is a trusted proxy, walking
|
walking the chain right-to-left to the rightmost non-proxy entry, so a client
|
||||||
the chain right-to-left to the rightmost non-proxy entry, so a client
|
connecting directly cannot spoof its address; the resolved address is stored
|
||||||
connecting directly cannot spoof its address; the resolved address is
|
in the request context by a new middleware and used by the request-logging
|
||||||
stored in the request context by a new middleware and used by the
|
middleware and the login-attempt logs in place of the raw peer address;
|
||||||
request-logging middleware and the login-attempt logs in place of the
|
documented in `README.md` and `config.example.yml`.
|
||||||
raw peer address; documented in `README.md` and `config.example.yml`.
|
|
||||||
- 2026-09-21 blocked networks configuration extending SSRF protection: a
|
- 2026-09-21 blocked networks configuration extending SSRF protection: a
|
||||||
`blocked_networks` config key taking a list of CIDRs (parsed with
|
`blocked_networks` config key taking a list of CIDRs (parsed with `net/netip`,
|
||||||
`net/netip`, an invalid entry aborts startup naming the key and value),
|
an invalid entry aborts startup naming the key and value), added to the
|
||||||
added to the built-in blocklist rather than replacing it; the built-in
|
built-in blocklist rather than replacing it; the built-in ranges extended to
|
||||||
ranges extended to CGNAT `100.64.0.0/10`, IETF protocol assignments
|
CGNAT `100.64.0.0/10`, IETF protocol assignments `192.0.0.0/24`, benchmark
|
||||||
`192.0.0.0/24`, benchmark `198.18.0.0/15`, and NAT64 `64:ff9b::/96`
|
`198.18.0.0/15`, and NAT64 `64:ff9b::/96` (IPv4-mapped forms covered);
|
||||||
(IPv4-mapped forms covered); enforcement stays in the dial-time
|
enforcement stays in the dial-time re-resolution so the DNS-rebinding window
|
||||||
re-resolution so the DNS-rebinding window remains closed; documented in
|
remains closed; documented in `README.md` and `config.example.yml`.
|
||||||
`README.md` and `config.example.yml`.
|
- 2026-09-21 validate dimensions and fit mode on the encrypted-URL route and the
|
||||||
- 2026-09-21 validate dimensions and fit mode on the encrypted-URL
|
token generator (closes #62): `imgcache.ValidateDimension` alone holds the
|
||||||
route and the token generator (closes #62): `imgcache.ValidateDimension`
|
`MaxDimension` bound and is used by the path parser, by the new
|
||||||
alone holds the `MaxDimension` bound and is used by the path parser, by
|
`ValidateImageRequest` (which also applies `ValidateFitMode`) and by the
|
||||||
the new `ValidateImageRequest` (which also applies `ValidateFitMode`)
|
generator; both the `/v1/image/` and `/v1/e/` routes call
|
||||||
and by the generator; both the `/v1/image/` and `/v1/e/` routes call
|
`ValidateImageRequest`, so an over-limit size or an unknown fit mode is a 400
|
||||||
`ValidateImageRequest`, so an over-limit size or an unknown fit mode is a
|
rather than an out-of-memory or a 500 from the processor; the URL generator
|
||||||
400 rather than an out-of-memory or a 500 from the processor; the URL
|
answers 400 naming the field for a `width` or `height` that is not a number or
|
||||||
generator answers 400 naming the field for a `width` or `height` that is
|
fails the shared check, a `quality` that is not a number from 1 to 100, a
|
||||||
not a number or fails the shared check, a `quality` that is not a number
|
`ttl` that is not a number from 0 to the largest number of seconds the expiry
|
||||||
from 1 to 100, a `ttl` that is not a number from 0 to the largest number
|
calculation can hold, or an unknown `fit`; an empty `quality` is 85 and an
|
||||||
of seconds the expiry calculation can hold, or an unknown `fit`; an empty
|
empty `ttl` never expires; the form's width and height inputs stop at 8192
|
||||||
`quality` is 85 and
|
- 2026-09-21 http.Server hardening (closes #92): added `HTTPReadHeaderTimeout`
|
||||||
an empty `ttl` never expires; the form's width and height inputs stop at
|
(10s, bounds the slowloris header dribble) and `HTTPIdleTimeout` (120s, bounds
|
||||||
8192
|
keep-alive reuse) alongside the existing timeouts and wired them onto the
|
||||||
- 2026-09-21 http.Server hardening (closes #92): added
|
server; added a `LimitBody` middleware capping the two form POST bodies
|
||||||
`HTTPReadHeaderTimeout` (10s, bounds the slowloris header dribble) and
|
(`POST /`, `POST /generate`) at `MaxFormBytes` (1 MiB) and returning 413,
|
||||||
`HTTPIdleTimeout` (120s, bounds keep-alive reuse) alongside the
|
applied ahead of the CSRF middleware so an oversized body is refused as 413
|
||||||
existing timeouts and wired them onto the server; added a `LimitBody`
|
rather than being read as a missing CSRF token (403); left `WriteTimeout` at
|
||||||
middleware capping the two form POST bodies (`POST /`, `POST /generate`)
|
60s unchanged
|
||||||
at `MaxFormBytes` (1 MiB) and returning 413, applied ahead of the CSRF
|
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical `.golangci.yml`
|
||||||
middleware so an oversized body is refused as 413 rather than being read
|
(v2 schema, `default: all` minus six disabled linters, `lll` 88, tests
|
||||||
as a missing CSRF token (403); left `WriteTimeout` at 60s unchanged
|
included): bumped the pinned `golangci/golangci-lint:v2.12.2-alpine` image in
|
||||||
- 2026-08-07 update golangci-lint to v2.12.2 with the canonical
|
`Dockerfile` and the release-archive sha256 pins in `script/bootstrap`; fixed
|
||||||
`.golangci.yml` (v2 schema, `default: all` minus six disabled
|
the findings the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
|
||||||
linters, `lll` 88, tests included): bumped the pinned
|
|
||||||
`golangci/golangci-lint:v2.12.2-alpine` image in `Dockerfile` and the
|
|
||||||
release-archive sha256 pins in `script/bootstrap`; fixed the findings
|
|
||||||
the stricter config surfaced (notably `paralleltest`, `wsl_v5`,
|
|
||||||
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
|
`goconst`, `lll`, `noinlineerr`, `err113`, `errcheck`, `testpackage` —
|
||||||
white-box test files renamed to `*_internal_test.go`), including #55's
|
white-box test files renamed to `*_internal_test.go`), including #55's code
|
||||||
code absorbed after it merged, iterating the pinned linter to
|
absorbed after it merged, iterating the pinned linter to `0 issues.`; no
|
||||||
`0 issues.`; no single finding total is substantiable, since
|
single finding total is substantiable, since golangci-lint's `uniq-by-line`
|
||||||
golangci-lint's `uniq-by-line` reveals new findings on a line as
|
reveals new findings on a line as others there are fixed — the documented
|
||||||
others there are fixed — the documented re-measurements were 81 after
|
re-measurements were 81 after the #53 merge and 149 after the #55 merge; three
|
||||||
the #53 merge and 149 after the #55 merge; three behavior changes, so
|
behavior changes, so not a pure no-op: `Cache.StoreVariant` now takes a
|
||||||
not a pure no-op: `Cache.StoreVariant` now takes a `context.Context`
|
`context.Context` (`noctx`), so a cancelled request skips its best-effort
|
||||||
(`noctx`), so a cancelled request skips its best-effort accounting
|
accounting row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
|
||||||
row; `MetadataStorage.Store`'s cleanup defer was dead on `main` and
|
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and the
|
||||||
leaked `.tmp-*.json` on failure, now fixed with explicit removals; and
|
`signing_key` validation error text gained `value too short: `; the eviction
|
||||||
the `signing_key` validation error text gained `value too short: `;
|
loop's uncancellable context is deferred to #102 under a
|
||||||
the eviction loop's uncancellable context is deferred to #102 under a
|
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep the
|
||||||
`//nolint:contextcheck`; three `//nolint:tagliatelle` directives keep
|
snake_case JSON wire/disk formats unchanged; `make check` green
|
||||||
the snake_case JSON wire/disk formats unchanged; `make check` green
|
- 2026-08-07 implement cache size management and eviction (closes #51): new
|
||||||
- 2026-08-07 implement cache size management and eviction (closes
|
`cache_max_bytes` config key validated by the startup framework (explicit
|
||||||
#51): new `cache_max_bytes` config key validated by the startup
|
values used exactly with no floor, `0` disables the disk cache entirely,
|
||||||
framework (explicit values used exactly with no floor, `0` disables
|
omitted defaults to max(75% of free space on the filesystem containing
|
||||||
the disk cache entirely, omitted defaults to max(75% of free space
|
`<state_dir>/cache/`, 500 MiB), logged at startup); processed variants are now
|
||||||
on the filesystem containing `<state_dir>/cache/`, 500 MiB), logged
|
tracked in the database (a new `variant_content` table and an LRU timestamp on
|
||||||
at startup); processed variants are now tracked in the database (a
|
`source_content`) so total usage is two SUMs, never a directory scan on the
|
||||||
new `variant_content` table and an LRU timestamp on `source_content`)
|
hot path; a background goroutine evicts globally least-recently-used entries
|
||||||
so total usage is two SUMs, never a directory scan on the hot path; a
|
(variants and source blobs merged) to the limit, woken by a periodic ticker
|
||||||
background goroutine evicts globally least-recently-used entries
|
and by write-pressure notifications from stores; a source blob and ALL of its
|
||||||
(variants and source blobs merged) to the limit, woken by a periodic
|
`source_metadata` references are deleted in one transaction before the file is
|
||||||
ticker and by write-pressure notifications from stores; a source
|
unlinked, so multi-referenced blobs are never removed while referenced and
|
||||||
blob and ALL of its `source_metadata` references are deleted in one
|
rows never point at deleted files; a startup and periodic reconciliation pass
|
||||||
transaction before the file is unlinked, so multi-referenced blobs
|
adopts untracked variant files, drops rows for missing files, removes
|
||||||
are never removed while referenced and rows never point at deleted
|
unreachable source blobs, and sweeps stale temp files
|
||||||
files; a startup and periodic reconciliation pass adopts untracked
|
- 2026-08-07 validate configuration on startup, fail fast on bad config (closes
|
||||||
variant files, drops rows for missing files, removes unreachable
|
#52): a config value that is set but unparseable or invalid aborts startup
|
||||||
source blobs, and sweeps stale temp files
|
naming the key and value (defaults apply only to omitted keys), unknown config
|
||||||
- 2026-08-07 validate configuration on startup, fail fast on bad
|
keys abort startup, a malformed config file aborts instead of being skipped,
|
||||||
config (closes #52): a config value that is set but unparseable or
|
and `state_dir` is verified creatable and writable before the listener binds
|
||||||
invalid aborts startup naming the key and value (defaults apply only
|
- 2026-08-07 manual test pass of the auth and encrypted URL flows against a
|
||||||
to omitted keys), unknown config keys abort startup, a malformed
|
locally built and running `pixad` (built from `main` at `6573b9d`, port 18099,
|
||||||
config file aborts instead of being skipped, and `state_dir` is
|
local throwaway config); all six checks passed, plus all nine tests in
|
||||||
verified creatable and writable before the listener binds
|
`scripts/manual-test.sh` (closes #49):
|
||||||
- 2026-08-07 manual test pass of the auth and encrypted URL flows
|
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login` page with
|
||||||
against a locally built and running `pixad` (built from `main` at
|
`name="key"` password form
|
||||||
`6573b9d`, port 18099, local throwaway config); all six checks
|
- [x] wrong key shows an error: POST `/` with `key=wrong-key` returned HTTP
|
||||||
passed, plus all nine tests in `scripts/manual-test.sh` (closes #49):
|
200 login page containing "Invalid signing key"
|
||||||
- [x] visit `/` and see the login form: HTTP 200, `Pixa - Login`
|
- [x] correct signing key shows the generator form: POST `/` returned HTTP
|
||||||
page with `name="key"` password form
|
303 to `/` with
|
||||||
- [x] wrong key shows an error: POST `/` with `key=wrong-key`
|
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`; GET
|
||||||
returned HTTP 200 login page containing "Invalid signing key"
|
`/` with that cookie rendered `Pixa - URL Generator` with the
|
||||||
- [x] correct signing key shows the generator form: POST `/`
|
`/generate` form and logout link
|
||||||
returned HTTP 303 to `/` with
|
|
||||||
`Set-Cookie: pixa_session=...; HttpOnly; Secure; SameSite=Strict`;
|
|
||||||
GET `/` with that cookie rendered `Pixa - URL Generator` with the
|
|
||||||
`/generate` form and logout link
|
|
||||||
- [x] a generated encrypted URL serves the image: POST `/generate`
|
- [x] a generated encrypted URL serves the image: POST `/generate`
|
||||||
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned
|
(ttl=3600) produced a `/v1/e/<token>/img.jpeg` URL that returned HTTP
|
||||||
HTTP 200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of
|
200, `Content-Type: image/jpeg`, an 800x600 baseline JPEG of 61706
|
||||||
61706 bytes
|
bytes
|
||||||
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched
|
- [x] an expired URL (short TTL) returns 410: a ttl=1 URL fetched after 3 s
|
||||||
after 3 s returned HTTP 410 Gone with
|
returned HTTP 410 Gone with
|
||||||
`{"error":"URL has expired","status":410,...}`
|
`{"error":"URL has expired","status":410,...}`
|
||||||
- [x] logout redirects back to login: GET `/logout` returned HTTP
|
- [x] logout redirects back to login: GET `/logout` returned HTTP 303 to `/`
|
||||||
303 to `/` with `Set-Cookie: pixa_session=; Max-Age=0`;
|
with `Set-Cookie: pixa_session=; Max-Age=0`; subsequent GET `/`
|
||||||
subsequent GET `/` rendered the login form again
|
rendered the login form again
|
||||||
- 2026-08-07 fix the two remaining gosec findings (G124 in
|
- 2026-08-07 fix the two remaining gosec findings (G124 in internal/session):
|
||||||
internal/session): session cookies now always carry
|
session cookies now always carry Secure/HttpOnly/SameSite=Strict on both the
|
||||||
Secure/HttpOnly/SameSite=Strict on both the set and clear paths;
|
set and clear paths; `make check` green (closes #47)
|
||||||
`make check` green (closes #47)
|
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints,
|
shims, README Entrypoints section
|
||||||
Makefile shims, README Entrypoints section
|
|
||||||
- 2026-04-07 extract magic byte detection into internal/magic (#42)
|
- 2026-04-07 extract magic byte detection into internal/magic (#42)
|
||||||
- 2026-03-25 extract allowlist package from internal/imgcache (#41)
|
- 2026-03-25 extract allowlist package from internal/imgcache (#41)
|
||||||
- 2026-03-25 move schema_migrations table creation into 000.sql (#36)
|
- 2026-03-25 move schema_migrations table creation into 000.sql (#36)
|
||||||
- 2026-03-20 enforce and document exact-match-only signature
|
- 2026-03-20 enforce and document exact-match-only signature verification (#40)
|
||||||
verification (#40)
|
- 2026-03-20 bound imageprocessor.Process input read to prevent unbounded memory
|
||||||
- 2026-03-20 bound imageprocessor.Process input read to prevent
|
use (#37); consolidate appname into an internal/globals constant (#34)
|
||||||
unbounded memory use (#37); consolidate appname into an
|
|
||||||
internal/globals constant (#34)
|
|
||||||
- 2026-03-18 parse version prefix from migration filenames (#33)
|
- 2026-03-18 parse version prefix from migration filenames (#33)
|
||||||
- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
|
- 2026-03-15 QA audit fixes for 1.0/MVP readiness (#25)
|
||||||
- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for
|
- 2026-03-02 split Dockerfile with pre-built golangci-lint stage for faster CI
|
||||||
faster CI (#23)
|
(#23)
|
||||||
- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images,
|
- 2026-02-25 repo policy compliance: CI workflow, hash-pinned images,
|
||||||
golangci-lint and gosec fixes of that date (#14); arm64 Docker build
|
golangci-lint and gosec fixes of that date (#14); arm64 Docker build fix (#16)
|
||||||
fix (#16)
|
- 2026-01-08 WebP and AVIF encoding support via govips (both former P0 image
|
||||||
- 2026-01-08 WebP and AVIF encoding support via govips (both former P0
|
processing items, now done)
|
||||||
image processing items, now done)
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user