10 Commits
Author SHA1 Message Date
clawbot 957ec85f97 Re-vendor the canonical files from sneak/prompts at 0653579 (closes #221)
check / check (push) Waiting to run
REPO_POLICIES.md, script/cibuild, script/docker, script/lint and
script/test are their copies at that commit. In every Dockerfile stage
the system package install and the Go module download now come before
ARG CACHEBUST, so check builds keep those layers until the base image,
the install line, go.mod or go.sum change; every step that lints, tests
or compiles still reruns. golangci-lint run also mounts golangci-lint's
cache. The stages install their packages with apt-get and apk directly
instead of through script/bootstrap --cgo. The test phase installs no
recommended packages and names libheif's AVIF encoder,
libheif-plugin-aomenc, which the tests use.

Model: opus-5-5
2026-10-19 06:11:43 +02:00
clawbot 930e43961e Read a trusted_proxies entry in the IPv4-mapped form as its IPv4 range (closes #303)
check / check (push) Canceled after 0s
An entry such as ::ffff:10.0.0.0/104 in trusted_proxies was accepted
but trusted nothing, because the client address is matched as the IPv4
address an IPv4-mapped address holds. The conversion that blocked_networks
gained in #301 now lives in
parseCIDRList, which reads both settings: such an entry is read as the
IPv4 range it covers, and one shorter than /96 stops startup with an
error naming the setting being read and quoting the entry as written.
README.md says so for trusted_proxies.

Model: opus-5-5
2026-10-19 05:29:05 +02:00
clawbot 9eacfa6406 Space reconciliation passes by ten times the last pass's time (closes #250)
check / check (push) Canceled after 0s
The reconciliation pass reads a row and stats a file for every cached
file, and it ran on every eviction tick, so on a large cache a pass
longer than the tick was followed by the next one at once. It still runs
at startup and is considered on each tick, but it now starts no sooner
than ten times as long after the last pass ended as that pass took, so
it takes at most about a tenth of the time however large the cache.
Eviction and the removal of expired negative cache rows still run on
every tick. The comments that bound the drift, and README.md, give the
new bound. No existing test changed.

Model: opus-5-5
2026-10-19 05:00:48 +02:00
clawbot 0ab15324f7 Read a blocked_networks entry in the IPv4-mapped form as its IPv4 range (closes #301)
check / check (push) Canceled after 0s
The fetcher matches an IPv4-mapped address (::ffff:a.b.c.d) as the
IPv4 address it holds, so a blocked_networks entry written in that form
was accepted and matched nothing. Such an entry is now kept as the IPv4
range it covers: ::ffff:10.0.0.0/104 becomes 10.0.0.0/8, and
::ffff:0:0/96 becomes 0.0.0.0/0. One shorter than /96 covers more than
IPv4 addresses and aborts startup naming the setting. README.md says
both. trusted_proxies shares the list parser, has the same fault and is
unchanged here (issue 303).

Model: opus-5-5
2026-10-19 03:00:43 +02:00
clawbot faad959bea Refuse local-use NAT64 addresses, and IPv4-compatible ones in blocked_networks (closes #299)
check / check (push) Canceled after 0s
The check that keeps pixa from fetching internal addresses let through
64:ff9b:1::/48, the local-use NAT64 range, and matched an address in the
IPv4-compatible form (::/96) against blocked_networks only as IPv6, so
::203.0.113.5 passed with 203.0.113.0/24 listed. The local-use NAT64
range is now blocked next to 64:ff9b::/96, and an address in ::/96 is
refused when blocked_networks holds it as written or the IPv4 address it
holds. The mapped form stays matched as IPv4 only, as Go keeps plain
IPv4 addresses in that form; README.md says so.

Model: opus-5-5
2026-10-16 06:52:17 +02:00
clawbot 3a6b3bc202 Refuse an IPv6 address that holds an internal IPv4 address (closes #297)
check / check (push) Canceled after 0s
The check that keeps pixa from fetching internal addresses read an
IPv6 address as IPv4 only in the mapped form (::ffff:a.b.c.d), so the
IPv4-compatible form, such as ::127.0.0.1, was judged as IPv6 and
passed. An address in ::/96 is now judged by the IPv4 address it holds,
so one holding a private, loopback or link-local address is refused on
both image routes before any upstream request. The NAT64 range
64:ff9b::/96 stays blocked whole, as it was on next.

Model: opus-5-5
2026-10-16 05:31:55 +02:00
clawbot 08d6c3c459 Fetch an IPv6 source host written without a port (closes #295)
check / check (push) Canceled after 0s
The fetcher took the host from the source URL and stripped a port with
net.SplitHostPort, which fails when there is no port, so a bracketed
IPv6 host such as [2001:db8::a] kept its brackets, its name lookup
failed, and the request answered 500. validateURL now takes the host
with url.URL.Hostname(), which drops the brackets and any port, so such
a host is fetched like one with a port, through /v1/image/ and through
an encrypted URL, and a private IPv6 address is still refused.

Model: opus-5-5
2026-10-16 05:00:47 +02:00
clawbot efa431ba18 Escape state_dir in the db_url derived from it (closes #290)
check / check (push) Canceled after 0s
When db_url is omitted, pixa derives it from state_dir, and it put the
directory name into a file: URL as it was, so a state_dir holding %, ?
or # opened another file and lost the journal mode. The derived URL is
now built with net/url, which percent-encodes the directory name; an
ordinary path gives the same URL as before. When the path written into
the URL starts with //, as for a state_dir of / or one starting with //,
the empty host is kept so SQLite does not read the first name in the
path as a host.

Model: opus-5-5
2026-10-16 04:37:54 +02:00
clawbot 8b4dcda549 Keep a side computed for a size with one side 0 at 1 or more (closes #283)
check / check (push) Canceled after 0s
A size with one side 0 has that side worked out from the source's
proportions by integer division, and nothing kept it at 1 or more. When
it rounded down to 0, libvips was asked for a square: an 8192x1 source
at 100x0 came out 100x100, at 1300x0 it asked for one block of about
39 GB, and some shapes answered 500. Each computed side, and each side
contain, inside and outside work out again from the scale, is now at
least 1. README.md now says what fit=outside does and what a 0 side
means; with fit=outside an 8192x1 source at 100x0 comes out 8192x1, as
covering the size requires.

Model: opus-5-5
2026-10-16 04:17:59 +02:00
clawbot 6d271d4617 Make one image URL per source on the generator page for host case and an empty path (closes #293)
check / check (push) Canceled after 0s
The generator page passed the source host and path into the encrypted
URL as entered, so a host spelled in another letter case, or a source
with no path against the same source with /, made separate cache keys
and fetched one upstream image twice. The page now lower-cases the
source host, as /v1/image/ does, and writes an empty source path as /,
before it builds the encrypted URL. The checks the page gained for
#288 still hold.

Model: opus-5-5
2026-10-16 04:04:19 +02:00
24 changed files with 1207 additions and 170 deletions
+31 -22
View File
@@ -1,12 +1,16 @@
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# In every stage ARG CACHEBUST comes right after the steps that install
# system packages and download Go's modules, or right after FROM in a
# stage with neither, and every step that lints, tests or compiles
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase. script/lint builds it alone. The linter is run directly:
# `make lint` and script/lint are themselves a docker build.
# golangci/golangci-lint:v2.14.0, 2026-10-15
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
ARG CACHEBUST
# The linter compiles every package, and govips needs the libvips
# headers for that. REPO_POLICIES.md has the lint phase install them
@@ -18,25 +22,29 @@ RUN apt-get update \
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./...
# Test phase. script/test builds it alone. -race needs cgo and so a C
# compiler, which the Debian Go image ships and the alpine one does not.
# golang:1.25.4-trixie, 2026-10-15
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
ARG CACHEBUST
# govips compiles against the libvips and libheif headers, and the tests
# need libvips' JPEG XL support; libvips-dev brings all three. The tests
# also encode AVIF, which needs libheif's AVIF encoder,
# libheif-plugin-aomenc: Debian only recommends it, so it is named here.
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev libheif-plugin-aomenc \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
# script/bootstrap --cgo installs the build dependencies (the libvips
# and libheif headers, and libvips' JPEG XL support, which the tests
# need) and downloads the Go modules.
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
RUN go mod download
ARG CACHEBUST
COPY . .
# Without -v first; on a failure, again with -v for the details, and
@@ -52,22 +60,23 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
# Build stage. Nothing is wanted from the two phases above: these copies
# make BuildKit build them first, so this stage runs only when lint and
# test passed.
# make BuildKit build them first, so nothing after them runs unless lint
# and test passed.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
# git for the version, and for govips a C compiler, pkg-config and the
# libvips and libheif headers
RUN apk add --no-cache git build-base pkgconfig vips-dev libheif-dev
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
WORKDIR /src
# Build dependencies, git among them, and Go modules, as in the test
# phase
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
@@ -102,7 +111,6 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
# the libvips and musl it was built with.
# alpine:3.22, 2026-10-08
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
ARG CACHEBUST
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
# support, without which pixad does not start.
@@ -113,6 +121,7 @@ RUN apk add --no-cache \
ca-certificates \
tzdata \
su-exec
ARG CACHEBUST
# Copy binary from builder
COPY --from=builder /pixad /usr/local/bin/pixad
+72 -38
View File
@@ -102,8 +102,8 @@ it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
support, which on Alpine is the `vips-jxl` package and which the nix and brew
packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04
on. Building pixa also needs the development files of libvips and libheif,
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these,
as the `Dockerfile` does where it compiles pixa. Plain `script/bootstrap`, which
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these;
the `Dockerfile` installs its own packages. Plain `script/bootstrap`, which
`script/setup` and `script/cibuild` run, installs git, make and Go, and Node,
Yarn and the prettier pinned in `yarn.lock` for formatting the markdown, but
none of the C libraries: the checks compile pixa in Docker. Docker itself must
@@ -199,7 +199,11 @@ into place, so a crash leaves no file under its own name with part of its data.
When pixa starts, and every five minutes after, it removes each cached source or
transformed image whose file is empty, or not the size the database records for
it, with a warning naming the file; the next request for it fetches the source
or converts the image again.
or converts the image again. Looking for those files reads the database once for
every cached file, so pixa skips each five-minute look that would start less
than ten times as long after the last look ended as that look took. After a look
that took one minute, the next one starts 10 to 15 minutes after that look
ended.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, pixa keeps in memory the content types of the 10,000
@@ -473,9 +477,11 @@ nor change what it asks for.
`jxl` for a form with no format.
The token holds the source's host, path and query and the size, format, quality,
fit and expiry, encrypted with a key derived from `signing_key`. The source
URL's scheme is not kept: the image is fetched like any other (see Routes), and
the blocked networks still apply.
fit and expiry, encrypted with a key derived from `signing_key`. The host is
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
fetched and cached once. The source URL's scheme is not kept: the image is
fetched like any other (see Routes), and the blocked networks still apply.
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
The expiry is fixed in the token when the URL is made and cannot be changed or
@@ -534,8 +540,13 @@ Where:
- `query` — source query string as under Routes, empty string if none, with `%`
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
query and path `/a` with query `b:` are signed differently
- `width` — requested width in pixels, `0` for original
- `height` — requested height in pixels, `0` for original
- `width` — requested width in pixels, or `0`: the width is then worked out from
the source's proportions and the requested height, rounded down but at least
1, or is the source's own width when the height is `0` too (the `orig` size)
- `height` — requested height in pixels, or `0`: the height is then worked out
from the source's proportions and the requested width, rounded down but at
least 1, or is the source's own height when the width is `0` too (the `orig`
size)
- `format` — output format, one of those listed under Routes, with `original`
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
@@ -548,7 +559,10 @@ Where:
with 400
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
anything else, an empty `fit=` included, is refused with 400
anything else, an empty `fit=` included, is refused with 400. `outside` scales
the image to cover the requested size, keeping its proportions, so it can come
out larger than that size on one side: an 8192x1 source at `100x0`, which is
100x1 by its proportions, comes out at 8192x1
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
@@ -684,7 +698,12 @@ Key settings in more detail:
and generator pages. Default: empty
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
benchmark, NAT64, and the like); an invalid CIDR aborts startup
benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
plain IPv4 address. An entry written in that form is read as the IPv4 range it
covers, so `::ffff:10.0.0.0/104` refuses what `10.0.0.0/8` refuses; such an
entry shorter than `/96` covers more than IPv4 addresses and aborts startup
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
of these ranges; the logged and login-recorded client address is then the
@@ -694,16 +713,22 @@ Key settings in more detail:
omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a
private network; an explicitly empty list (`[]`) trusts no one, and an
explicit list replaces the default. An invalid CIDR aborts startup. Set this
to the address pixa sees for requests that come through your proxy, such as
`172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else
(see the login limit and `image_requests_per_minute` under Routes). For a
proxy on the Docker host that connects to pixa over `127.0.0.1`, that address
is the gateway of the container's Docker network (`172.17.0.1` on the default
bridge), not the proxy's own address; a proxy that connects through another of
the host's addresses is seen with that address. To be sure which address it
is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
through the proxy, and read `remoteIP` in pixa's request log line for it
explicit list replaces the default. An invalid CIDR aborts startup. An IPv6
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds, so an IPv6 range such as `::/0` trusts neither it nor a
plain IPv4 address. An entry written in that form is read as the IPv4 range it
covers, so `::ffff:10.0.0.0/104` trusts what `10.0.0.0/8` trusts; such an
entry shorter than `/96` covers more than IPv4 addresses and aborts startup.
Set this to the address pixa sees for requests that come through your proxy,
such as `172.17.0.1/32`, when the defaults do not cover it, or to trust
nothing else (see the login limit and `image_requests_per_minute` under
Routes). For a proxy on the Docker host that connects to pixa over
`127.0.0.1`, that address is the gateway of the container's Docker network
(`172.17.0.1` on the default bridge), not the proxy's own address; a proxy
that connects through another of the host's addresses is seen with that
address. To be sure which address it is, set this to `[]` (or
`PIXA_TRUSTED_PROXIES` to empty), send a request through the proxy, and read
`remoteIP` in pixa's request log line for it
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
as a duration such as `30s` (the default) or `2m`
- `upstream_max_response_size` — largest upstream response accepted, in bytes;
@@ -715,11 +740,14 @@ Key settings in more detail:
- `signing_key` — HMAC secret for URL signatures
- `db_url` — the SQLite database to open; omitted, it is
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
database in WAL mode. pixa opens one connection to it, so its own reads and
writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
`db_url`, so a write that finds another program writing to the file waits up
to five seconds for it instead of failing. WAL mode comes only from the URL:
keep `_pragma=journal_mode(WAL)` in one you set
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
`#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/`
or one starting with `//` it starts `file:////`, so SQLite does not read the
first name in the path as a host. pixa opens one connection to it, so its own
reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to
any `db_url`, so a write that finds another program writing to the file waits
up to five seconds for it instead of failing. WAL mode comes only from the
URL: keep `_pragma=journal_mode(WAL)` in one you set
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
cache entirely; omitted defaults to 75% of the sum of the free space on the
filesystem containing `<state_dir>/cache/` and the bytes of source and
@@ -809,19 +837,25 @@ provide:
`script/precommit`
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
build argument, which every stage of the `Dockerfile` declares first, so every
`RUN` step, the lint and test phases included, runs on every build instead of
coming from the build cache. Go's build cache is kept from build to build: the
steps that compile mount it, so a package whose inputs are unchanged is not
compiled again, and both `go test` runs pass `-count=1` so that no stored test
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
libvips and libheif; the image pixa ships is built and runs on Alpine.
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
`script/setup` and `script/install-precommit` are the standard copies from
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
the standard copies with pixa's `gofmt` step kept before prettier. prettier
formats the markdown only, not the HTML templates, as it cannot parse a Go
template action inside a tag.
build argument, which every stage of the `Dockerfile` declares, so every `RUN`
step after that line, the lint and test steps included, runs on every build
instead of coming from the build cache. Above the line are the steps that
install system packages and download the Go modules; Docker keeps their layers
until the base image, the step itself, `go.mod` or `go.sum` changes. Go's build
cache is kept from build to build: the steps that compile mount it, so a package
whose inputs are unchanged is not compiled again, and both `go test` runs pass
`-count=1` so that no stored test result is replayed. The lint step also mounts
golangci-lint's cache, which reuses a package's findings only when its files,
those of the packages it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged. The `test`
phase runs on the Debian Go image, with Debian's libvips and libheif; the image
pixa ships is built and runs on Alpine. `script/check`, `script/cibuild`,
`script/docker`, `script/lint`, `script/test`, `script/setup` and
`script/install-precommit` are the standard copies from `sneak/prompts`, kept
identical to them. `script/fmt` and `script/fmt-check` are the standard copies
with pixa's `gofmt` step kept before prettier. prettier formats the markdown
only, not the HTML templates, as it cannot parse a Go template action inside a
tag.
## Load Test
+68 -35
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-10-15
last_modified: 2026-10-18
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -159,8 +159,11 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own golangci-lint cache, its
own `TMPDIR` and a digest-pinned binary, so neither is reachable.
org, in both directions. A container has its own `TMPDIR`, so the lock is
out of reach. The lint phase does keep golangci-lint's cache between check
builds (below), but it lints every checkout at `/src`, and a finding is
taken from that cache only for the same files, settings and golangci-lint
version.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
invalidates a `COPY` layer only when the copied content changes, so on an
@@ -170,15 +173,21 @@ style conventions are in separate documents:
id, so no two builds share a value, not even the three in one
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. Every
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
are compared by content and need nothing. A stage without the `ARG` line is
served from cache on an unchanged tree, so a repository re-vendors the scripts
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
it reruns every step too, but it also empties every cache mount, and Go's
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
packages survive between check builds, would start empty every time. A bare
two gate phases and `script/fmt-check`, and builds no image of its own. In
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
that install system packages and download Go's modules, or right after `FROM`
in a stage with neither, and every step that lints, tests or compiles comes
after it. A step that installs anything else as well, such as a
`script/bootstrap` that also installs node packages, comes after it too. A new
value reruns every `RUN` step after that line; `COPY` steps are compared by
content and need nothing. Docker keeps the layers above the line until the
base image, their own lines or the files they copy change. A stage without the
`ARG` line is served from cache on an unchanged tree, so a repository
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
not pass `--no-cache`: it reruns every step too, but it also builds the layers
above the line again and empties every cache mount, so Go's build cache and
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every
@@ -194,19 +203,25 @@ style conventions are in separate documents:
`Dockerfile`:
```dockerfile
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# In every stage ARG CACHEBUST comes right after the steps that install
# system packages and download Go's modules, or right after FROM in a
# stage with neither, and every step that lints, tests or compiles
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
ARG CACHEBUST
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
@@ -214,10 +229,10 @@ style conventions are in separate documents:
# cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
ARG CACHEBUST
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \
@@ -225,19 +240,19 @@ style conventions are in separate documents:
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# are what make BuildKit build them first, so nothing after them can
# run unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
COPY . .
# The VERSION build arg when one is given, otherwise
@@ -274,16 +289,26 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
it for root), and only on the steps that compile: `golangci-lint run`,
both `go test` runs and `go build`. It keeps compiled packages between
check builds, and Go reuses one only when all its inputs are unchanged, so
it saves the compile without skipping a check. Nothing else is kept:
golangci-lint's own cache would replay stored findings in place of a lint
run, and Go builds from whatever files a kept module cache holds without
checking them again (only `go mod verify` does), so a file changed there
would go unnoticed. Go's build cache also holds test results, which is why
both `go test` runs carry `-count=1`.
- Check builds keep these, and none of them can hand back a stale result:
- The layers above `ARG CACHEBUST`: the system packages and Go's
modules. Docker installs the packages again when the base image or the
install line changes, and downloads the modules again when `go.mod` or
`go.sum` changes. Go checks every module against `go.sum` as it
downloads it, and no later build can change a kept layer. The modules
are kept in a layer rather than a cache mount because a cache mount is
shared by every build on the host, and Go builds from whatever files
it holds without checking them again (only `go mod verify` does), so a
file changed there would go unnoticed.
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
root), mounted on the steps that compile: `golangci-lint run`, both
`go test` runs and `go build`. Go reuses a compiled package only when
all its inputs are unchanged. The cache also holds test results, which
is why both `go test` runs carry `-count=1`.
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
@@ -292,12 +317,20 @@ style conventions are in separate documents:
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
lists in the same `RUN`, so the layer does not keep them. The test phase,
on the Debian Go image, installs the libraries its tests need the same
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
Docker keeps it until the base image or the install line changes:
```dockerfile
FROM golangci/golangci-lint@sha256:... AS lint
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
+67
View File
@@ -30,6 +30,73 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-18 re-vendored from `sneak/prompts` commit `0653579` (closes #221):
each `Dockerfile` stage installs its system packages and downloads the Go
modules above `ARG CACHEBUST`, so check builds keep them, and the lint step
keeps golangci-lint's cache.
- 2026-10-19 a `trusted_proxies` entry in the IPv4-mapped form trusted nothing;
it is now read as the IPv4 range it covers by the same code that reads
`blocked_networks`, and such an entry shorter than `/96` aborts startup
(closes #303).
- 2026-10-18 the next reconciliation pass waits at least ten times as long as
the last one took, so passes on a large cache no longer run back to back
(closes #250).
- 2026-10-18 a `blocked_networks` entry in the IPv4-mapped form refused nothing;
`::ffff:10.0.0.0/104` is now read as `10.0.0.0/8`, and such an entry shorter
than `/96` aborts startup (closes #301).
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it
refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible
form (`::/96`) when `blocked_networks` contains either the address as written
or the IPv4 address it holds (closes #299). A source host such as
`[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that
range reaches the IPv4 address it holds, here `10.0.0.1`; and with
`203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but
`[::203.0.113.5]` was fetched. Both now answer 403. The mapped form
(`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go
keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must
not refuse every IPv4 address.
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
fetched, which on a network that routes it reaches the internal IPv4 host; it
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
as a whole, as one of the built-in blocked ranges, and still is.
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
and through an encrypted URL (closes #295). The fetcher's check of the source
URL kept the brackets when no port followed them, could not look the host up,
and the request answered 500. It now checks the address without brackets, so a
private one is refused with 403, as with a port.
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
`?` or `#` ended the path early, and the journal mode parameter was lost. The
derived `db_url` is now built with Go's `net/url`, which writes those
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
with `//` it starts `file:////`, so SQLite no longer reads the first name in
the path as a host.
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
proportions at 1 or more (closes #283): it was rounded down, so for a thin
source it came out at 0, and libvips then made the size square. An 8192x1
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
`fit=contain` and `fit=inside` work both sides out again from the smaller of
the two scales and `fit=outside` from the larger, and each keeps both sides at
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
process running it alone peaked at about 34 MiB resident. With `fit=outside`
the image is scaled to cover the size, so that source at `100x0` comes out at
8192x1. `README.md` now says this under Signature Specification, and says how
a `0` side is worked out where it said `0` meant the original.
- 2026-10-16 the generator page lower-cases the source host and writes an empty
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
(closes #293). A host written in upper and in lower case, or a source URL with
no path and with the path `/`, made two cache keys and two upstream requests
for one image.
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
source host, path and query, not joined as text (closes #288). The source
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
@@ -88,3 +88,56 @@ func TestBlockedNetworksInvalidAbortsStartup(t *testing.T) {
},
})
}
// TestBlockedNetworksMappedEntryIsIPv4Range checks that an entry written in
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
// since the fetcher matches an IPv4-mapped address as the IPv4 address it
// holds. /96, the shortest such entry, covers every IPv4 address.
func TestBlockedNetworksMappedEntryIsIPv4Range(t *testing.T) {
t.Parallel()
yamlContent := signingKeyLine + `blocked_networks:
- "::ffff:10.0.0.0/104"
- "::ffff:0:0/96"
`
c, err := configFromYAML(t, yamlContent)
if err != nil {
t.Fatalf("IPv4-mapped blocked_networks entries should load, got error: %v", err)
}
want := []string{"10.0.0.0/8", "0.0.0.0/0"}
if len(c.BlockedNetworks) != len(want) {
t.Fatalf("BlockedNetworks = %v, want %d entries", c.BlockedNetworks, len(want))
}
for i, w := range want {
if got := c.BlockedNetworks[i].String(); got != w {
t.Errorf("BlockedNetworks[%d] = %q, want %q", i, got, w)
}
}
}
// TestBlockedNetworksMappedEntryShorterThan96AbortsStartup checks that an
// entry in the IPv4-mapped form shorter than /96, which covers more than the
// IPv4 addresses, aborts startup with an error naming the key and the value.
func TestBlockedNetworksMappedEntryShorterThan96AbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, []abortCase{
{
name: "mapped-shorter-than-96",
yaml: signingKeyLine + `blocked_networks:
- "::ffff:0.0.0.0/80"
`,
wantErrSubstrings: []string{keyBlockedNetworks, "::ffff:0.0.0.0/80"},
},
{
name: "mapped-95",
yaml: signingKeyLine + `blocked_networks:
- "::ffff:0:0/95"
`,
wantErrSubstrings: []string{keyBlockedNetworks},
},
})
}
+47 -2
View File
@@ -91,6 +91,7 @@ var (
errNotABoolean = errors.New("not a boolean")
errNotAStringList = errors.New("not a list of strings")
errNotAValidCIDR = errors.New("not a valid CIDR network")
errMappedTooShort = errors.New("in the IPv4-mapped form but shorter than /96")
errNotAMetricsMap = errors.New("not a map of metrics settings")
errEmptyListEntry = errors.New("list contains an empty entry")
errEmptyEntry = errors.New("contains an empty entry")
@@ -338,8 +339,7 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
settingName(keyDBURL), errValueEmpty)
}
// The driver sets the journal mode only through a _pragma parameter.
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
c.DBURL = defaultDBURL(c.StateDir)
}
if loader.err != nil {
@@ -354,6 +354,30 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
return c, nil
}
// defaultDBURL returns the db_url used when db_url is omitted: the file
// state.sqlite3 in stateDir, in WAL mode, which the driver sets only
// through a _pragma parameter. String() percent-encodes any %, ? or # in
// the directory name, which SQLite decodes back, so the URL names exactly
// this file. OmitHost writes file:/var/lib/pixa/... instead of
// file:///var/lib/pixa/..., and keeps a relative stateDir from being read
// as a host name. The path written into the URL is stateDir followed by
// /state.sqlite3; when it starts with //, as it does for a stateDir of /
// or one starting with //, the empty host is kept, so the URL starts
// file://// and SQLite does not read the first name in the path as a
// host.
func defaultDBURL(stateDir string) string {
dbPath := stateDir + "/state.sqlite3"
dbURL := url.URL{
Scheme: "file",
OmitHost: !strings.HasPrefix(dbPath, "//"),
Path: dbPath,
RawQuery: "_pragma=journal_mode(WAL)",
}
return dbURL.String()
}
// validateKnownKeys rejects configuration files containing keys the
// application does not understand, so typos fail at startup instead of
// being silently ignored, and rejects keys that are explicitly set to
@@ -1225,6 +1249,13 @@ func defaultTrustedProxies() []netip.Prefix {
// wrong type, an empty entry, a non-string entry, or an unparseable CIDR
// aborts startup naming the key and the offending value; the default
// (an empty list) applies only to an omitted key.
//
// An entry written in the IPv4-mapped form (::ffff:a.b.c.d) is kept as the
// IPv4 range it covers: ::ffff:10.0.0.0/104 is kept as 10.0.0.0/8. Both
// blocked_networks and trusted_proxies are matched against an IPv4-mapped
// address as the IPv4 address it holds, so the entry as written would match
// nothing. Such an entry shorter than /96 covers more than the IPv4
// addresses and aborts startup.
func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
raw, ok := lookupValue(sc, key)
if !ok {
@@ -1240,6 +1271,10 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
return nil, err
}
// An IPv4-mapped address is the 96 bits of ::ffff:0:0/96 followed by
// the 32 bits of the IPv4 address it holds.
const mappedPrefixBits = 96
prefixes := make([]netip.Prefix, 0, len(entries))
for _, entry := range entries {
@@ -1249,6 +1284,16 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
settingName(key), entry, errNotAValidCIDR)
}
if prefix.Addr().Is4In6() && prefix.Bits() < mappedPrefixBits {
return nil, fmt.Errorf("%s: value %q is %w",
settingName(key), entry, errMappedTooShort)
}
if prefix.Addr().Is4In6() {
prefix = netip.PrefixFrom(prefix.Addr().Unmap(),
prefix.Bits()-mappedPrefixBits)
}
prefixes = append(prefixes, prefix)
}
@@ -135,6 +135,104 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
}
}
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
// derived from a state_dir whose name holds %41, ? and #, and checks that
// the database is state.sqlite3 in that directory, in WAL mode. Put into
// a file: URL as it is, the name would lose its meaning: %41 is read as
// A, and the path ends at the ?, which also loses the journal mode.
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
t.Parallel()
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
err := os.Mkdir(stateDir, 0o750)
if err != nil {
t.Fatalf("failed to create %q: %v", stateDir, err)
}
// Quoted so YAML takes the whole name as it is.
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
}
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
// derived from a state_dir that starts with //, and checks that the
// database is state.sqlite3 in that directory, in WAL mode. Written right
// after file:, such a name gives a URL starting file://, and SQLite reads
// the first directory name as a host and refuses to open it.
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
t.Parallel()
dir := t.TempDir()
stateDir := "/" + dir
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
}
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
// state_dir of /. The path written into the URL is then
// //state.sqlite3, and written right after file: it gives a URL starting
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
// it. The test only checks the URL, since it cannot write to /.
func TestDerivedDBURLForRootStateDir(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
if err != nil {
t.Fatalf("config with state_dir / should be valid, got: %v", err)
}
t.Logf("state_dir / gives db_url %q", c.DBURL)
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
if c.DBURL != wantDBURL {
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
}
}
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
// uses, and checks that the database is in WAL mode and is the file
// dbFile.
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
t.Helper()
db, err := sql.Open("sqlite", dbURL)
if err != nil {
t.Fatalf("failed to open %q: %v", dbURL, err)
}
t.Cleanup(func() { _ = db.Close() })
var journalMode string
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
}
if journalMode != "wal" {
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
}
_, err = os.Stat(dbFile)
if err != nil {
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
}
}
func TestExplicitValidValuesAreUsed(t *testing.T) {
t.Parallel()
@@ -83,3 +83,41 @@ func TestTrustedProxiesInvalidAbortsStartup(t *testing.T) {
},
})
}
// TestTrustedProxiesMappedEntryIsIPv4Range checks that an entry written in
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
// since the client address is matched as the IPv4 address it holds. /96, the
// shortest such entry, covers every IPv4 address.
func TestTrustedProxiesMappedEntryIsIPv4Range(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+
`trusted_proxies: ["::ffff:10.0.0.0/104", "::ffff:0:0/96"]`+"\n")
if err != nil {
t.Fatalf("IPv4-mapped trusted_proxies entries should load: %v", err)
}
got := make([]string, len(c.TrustedProxies))
for i, p := range c.TrustedProxies {
got[i] = p.String()
}
if joined := strings.Join(got, ","); joined != "10.0.0.0/8,0.0.0.0/0" {
t.Errorf("TrustedProxies = %v, want 10.0.0.0/8 and 0.0.0.0/0", got)
}
}
// TestTrustedProxiesMappedEntryShorterThan96AbortsStartup checks that an
// entry in the IPv4-mapped form shorter than /96, which covers more than the
// IPv4 addresses, aborts startup with an error naming the key and the value.
func TestTrustedProxiesMappedEntryShorterThan96AbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, []abortCase{
{
name: "mapped-95",
yaml: signingKeyLine + `trusted_proxies: ["::ffff:0:0/95"]` + "\n",
wantErrSubstrings: []string{keyTrustedProxies, "::ffff:0:0/95"},
},
})
}
+20 -1
View File
@@ -127,7 +127,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
sourceURL := r.FormValue("url")
// Validate source URL
parsed, err := url.Parse(sourceURL)
parsed, err := parseSourceURL(sourceURL)
if err != nil || parsed.Host == "" {
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
http.StatusBadRequest)
@@ -198,6 +198,25 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
}
}
// parseSourceURL parses the source URL entered on the generator page with its
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
// source URL, so that each spelling of one source URL makes one cache key and
// one upstream request.
func parseSourceURL(sourceURL string) (*url.URL, error) {
parsed, err := url.Parse(sourceURL)
if err != nil {
return nil, err
}
parsed.Host = strings.ToLower(parsed.Host)
if parsed.Path == "" {
parsed.Path = "/"
}
return parsed, nil
}
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
// image URL made from the source URL source, is source as it was written,
// apart from the scheme, which the token does not keep. It is not when source
+39 -12
View File
@@ -59,8 +59,9 @@ const (
// builtinBlockedPrefixes are internal or special-use ranges that Go's
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
// already cover. They are always blocked, in addition to any
// operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped
// before matching, so these IPv4 ranges are caught in both forms.
// operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6
// addresses are read as the IPv4 address they hold before matching, so
// these IPv4 ranges are caught in all three forms.
//
//nolint:gochecknoglobals // immutable built-in blocklist
var builtinBlockedPrefixes = []netip.Prefix{
@@ -68,6 +69,7 @@ var builtinBlockedPrefixes = []netip.Prefix{
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
}
// Fetcher errors.
@@ -473,16 +475,16 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
return ErrUnsupportedScheme
}
// Parse to extract host
host := extractHost(rawURL)
if host == "" {
return ErrInvalidHost
parsedURL, err := neturl.Parse(rawURL)
if err != nil {
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
}
// Remove port if present
h, _, err := net.SplitHostPort(host)
if err == nil {
host = h
// Hostname drops any port, and the brackets of an IPv6 address whether
// or not a port follows them
host := parsedURL.Hostname()
if host == "" {
return ErrInvalidHost
}
// Block obvious localhost patterns
@@ -543,6 +545,9 @@ func isPrivateIP(ip net.IP) bool {
return true
}
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
ip = ipv4CompatibleToIPv4(ip)
// Check for loopback
if ip.IsLoopback() {
return true
@@ -594,7 +599,12 @@ func isPrivateIP(ip net.IP) bool {
}
// isBlockedIP reports whether ip is refused, either by the built-in
// internal-range check or by one of the operator-supplied prefixes.
// internal-range check or by one of the operator-supplied prefixes. An
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
// not refuse every IPv4 address.
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
if isPrivateIP(ip) {
return true
@@ -607,11 +617,28 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
addr = addr.Unmap()
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
// addr refuses more only for an IPv4-compatible address.
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
return prefix.Contains(addr)
return prefix.Contains(addr) || prefix.Contains(asIPv4)
})
}
// ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
// address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
// any other address unchanged.
func ipv4CompatibleToIPv4(ip net.IP) net.IP {
addr, ok := netip.AddrFromSlice(ip)
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
return ip[12:]
}
return ip
}
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
// before connecting. New wraps dialSSRFSafe with the operator-supplied
// blocklist; this entry point enforces the built-in ranges alone.
@@ -186,6 +186,44 @@ func TestIsPrivateIP(t *testing.T) {
}
}
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
// link-local IPv4 address is refused as that address is, and one holding a
// public IPv4 address in the IPv4-compatible form passes.
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
t.Parallel()
tests := []struct {
ip string
want bool
}{
{"::127.0.0.1", true}, // loopback
{"::10.0.0.1", true}, // private
{"::169.254.169.254", true}, // link-local
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
{"::8.8.8.8", false}, // public
}
for _, tc := range tests {
t.Run(tc.ip, func(t *testing.T) {
t.Parallel()
ip := net.ParseIP(tc.ip)
if ip == nil {
t.Fatalf("failed to parse IP %q", tc.ip)
}
got := isPrivateIP(ip)
t.Logf("isPrivateIP(%s) = %v", ip, got)
if got != tc.want {
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
}
})
}
}
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
t.Parallel()
@@ -50,3 +50,69 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
})
}
}
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
// one side 0, the side computed from the input's proportions is 1 where it
// would round down to 0, and that Process makes that size in every fit mode
// but FitOutside, which scales the input to cover the size and so can make
// it longer. Each case checks the size targetDimensions gives before it
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
// libvips ask for about 39 GB with FitCover.
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
wantWidth, wantHeight int
}{
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
}
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
for _, tt := range tests {
for _, fit := range fits {
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
t.Parallel()
width, height, err := targetDimensions(tt.size,
tt.inputWidth, tt.inputHeight)
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
width, height, err, tt.wantWidth, tt.wantHeight)
}
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
if fit == FitOutside {
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
}
if !ok {
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
FitOutside)
}
})
}
}
}
@@ -60,3 +60,45 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
})
}
}
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
// keeps each side at 1 or more when it works both sides out again from the
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
// point is just under 1 and so rounded down to 0, a width libvips refuses.
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
wantWidth, wantHeight int
}{
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
}
})
}
}
+20 -12
View File
@@ -311,8 +311,10 @@ func (p *ImageProcessor) Process(
// targetDimensions calculates the output dimensions for a requested size,
// scaling proportionally when only one dimension is given and keeping the
// original dimensions when both are zero. A side scaled proportionally can
// come out above MaxOutputDimension, which is ErrOutputTooLarge.
// original dimensions when both are zero. A side scaled proportionally is
// rounded down but kept at 1 or more, as for a thin image it would round
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
// which is ErrOutputTooLarge.
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
width, height := size.Width, size.Height
@@ -322,10 +324,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
return origWidth, origHeight, nil
case width == 0:
// Only height specified: calculate width proportionally
width = origWidth * height / origHeight
width = max(1, origWidth*height/origHeight)
case height == 0:
// Only width specified: calculate height proportionally
height = origHeight * width / origWidth
height = max(1, origHeight*width/origWidth)
}
if width > MaxOutputDimension || height > MaxOutputDimension {
@@ -479,13 +481,16 @@ func (p *ImageProcessor) resize(
return cropCentreAndResize(img, width, height)
case FitContain:
// Resize to fit within dimensions, maintaining aspect ratio
// Resize to fit within dimensions, maintaining aspect ratio. Scaled
// by the smaller scale, the short side of a thin image can round
// down to 0, which libvips refuses as a width, so each side is at
// least 1.
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
newW := max(1, int(float64(imgW)*scale))
newH := max(1, int(float64(imgH)*scale))
return img.Thumbnail(newW, newH, vips.InterestingNone)
@@ -503,19 +508,22 @@ func (p *ImageProcessor) resize(
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
newW := max(1, int(float64(imgW)*scale))
newH := max(1, int(float64(imgH)*scale))
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitOutside:
// Resize so smallest dimension fits, may exceed target on other dimension
// Resize so smallest dimension fits, may exceed target on other
// dimension. A side scaled to 1 can come out just under 1 in
// floating point, as 49 times 1/49 does, and round down to 0, which
// libvips refuses as a width, so each side is at least 1.
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := max(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
newW := max(1, int(float64(imgW)*scale))
newH := max(1, int(float64(imgH)*scale))
// The dimension that exceeds the target is held to the maximum too
if newW > MaxOutputDimension || newH > MaxOutputDimension {
+46 -22
View File
@@ -18,6 +18,14 @@ import (
// write-pressure wakeups triggered by stores.
const DefaultEvictionInterval = 5 * time.Minute
// reconciliationWaitFactor is how many times as long as a
// reconciliation pass took the eviction loop waits at least, after that
// pass ends, before it starts the next one. A pass reads a row and stats a
// file for every cached file, so its time grows with the cache; waiting
// ten times as long keeps reconciliation to at most a tenth of the time,
// however large the cache is.
const reconciliationWaitFactor = 10
// evictionBatchSize is how many LRU candidates of each class (variants
// and source blobs) one eviction pass fetches from the database.
const evictionBatchSize = 100
@@ -438,10 +446,12 @@ func (c *Cache) notifyWritePressure() {
}
// StartEviction launches the background eviction goroutine, which
// reconciles the database accounting with the cache directories and
// removes the expired negative cache rows at startup and again on every
// periodic tick thereafter, and evicts to the configured limit on the
// given periodic interval and on write-pressure notifications. On a
// periodic tick thereafter, reconciles the database accounting with the
// cache directories at startup and again on the first periodic tick
// that comes at least ten times as long after a reconciliation pass
// ended as that pass took, and evicts to the configured limit on the given
// periodic interval and on write-pressure notifications. On a
// disabled cache it only removes the expired negative cache rows, as
// the negative cache is used either way. It is a no-op when already
// started. The goroutine outlives the caller, so it runs with its own
@@ -482,7 +492,7 @@ func (c *Cache) StopEviction(ctx context.Context) error {
func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
defer close(c.evictionDone)
c.runReconciliationPass(ctx)
nextReconciliation := c.runReconciliationPass(ctx)
c.runCleanExpired(ctx)
c.runEvictionPass(ctx)
@@ -494,16 +504,20 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
case <-ctx.Done():
return
case <-ticker.C:
// Reconciliation walks the cache directories, so it only
// runs on the periodic ticker rather than on every
// write-pressure wakeup, keeping it off the per-store hot
// path. Reusing the eviction interval itself (rather than a
// separate, longer one) is a deliberate choice: it is the
// simplest option that still bounds how long a store's
// best-effort accounting insert can stay silently
// unaccounted for to one interval, on a process that is
// already running this loop regardless.
c.runReconciliationPass(ctx)
// Reconciliation walks the cache directories and reads a
// row for every file, so it runs only on the periodic
// ticker rather than on every write-pressure wakeup, and
// only on the first tick at or after nextReconciliation,
// which is ten times as long after the last pass ended as
// that pass took. Passes then never run back to back,
// however long one takes. A file whose best-effort
// accounting insert failed in a store stays unaccounted for
// until the next pass, up to about one eviction interval
// plus ten times the last pass's time.
if !time.Now().Before(nextReconciliation) {
nextReconciliation = c.runReconciliationPass(ctx)
}
c.runCleanExpired(ctx)
case <-c.evictionPressure:
}
@@ -527,17 +541,25 @@ func (c *Cache) runEvictionPass(ctx context.Context) {
}
// runReconciliationPass runs one reconciliation pass, logging failures
// instead of propagating them (the loop must keep running). It does
// nothing once ctx is cancelled.
func (c *Cache) runReconciliationPass(ctx context.Context) {
// instead of propagating them (the loop must keep running), and returns
// the earliest time the next pass may start: reconciliationWaitFactor
// times as long after this pass ended as it took. It does nothing once
// ctx is cancelled.
func (c *Cache) runReconciliationPass(ctx context.Context) time.Time {
if ctx.Err() != nil {
return
return time.Now()
}
started := time.Now()
err := c.reconcileAccounting(ctx)
if err != nil {
c.log.Warn("cache accounting reconciliation failed", "error", err)
}
ended := time.Now()
return ended.Add(reconciliationWaitFactor * ended.Sub(started))
}
// runCleanExpired runs CleanExpired, logging a failure instead of
@@ -556,7 +578,8 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// reconcileAccounting synchronizes the database size accounting with
// the actual contents of the cache directories. It runs at startup and
// again on every periodic eviction tick thereafter, off the request
// then on the first periodic eviction tick that comes at least ten times
// as long after the last pass ended as that pass took, off the request
// hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
// insert under transient DB contention), drops accounting rows whose
@@ -566,9 +589,10 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// files left behind by crashed writes, and last checks the total cache
// usage against the tables. It reads the tables a page at a time.
// Running it periodically, not just once, bounds how long such drift can
// accumulate unaccounted for on a long-running process to one eviction
// interval. Once ctx is cancelled, it stops at the next file or row and
// returns ctx's error.
// accumulate unaccounted for on a long-running process to about one
// eviction interval plus ten times the last pass's time.
// Once ctx is cancelled, it stops at the next file or row and returns
// ctx's error.
func (c *Cache) reconcileAccounting(ctx context.Context) error {
if c.disabled {
return nil
@@ -918,6 +918,89 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
}
}
// TestReconciliationPassesAreSpacedByTenTimesTheirTime makes the hook
// called after every read of the reconciliation pass sleep 10 ms, runs
// the eviction loop on a 10 ms interval, and records when each of those
// calls starts and ends. A pass starts before its first call and ends
// after its last, so the time from the first call's start to the last
// call's end is at most the pass's time. The next pass's first call must
// start at least ten times that long after the last call ended.
func TestReconciliationPassesAreSpacedByTenTimesTheirTime(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
// A pass made here, before the loop starts, counts the reads one
// pass makes on this empty cache, so the reads recorded below can be
// split into passes.
readsPerPass := 0
cache.reconciliationReadTestHook = func(int) { readsPerPass++ }
err := cache.reconcileAccounting(t.Context())
if err != nil {
t.Fatalf("reconcileAccounting failed: %v", err)
}
t.Logf("one reconciliation pass makes %d reads", readsPerPass)
if readsPerPass == 0 {
t.Fatal("a reconciliation pass made no read")
}
// The pass at startup and two after it.
const passes = 3
type read struct{ start, end time.Time }
// The channel holds every read the test looks at, so none of those
// is dropped; reads after them are, and never block the loop.
reads := make(chan read, passes*readsPerPass)
cache.reconciliationReadTestHook = func(int) {
start := time.Now()
time.Sleep(10 * time.Millisecond)
select {
case reads <- read{start: start, end: time.Now()}:
default:
}
}
cache.StartEviction(10 * time.Millisecond)
defer func() { _ = cache.StopEviction(t.Context()) }()
recorded := make([]read, 0, passes*readsPerPass)
timeout := time.After(15 * time.Second)
for len(recorded) < passes*readsPerPass {
select {
case r := <-reads:
recorded = append(recorded, r)
case <-timeout:
t.Fatalf("saw %d reconciliation reads in 15 s, want %d (%d passes)",
len(recorded), passes*readsPerPass, passes)
}
}
for pass := range passes - 1 {
first := recorded[pass*readsPerPass]
last := recorded[(pass+1)*readsPerPass-1]
next := recorded[(pass+1)*readsPerPass]
passTime := last.end.Sub(first.start)
wait := next.start.Sub(last.end)
t.Logf("pass %d took at least %v, and the next pass started %v after it",
pass+1, passTime, wait)
if wait < 10*passTime {
t.Errorf("pass %d took at least %v, and the next pass started %v "+
"after it, want at least %v", pass+1, passTime, wait, 10*passTime)
}
}
}
// TestStopEvictionInterruptsPassInProgress holds the test database's
// only connection, so the startup reconciliation pass waits for it, and
// checks that StopEviction stops that pass instead of waiting for the
+3 -1
View File
@@ -29,7 +29,9 @@ var (
const MaxDimension = imageprocessor.MaxOutputDimension
// ValidateDimension checks one requested width or height; name ("width" or
// "height") appears in the error. 0 means "original size" and is valid.
// "height") appears in the error. 0 is valid: that side is then worked out from
// the source's proportions and the other requested side, rounded down but at
// least 1, or is the source's own side when the other side is 0 too.
func ValidateDimension(name string, value int) error {
if value < 0 {
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
@@ -14,10 +14,12 @@ import (
"net"
"net/http"
"net/http/httptest"
"net/netip"
"os"
"path/filepath"
"sync/atomic"
"testing"
"time"
"go.uber.org/fx"
"go.uber.org/fx/fxtest"
@@ -183,6 +185,57 @@ func startImageProxy(
return s, db.DB(), stateDir
}
// startImageProxyWithBlockedNetworks starts the components pixad's fx app
// builds, from a config with a fresh state directory and blockedNetworks as
// blocked_networks. Unlike startImageProxy it provides no upstream fetcher, so
// the handlers build the one pixad uses, which checks every address it
// connects to against the built-in blocked ranges and blocked_networks. It
// returns the server with its routes.
func startImageProxyWithBlockedNetworks(
t *testing.T, blockedNetworks []netip.Prefix,
) *Server {
t.Helper()
stateDir := t.TempDir()
cfg := &config.Config{
SigningKey: testSigningKey,
StateDir: stateDir,
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
BlockedNetworks: blockedNetworks,
// A fetch that gets past the checks and finds nothing at the address
// fails within this time at the latest.
UpstreamFetchTimeout: 2 * time.Second,
// With no connection slots, the fetch would fail before connecting.
UpstreamConnections: config.DefaultUpstreamConnections,
DownstreamTimeout: config.DefaultDownstreamTimeout,
}
var (
h *handlers.Handlers
mw *middleware.Middleware
)
app := fxtest.New(t,
fx.Supply(cfg),
fx.Provide(
globals.New,
logger.New,
database.New,
healthcheck.New,
handlers.New,
middleware.New,
),
fx.Populate(&h, &mw),
)
app.RequireStart()
t.Cleanup(app.RequireStop)
s := &Server{config: cfg, mw: mw, h: h}
s.SetupRoutes()
return s
}
// getImage sends a GET for target to s and fails unless it answers 200.
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
t.Helper()
+294
View File
@@ -3,15 +3,20 @@ package server
import (
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
)
// percentSignPath is a source path holding a "%", which a URL writes %25.
@@ -308,6 +313,70 @@ func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
}
}
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
// spellings of one source URL on the generator page, requests the encrypted
// URL made for each, and checks that the upstream receives one request and
// that the second encrypted URL is served from the cache. One pair writes the
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
// an IPv6 address, whose hex digits have letter case, because the fetcher
// would look a host name up in DNS. It has a port, as the fetcher refuses an
// IPv6 address written without one. The other pair is a source URL with no
// path and with the path "/", both of which fetch "/".
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
t.Parallel()
tests := []struct {
name string
// sourceURLs are the two spellings, after "https://"
sourceURLs []string
// wantRequest is the path the upstream must receive
wantRequest string
}{
{"host in upper and in lower case",
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
"/a.png"},
{"no path and the path written as a slash",
[]string{upstreamHost, upstreamHost + "/"}, "/"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
var imageURLs []string
for _, sourceURL := range tt.sourceURLs {
rec := postGenerate(t, s, "https://"+sourceURL)
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
if rec.Code != http.StatusOK || match == nil {
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
rec.Code, rec.Body.String())
}
imageURLs = append(imageURLs, match[1])
}
getImage(t, s, imageURLs[0])
second := getImage(t, s, imageURLs[1])
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.wantRequest}) {
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
}
})
}
}
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
// generator page must make no encrypted URL for, and checks that it answers
// 400 and shows why. pixa keeps the source path decoded and fetches it
@@ -369,3 +438,228 @@ func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
})
}
}
// ipv6SourcePath is the source path of the image URLs signedImageURL and
// encryptedImageURL make.
const ipv6SourcePath = "/ipv6.png"
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
// on an IPv6 source host, written with and without a port, through a
// /v1/image/ URL with a signature and through an encrypted URL. The public
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
// as public, answers 200, and the upstream receives the request for
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
// receives no request: the test's fetcher connects every address to the test
// upstream, so only the fetcher's check of the source URL can refuse it.
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourceHost string
wantStatus int
// wantRequests are the request targets the upstream must receive
wantRequests []string
}{
{"public address without a port", "[2001:db8::a]",
http.StatusOK, []string{ipv6SourcePath}},
{"public address with a port", "[2001:db8::a]:8443",
http.StatusOK, []string{ipv6SourcePath}},
{"private address without a port", "[fd00::1]",
http.StatusForbidden, nil},
{"private address with a port", "[fd00::1]:8443",
http.StatusForbidden, nil},
}
for _, tt := range tests {
imageURLs := map[string]string{
"image URL with a signature": signedImageURL(t, tt.sourceHost),
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
}
for route, imageURL := range imageURLs {
t.Run(tt.name+" "+route, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s: %d", imageURL, rec.Code)
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d; body %s",
rec.Code, tt.wantStatus, rec.Body.String())
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, tt.wantRequests) {
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
}
})
}
}
}
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
// its original size as PNG, with a signature made with the test signing key,
// which a host not on the allowlist needs. The URL has no q or fit, so it is
// signed with the quality 85 and the fit cover.
func signedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
&signature.Request{
SourceHost: sourceHost,
SourcePath: ipv6SourcePath,
Format: string(imgcache.FormatPNG),
Quality: 85,
FitMode: string(imgcache.FitCover),
}, time.Hour)
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
}
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
// at its original size as PNG, made with the test signing key.
func encryptedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
generator, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
}
token, err := generator.Generate(&encurl.Payload{
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
Format: imgcache.FormatPNG,
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
return "/v1/e/" + token + "/img.png"
}
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
// through an encrypted URL. Each answers 403, and the upstream receives no
// request: the test's fetcher connects every address to the test upstream, so
// only the fetcher's check of the source URL can refuse it.
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
sourceHosts := []string{
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
"[::10.0.0.1]", // IPv4-compatible form of a private address
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
}
for _, sourceHost := range sourceHosts {
imageURLs := []string{
signedImageURL(t, sourceHost),
encryptedImageURL(t, sourceHost),
}
for _, imageURL := range imageURLs {
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
if rec.Code != http.StatusForbidden {
t.Errorf("source host %s: status = %d, want %d; body %s",
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
}
}
}
if got := upstream.requestTargets(); len(got) != 0 {
t.Errorf("upstream received %q, want no request", got)
}
}
// TestImageRoutesCheckIPv4AddressInIPv6SourceHost requests ipv6SourcePath on
// source hosts that hold an IPv4 address, through a /v1/image/ URL with a
// signature and through an encrypted URL, with the fetcher pixad uses.
// [64:ff9b:1::a00:1], in the local-use NAT64 range (64:ff9b:1::/48, RFC 8215),
// answers 403. [::203.0.113.5] holds the public address 203.0.113.5 in the
// IPv4-compatible form (::/96) and answers 403 when blocked_networks lists
// 203.0.113.0/24 or ::/96. [::ffff:203.0.113.5] holds it in the IPv4-mapped
// form (::ffff:0:0/96) and answers 403 when blocked_networks lists
// 203.0.113.0/24. Go keeps the plain address 203.0.113.5 in the mapped form
// too, and an IPv6 range must not refuse it: with ::/0 listed it is not
// refused. Neither is [::203.0.113.5] with blocked_networks empty. Where a
// host is not refused, the fetcher tries to connect to it, finds nothing
// there, and the request fails with another status.
func TestImageRoutesCheckIPv4AddressInIPv6SourceHost(t *testing.T) {
t.Parallel()
const (
ipv4CompatibleHost = "[::203.0.113.5]"
ipv4MappedHost = "[::ffff:203.0.113.5]"
)
tests := []struct {
name string
sourceHost string
blockedNetworks []netip.Prefix
wantForbidden bool
}{
{"local-use NAT64 form of 10.0.0.1", "[64:ff9b:1::a00:1]", nil, true},
{"IPv4-compatible form of an address in blocked_networks",
ipv4CompatibleHost,
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
{"IPv4-compatible form with ::/96 in blocked_networks",
ipv4CompatibleHost,
[]netip.Prefix{netip.MustParsePrefix("::/96")}, true},
{"IPv4-mapped form of an address in blocked_networks",
ipv4MappedHost,
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
{"plain IPv4 address with ::/0 in blocked_networks",
"203.0.113.5",
[]netip.Prefix{netip.MustParsePrefix("::/0")}, false},
{"IPv4-compatible form of an address with blocked_networks empty",
ipv4CompatibleHost, nil, false},
}
for _, tt := range tests {
imageURLs := map[string]string{
"image URL with a signature": signedImageURL(t, tt.sourceHost),
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
}
for route, imageURL := range imageURLs {
t.Run(tt.name+" "+route, func(t *testing.T) {
t.Parallel()
s := startImageProxyWithBlockedNetworks(t, tt.blockedNetworks)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s: %d", imageURL, rec.Code)
forbidden := rec.Code == http.StatusForbidden
if tt.wantForbidden && !forbidden {
t.Errorf("status = %d, want %d; body %s",
rec.Code, http.StatusForbidden, rec.Body.String())
}
if !tt.wantForbidden && forbidden {
t.Errorf("status = %d, want any other status; body %s",
rec.Code, rec.Body.String())
}
})
}
}
}
+2 -3
View File
@@ -19,9 +19,8 @@
# CGO image libraries (pkg-config, vips
# with its JPEG XL support, libheif) for
# the govips bindings instead of Node: to
# compile pixa, in the Dockerfile's test
# phase and build stage, which format
# nothing
# compile pixa outside Docker, as
# `make build` does
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+5 -4
View File
@@ -2,11 +2,12 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
# line, since a cached gate phase is a check that did not run; unlike
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
# before every step that lints, tests or compiles; a stage without that
# line is served from cache on an unchanged tree, with no warning.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+5 -4
View File
@@ -1,11 +1,12 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
# line, since a cached gate phase is a check that did not run; unlike
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
# before every step that lints, tests or compiles; a stage without that
# line is served from cache on an unchanged tree, with no warning.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+7 -5
View File
@@ -5,11 +5,13 @@
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. A new CACHEBUST value reruns every RUN step, since
# a cached lint layer is a lint that did not run; unlike --no-cache, it
# keeps Go's build cache. This relies on every stage of the Dockerfile
# declaring ARG CACHEBUST first; a stage without that line is served
# from cache on an unchanged tree, with no warning.
# --target names it. A new CACHEBUST value reruns every RUN step after
# the ARG CACHEBUST line, since a cached lint layer is a lint that did
# not run; unlike --no-cache, it keeps the layers above that line and
# the cache mounts. This relies on every stage of the Dockerfile
# declaring ARG CACHEBUST before every step that lints, tests or
# compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one.
set -eu
+6 -5
View File
@@ -3,11 +3,12 @@
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, and a new CACHEBUST value because a cached test layer is a test
# that did not run; unlike --no-cache, it keeps Go's build cache. This
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
# a stage without that line is served from cache on an unchanged tree,
# with no warning. --output type=cacheonly writes no image, since
# nothing uses one.
# that did not run; unlike --no-cache, it keeps the layers above the ARG
# CACHEBUST line and the cache mounts. This relies on every stage of the
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests
# or compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning. --output type=cacheonly writes no
# image, since nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"