Compare commits
6
Commits
next
..
fb78b881a6
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fb78b881a6 | ||
|
|
58566b78cc | ||
|
|
51528c92b4 | ||
|
|
1911489815 | ||
|
|
8c9623f56c | ||
|
|
3a06283a94 |
+22
-31
@@ -1,16 +1,12 @@
|
||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
||||
# system packages and download Go's modules, or right after FROM in a
|
||||
# stage with neither, and every step that lints, tests or compiles
|
||||
# comes after it. script/ passes a new value on every build, which
|
||||
# reruns every RUN step after it. Docker keeps the layers above it
|
||||
# until the base image, their own lines or go.mod and go.sum change.
|
||||
# The steps that compile mount Go's build cache, and golangci-lint run
|
||||
# also mounts golangci-lint's; both survive from build to build.
|
||||
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
||||
# on every build, which reruns every RUN step after it. The steps that
|
||||
# compile mount Go's build cache, which survives from build to build.
|
||||
|
||||
# Lint phase. script/lint builds it alone. The linter is run directly:
|
||||
# `make lint` and script/lint are themselves a docker build.
|
||||
# golangci/golangci-lint:v2.14.0, 2026-10-15
|
||||
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
|
||||
ARG CACHEBUST
|
||||
|
||||
# The linter compiles every package, and govips needs the libvips
|
||||
# headers for that. REPO_POLICIES.md has the lint phase install them
|
||||
@@ -22,29 +18,25 @@ RUN apt-get update \
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
COPY . .
|
||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
||||
golangci-lint run --config .golangci.yml ./...
|
||||
|
||||
# Test phase. script/test builds it alone. -race needs cgo and so a C
|
||||
# compiler, which the Debian Go image ships and the alpine one does not.
|
||||
# golang:1.25.4-trixie, 2026-10-15
|
||||
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
|
||||
|
||||
# govips compiles against the libvips and libheif headers, and the tests
|
||||
# need libvips' JPEG XL support; libvips-dev brings all three. The tests
|
||||
# also encode AVIF, which needs libheif's AVIF encoder,
|
||||
# libheif-plugin-aomenc: Debian only recommends it, so it is named here.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends libvips-dev libheif-plugin-aomenc \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
ARG CACHEBUST
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
# script/bootstrap --cgo installs the build dependencies (the libvips
|
||||
# and libheif headers, and libvips' JPEG XL support, which the tests
|
||||
# need) and downloads the Go modules.
|
||||
COPY script/ ./script/
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
RUN script/bootstrap --cgo
|
||||
|
||||
COPY . .
|
||||
|
||||
# Without -v first; on a failure, again with -v for the details, and
|
||||
@@ -60,23 +52,22 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
|
||||
|
||||
# Build stage. Nothing is wanted from the two phases above: these copies
|
||||
# make BuildKit build them first, so nothing after them runs unless lint
|
||||
# and test passed.
|
||||
# make BuildKit build them first, so this stage runs only when lint and
|
||||
# test passed.
|
||||
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
|
||||
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
|
||||
|
||||
# git for the version, and for govips a C compiler, pkg-config and the
|
||||
# libvips and libheif headers
|
||||
RUN apk add --no-cache git build-base pkgconfig vips-dev libheif-dev
|
||||
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
|
||||
COPY --from=lint /src/go.sum /dev/null
|
||||
COPY --from=test /src/go.sum /dev/null
|
||||
|
||||
WORKDIR /src
|
||||
|
||||
# Build dependencies, git among them, and Go modules, as in the test
|
||||
# phase
|
||||
COPY script/ ./script/
|
||||
COPY go.mod go.sum ./
|
||||
RUN script/bootstrap --cgo
|
||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||
RUN git config --system --add safe.directory /src
|
||||
|
||||
@@ -111,6 +102,7 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
# the libvips and musl it was built with.
|
||||
# alpine:3.22, 2026-10-08
|
||||
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
|
||||
ARG CACHEBUST
|
||||
|
||||
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
|
||||
# support, without which pixad does not start.
|
||||
@@ -121,7 +113,6 @@ RUN apk add --no-cache \
|
||||
ca-certificates \
|
||||
tzdata \
|
||||
su-exec
|
||||
ARG CACHEBUST
|
||||
|
||||
# Copy binary from builder
|
||||
COPY --from=builder /pixad /usr/local/bin/pixad
|
||||
|
||||
@@ -102,8 +102,8 @@ it uses libvips through CGO. pixad does not start unless libvips has its JPEG XL
|
||||
support, which on Alpine is the `vips-jxl` package and which the nix and brew
|
||||
packages of libvips include, as do the apt ones from Debian 12 and Ubuntu 24.04
|
||||
on. Building pixa also needs the development files of libvips and libheif,
|
||||
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these;
|
||||
the `Dockerfile` installs its own packages. Plain `script/bootstrap`, which
|
||||
`pkg-config` and a C compiler. `script/bootstrap --cgo` installs all of these,
|
||||
as the `Dockerfile` does where it compiles pixa. Plain `script/bootstrap`, which
|
||||
`script/setup` and `script/cibuild` run, installs git, make and Go, and Node,
|
||||
Yarn and the prettier pinned in `yarn.lock` for formatting the markdown, but
|
||||
none of the C libraries: the checks compile pixa in Docker. Docker itself must
|
||||
@@ -199,11 +199,7 @@ into place, so a crash leaves no file under its own name with part of its data.
|
||||
When pixa starts, and every five minutes after, it removes each cached source or
|
||||
transformed image whose file is empty, or not the size the database records for
|
||||
it, with a warning naming the file; the next request for it fetches the source
|
||||
or converts the image again. Looking for those files reads the database once for
|
||||
every cached file, so pixa skips each five-minute look that would start less
|
||||
than ten times as long after the last look ended as that look took. After a look
|
||||
that took one minute, the next one starts 10 to 15 minutes after that look
|
||||
ended.
|
||||
or converts the image again.
|
||||
|
||||
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
|
||||
Test). Toward it, pixa keeps in memory the content types of the 10,000
|
||||
@@ -477,11 +473,9 @@ nor change what it asks for.
|
||||
`jxl` for a form with no format.
|
||||
|
||||
The token holds the source's host, path and query and the size, format, quality,
|
||||
fit and expiry, encrypted with a key derived from `signing_key`. The host is
|
||||
kept in lower case and an empty path as `/`, as for `/v1/image/`, so
|
||||
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image,
|
||||
fetched and cached once. The source URL's scheme is not kept: the image is
|
||||
fetched like any other (see Routes), and the blocked networks still apply.
|
||||
fit and expiry, encrypted with a key derived from `signing_key`. The source
|
||||
URL's scheme is not kept: the image is fetched like any other (see Routes), and
|
||||
the blocked networks still apply.
|
||||
|
||||
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
|
||||
The expiry is fixed in the token when the URL is made and cannot be changed or
|
||||
@@ -540,13 +534,8 @@ Where:
|
||||
- `query` — source query string as under Routes, empty string if none, with `%`
|
||||
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
|
||||
query and path `/a` with query `b:` are signed differently
|
||||
- `width` — requested width in pixels, or `0`: the width is then worked out from
|
||||
the source's proportions and the requested height, rounded down but at least
|
||||
1, or is the source's own width when the height is `0` too (the `orig` size)
|
||||
- `height` — requested height in pixels, or `0`: the height is then worked out
|
||||
from the source's proportions and the requested width, rounded down but at
|
||||
least 1, or is the source's own height when the width is `0` too (the `orig`
|
||||
size)
|
||||
- `width` — requested width in pixels, `0` for original
|
||||
- `height` — requested height in pixels, `0` for original
|
||||
- `format` — output format, one of those listed under Routes, with `original`
|
||||
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
|
||||
format has the signature of the same URL ending in `.jxl`; `auto` is signed as
|
||||
@@ -559,10 +548,7 @@ Where:
|
||||
with 400
|
||||
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
|
||||
outside), or `cover` when the URL has no `fit`; a request whose `fit` is
|
||||
anything else, an empty `fit=` included, is refused with 400. `outside` scales
|
||||
the image to cover the requested size, keeping its proportions, so it can come
|
||||
out larger than that size on one side: an 8192x1 source at `100x0`, which is
|
||||
100x1 by its proportions, comes out at 8192x1
|
||||
anything else, an empty `fit=` included, is refused with 400
|
||||
|
||||
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
|
||||
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
|
||||
@@ -698,12 +684,7 @@ Key settings in more detail:
|
||||
and generator pages. Default: empty
|
||||
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
|
||||
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
|
||||
benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6
|
||||
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
||||
address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
|
||||
plain IPv4 address. An entry written in that form is read as the IPv4 range it
|
||||
covers, so `::ffff:10.0.0.0/104` refuses what `10.0.0.0/8` refuses; such an
|
||||
entry shorter than `/96` covers more than IPv4 addresses and aborts startup
|
||||
benchmark, NAT64, and the like); an invalid CIDR aborts startup
|
||||
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
|
||||
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
|
||||
of these ranges; the logged and login-recorded client address is then the
|
||||
@@ -713,22 +694,16 @@ Key settings in more detail:
|
||||
omitted key defaults to the RFC 1918 private ranges (`10.0.0.0/8`,
|
||||
`172.16.0.0/12`, `192.168.0.0/16`), since pixa is deployed behind a proxy on a
|
||||
private network; an explicitly empty list (`[]`) trusts no one, and an
|
||||
explicit list replaces the default. An invalid CIDR aborts startup. An IPv6
|
||||
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
|
||||
address it holds, so an IPv6 range such as `::/0` trusts neither it nor a
|
||||
plain IPv4 address. An entry written in that form is read as the IPv4 range it
|
||||
covers, so `::ffff:10.0.0.0/104` trusts what `10.0.0.0/8` trusts; such an
|
||||
entry shorter than `/96` covers more than IPv4 addresses and aborts startup.
|
||||
Set this to the address pixa sees for requests that come through your proxy,
|
||||
such as `172.17.0.1/32`, when the defaults do not cover it, or to trust
|
||||
nothing else (see the login limit and `image_requests_per_minute` under
|
||||
Routes). For a proxy on the Docker host that connects to pixa over
|
||||
`127.0.0.1`, that address is the gateway of the container's Docker network
|
||||
(`172.17.0.1` on the default bridge), not the proxy's own address; a proxy
|
||||
that connects through another of the host's addresses is seen with that
|
||||
address. To be sure which address it is, set this to `[]` (or
|
||||
`PIXA_TRUSTED_PROXIES` to empty), send a request through the proxy, and read
|
||||
`remoteIP` in pixa's request log line for it
|
||||
explicit list replaces the default. An invalid CIDR aborts startup. Set this
|
||||
to the address pixa sees for requests that come through your proxy, such as
|
||||
`172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else
|
||||
(see the login limit and `image_requests_per_minute` under Routes). For a
|
||||
proxy on the Docker host that connects to pixa over `127.0.0.1`, that address
|
||||
is the gateway of the container's Docker network (`172.17.0.1` on the default
|
||||
bridge), not the proxy's own address; a proxy that connects through another of
|
||||
the host's addresses is seen with that address. To be sure which address it
|
||||
is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
|
||||
through the proxy, and read `remoteIP` in pixa's request log line for it
|
||||
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
|
||||
as a duration such as `30s` (the default) or `2m`
|
||||
- `upstream_max_response_size` — largest upstream response accepted, in bytes;
|
||||
@@ -837,25 +812,19 @@ provide:
|
||||
`script/precommit`
|
||||
|
||||
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
|
||||
build argument, which every stage of the `Dockerfile` declares, so every `RUN`
|
||||
step after that line, the lint and test steps included, runs on every build
|
||||
instead of coming from the build cache. Above the line are the steps that
|
||||
install system packages and download the Go modules; Docker keeps their layers
|
||||
until the base image, the step itself, `go.mod` or `go.sum` changes. Go's build
|
||||
cache is kept from build to build: the steps that compile mount it, so a package
|
||||
whose inputs are unchanged is not compiled again, and both `go test` runs pass
|
||||
`-count=1` so that no stored test result is replayed. The lint step also mounts
|
||||
golangci-lint's cache, which reuses a package's findings only when its files,
|
||||
those of the packages it imports, `go.mod`, the linters and their settings in
|
||||
`.golangci.yml`, and the golangci-lint version are all unchanged. The `test`
|
||||
phase runs on the Debian Go image, with Debian's libvips and libheif; the image
|
||||
pixa ships is built and runs on Alpine. `script/check`, `script/cibuild`,
|
||||
`script/docker`, `script/lint`, `script/test`, `script/setup` and
|
||||
`script/install-precommit` are the standard copies from `sneak/prompts`, kept
|
||||
identical to them. `script/fmt` and `script/fmt-check` are the standard copies
|
||||
with pixa's `gofmt` step kept before prettier. prettier formats the markdown
|
||||
only, not the HTML templates, as it cannot parse a Go template action inside a
|
||||
tag.
|
||||
build argument, which every stage of the `Dockerfile` declares first, so every
|
||||
`RUN` step, the lint and test phases included, runs on every build instead of
|
||||
coming from the build cache. Go's build cache is kept from build to build: the
|
||||
steps that compile mount it, so a package whose inputs are unchanged is not
|
||||
compiled again, and both `go test` runs pass `-count=1` so that no stored test
|
||||
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
|
||||
libvips and libheif; the image pixa ships is built and runs on Alpine.
|
||||
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
|
||||
`script/setup` and `script/install-precommit` are the standard copies from
|
||||
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
|
||||
the standard copies with pixa's `gofmt` step kept before prettier. prettier
|
||||
formats the markdown only, not the HTML templates, as it cannot parse a Go
|
||||
template action inside a tag.
|
||||
|
||||
## Load Test
|
||||
|
||||
|
||||
+34
-67
@@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Repository Policies
|
||||
last_modified: 2026-10-18
|
||||
last_modified: 2026-10-15
|
||||
---
|
||||
|
||||
This document covers repository structure, tooling, and workflow standards. Code
|
||||
@@ -159,11 +159,8 @@ style conventions are in separate documents:
|
||||
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
|
||||
exit non-zero with `parallel golangci-lint is running` — a status a caller
|
||||
cannot tell from real findings. Both have produced wrong verdicts in this
|
||||
org, in both directions. A container has its own `TMPDIR`, so the lock is
|
||||
out of reach. The lint phase does keep golangci-lint's cache between check
|
||||
builds (below), but it lints every checkout at `/src`, and a finding is
|
||||
taken from that cache only for the same files, settings and golangci-lint
|
||||
version.
|
||||
org, in both directions. A container has its own golangci-lint cache, its
|
||||
own `TMPDIR` and a digest-pinned binary, so neither is reachable.
|
||||
|
||||
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
|
||||
invalidates a `COPY` layer only when the copied content changes, so on an
|
||||
@@ -173,21 +170,15 @@ style conventions are in separate documents:
|
||||
id, so no two builds share a value, not even the three in one
|
||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
||||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
||||
that install system packages and download Go's modules, or right after `FROM`
|
||||
in a stage with neither, and every step that lints, tests or compiles comes
|
||||
after it. A step that installs anything else as well, such as a
|
||||
`script/bootstrap` that also installs node packages, comes after it too. A new
|
||||
value reruns every `RUN` step after that line; `COPY` steps are compared by
|
||||
content and need nothing. Docker keeps the layers above the line until the
|
||||
base image, their own lines or the files they copy change. A stage without the
|
||||
`ARG` line is served from cache on an unchanged tree, so a repository
|
||||
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
||||
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
||||
above the line again and empties every cache mount, so Go's build cache and
|
||||
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
||||
survive between check builds, would start empty every time. A bare
|
||||
two gate phases and `script/fmt-check`, and builds no image of its own. Every
|
||||
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
|
||||
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
|
||||
are compared by content and need nothing. A stage without the `ARG` line is
|
||||
served from cache on an unchanged tree, so a repository re-vendors the scripts
|
||||
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
|
||||
it reruns every step too, but it also empties every cache mount, and Go's
|
||||
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
|
||||
packages survive between check builds, would start empty every time. A bare
|
||||
`docker build .` is not evidence that anything ran: a sub-second build
|
||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
||||
`docker builder prune` and friends destroy a build cache shared with every
|
||||
@@ -203,25 +194,19 @@ style conventions are in separate documents:
|
||||
`Dockerfile`:
|
||||
|
||||
```dockerfile
|
||||
# In every stage ARG CACHEBUST comes right after the steps that install
|
||||
# system packages and download Go's modules, or right after FROM in a
|
||||
# stage with neither, and every step that lints, tests or compiles
|
||||
# comes after it. script/ passes a new value on every build, which
|
||||
# reruns every RUN step after it. Docker keeps the layers above it
|
||||
# until the base image, their own lines or go.mod and go.sum change.
|
||||
# The steps that compile mount Go's build cache, and golangci-lint run
|
||||
# also mounts golangci-lint's; both survive from build to build.
|
||||
# Every stage declares ARG CACHEBUST first: script/ passes a new value
|
||||
# on every build, which reruns every RUN step after it. The steps that
|
||||
# compile mount Go's build cache, which survives from build to build.
|
||||
|
||||
# Lint phase
|
||||
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
|
||||
FROM golangci/golangci-lint@sha256:... AS lint
|
||||
ARG CACHEBUST
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
COPY . .
|
||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
--mount=type=cache,target=/root/.cache/golangci-lint \
|
||||
golangci-lint run --config .golangci.yml ./...
|
||||
|
||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||
@@ -229,10 +214,10 @@ style conventions are in separate documents:
|
||||
# cache also holds test results, which go test would otherwise replay.
|
||||
# golang:1.x, YYYY-MM-DD
|
||||
FROM golang@sha256:... AS test
|
||||
ARG CACHEBUST
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
COPY . .
|
||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
||||
@@ -240,19 +225,19 @@ style conventions are in separate documents:
|
||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
||||
|
||||
# Build stage. Nothing is wanted from either phase above; the copies
|
||||
# are what make BuildKit build them first, so nothing after them can
|
||||
# run unless lint and test passed.
|
||||
# are what make BuildKit build them first, so this stage cannot run
|
||||
# unless lint and test passed.
|
||||
# golang:1.x-alpine, YYYY-MM-DD
|
||||
FROM golang@sha256:... AS builder
|
||||
RUN apk add --no-cache git
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
COPY --from=lint /src/go.sum /dev/null
|
||||
COPY --from=test /src/go.sum /dev/null
|
||||
RUN apk add --no-cache git
|
||||
# A tar-stream context keeps the sender's file owners, which git refuses.
|
||||
RUN git config --system --add safe.directory /src
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY . .
|
||||
|
||||
# The VERSION build arg when one is given, otherwise
|
||||
@@ -289,26 +274,16 @@ style conventions are in separate documents:
|
||||
- Keep the runtime stage last, and if you add a stage after it, give it the
|
||||
same two copies. A plain `docker build .` builds the last stage's chain
|
||||
and nothing else.
|
||||
- Check builds keep these, and none of them can hand back a stale result:
|
||||
- The layers above `ARG CACHEBUST`: the system packages and Go's
|
||||
modules. Docker installs the packages again when the base image or the
|
||||
install line changes, and downloads the modules again when `go.mod` or
|
||||
`go.sum` changes. Go checks every module against `go.sum` as it
|
||||
downloads it, and no later build can change a kept layer. The modules
|
||||
are kept in a layer rather than a cache mount because a cache mount is
|
||||
shared by every build on the host, and Go builds from whatever files
|
||||
it holds without checking them again (only `go mod verify` does), so a
|
||||
file changed there would go unnoticed.
|
||||
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
|
||||
root), mounted on the steps that compile: `golangci-lint run`, both
|
||||
`go test` runs and `go build`. Go reuses a compiled package only when
|
||||
all its inputs are unchanged. The cache also holds test results, which
|
||||
is why both `go test` runs carry `-count=1`.
|
||||
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
|
||||
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
|
||||
a package's findings only when the package's files and those of every
|
||||
package it imports, `go.mod`, the linters and their settings in
|
||||
`.golangci.yml`, and the golangci-lint version are all unchanged.
|
||||
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
|
||||
it for root), and only on the steps that compile: `golangci-lint run`,
|
||||
both `go test` runs and `go build`. It keeps compiled packages between
|
||||
check builds, and Go reuses one only when all its inputs are unchanged, so
|
||||
it saves the compile without skipping a check. Nothing else is kept:
|
||||
golangci-lint's own cache would replay stored findings in place of a lint
|
||||
run, and Go builds from whatever files a kept module cache holds without
|
||||
checking them again (only `go mod verify` does), so a file changed there
|
||||
would go unnoticed. Go's build cache also holds test results, which is why
|
||||
both `go test` runs carry `-count=1`.
|
||||
- If the project uses `//go:embed` directives that reference build artifacts
|
||||
(e.g. a web frontend compiled in a separate stage), the lint phase must
|
||||
create placeholder files so the embed directives resolve. Example:
|
||||
@@ -317,20 +292,12 @@ style conventions are in separate documents:
|
||||
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
|
||||
has no `apk`, so install with `apt-get` under the Debian package name
|
||||
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
|
||||
lists in the same `RUN`, so the layer does not keep them. The test phase,
|
||||
on the Debian Go image, installs the libraries its tests need the same
|
||||
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
|
||||
Docker keeps it until the base image or the install line changes:
|
||||
lists in the same `RUN`, so the layer does not keep them:
|
||||
|
||||
```dockerfile
|
||||
FROM golangci/golangci-lint@sha256:... AS lint
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends libvips-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
ARG CACHEBUST
|
||||
```
|
||||
|
||||
- `.dockerignore` lets `.git` into the build context. It keeps out every git
|
||||
|
||||
@@ -30,46 +30,6 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-18 re-vendored from `sneak/prompts` commit `0653579` (closes #221):
|
||||
each `Dockerfile` stage installs its system packages and downloads the Go
|
||||
modules above `ARG CACHEBUST`, so check builds keep them, and the lint step
|
||||
keeps golangci-lint's cache.
|
||||
- 2026-10-19 a `trusted_proxies` entry in the IPv4-mapped form trusted nothing;
|
||||
it is now read as the IPv4 range it covers by the same code that reads
|
||||
`blocked_networks`, and such an entry shorter than `/96` aborts startup
|
||||
(closes #303).
|
||||
- 2026-10-18 the next reconciliation pass waits at least ten times as long as
|
||||
the last one took, so passes on a large cache no longer run back to back
|
||||
(closes #250).
|
||||
- 2026-10-18 a `blocked_networks` entry in the IPv4-mapped form refused nothing;
|
||||
`::ffff:10.0.0.0/104` is now read as `10.0.0.0/8`, and such an entry shorter
|
||||
than `/96` aborts startup (closes #301).
|
||||
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||
addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it
|
||||
refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible
|
||||
form (`::/96`) when `blocked_networks` contains either the address as written
|
||||
or the IPv4 address it holds (closes #299). A source host such as
|
||||
`[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that
|
||||
range reaches the IPv4 address it holds, here `10.0.0.1`; and with
|
||||
`203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but
|
||||
`[::203.0.113.5]` was fetched. Both now answer 403. The mapped form
|
||||
(`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go
|
||||
keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must
|
||||
not refuse every IPv4 address.
|
||||
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
|
||||
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
|
||||
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
|
||||
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
|
||||
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
|
||||
fetched, which on a network that routes it reaches the internal IPv4 host; it
|
||||
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
|
||||
as a whole, as one of the built-in blocked ranges, and still is.
|
||||
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
|
||||
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
|
||||
and through an encrypted URL (closes #295). The fetcher's check of the source
|
||||
URL kept the brackets when no port followed them, could not look the host up,
|
||||
and the request answered 500. It now checks the address without brackets, so a
|
||||
private one is refused with 403, as with a port.
|
||||
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
|
||||
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
|
||||
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
|
||||
@@ -78,25 +38,6 @@ P2: security: per-origin rate limiting
|
||||
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
|
||||
with `//` it starts `file:////`, so SQLite no longer reads the first name in
|
||||
the path as a host.
|
||||
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
|
||||
proportions at 1 or more (closes #283): it was rounded down, so for a thin
|
||||
source it came out at 0, and libvips then made the size square. An 8192x1
|
||||
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
|
||||
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
|
||||
`fit=contain` and `fit=inside` work both sides out again from the smaller of
|
||||
the two scales and `fit=outside` from the larger, and each keeps both sides at
|
||||
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
|
||||
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
|
||||
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
|
||||
process running it alone peaked at about 34 MiB resident. With `fit=outside`
|
||||
the image is scaled to cover the size, so that source at `100x0` comes out at
|
||||
8192x1. `README.md` now says this under Signature Specification, and says how
|
||||
a `0` side is worked out where it said `0` meant the original.
|
||||
- 2026-10-16 the generator page lower-cases the source host and writes an empty
|
||||
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
|
||||
(closes #293). A host written in upper and in lower case, or a source URL with
|
||||
no path and with the path `/`, made two cache keys and two upstream requests
|
||||
for one image.
|
||||
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
|
||||
source host, path and query, not joined as text (closes #288). The source
|
||||
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
|
||||
|
||||
@@ -88,56 +88,3 @@ func TestBlockedNetworksInvalidAbortsStartup(t *testing.T) {
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// TestBlockedNetworksMappedEntryIsIPv4Range checks that an entry written in
|
||||
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
|
||||
// since the fetcher matches an IPv4-mapped address as the IPv4 address it
|
||||
// holds. /96, the shortest such entry, covers every IPv4 address.
|
||||
func TestBlockedNetworksMappedEntryIsIPv4Range(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
yamlContent := signingKeyLine + `blocked_networks:
|
||||
- "::ffff:10.0.0.0/104"
|
||||
- "::ffff:0:0/96"
|
||||
`
|
||||
|
||||
c, err := configFromYAML(t, yamlContent)
|
||||
if err != nil {
|
||||
t.Fatalf("IPv4-mapped blocked_networks entries should load, got error: %v", err)
|
||||
}
|
||||
|
||||
want := []string{"10.0.0.0/8", "0.0.0.0/0"}
|
||||
if len(c.BlockedNetworks) != len(want) {
|
||||
t.Fatalf("BlockedNetworks = %v, want %d entries", c.BlockedNetworks, len(want))
|
||||
}
|
||||
|
||||
for i, w := range want {
|
||||
if got := c.BlockedNetworks[i].String(); got != w {
|
||||
t.Errorf("BlockedNetworks[%d] = %q, want %q", i, got, w)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestBlockedNetworksMappedEntryShorterThan96AbortsStartup checks that an
|
||||
// entry in the IPv4-mapped form shorter than /96, which covers more than the
|
||||
// IPv4 addresses, aborts startup with an error naming the key and the value.
|
||||
func TestBlockedNetworksMappedEntryShorterThan96AbortsStartup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runAbortCases(t, []abortCase{
|
||||
{
|
||||
name: "mapped-shorter-than-96",
|
||||
yaml: signingKeyLine + `blocked_networks:
|
||||
- "::ffff:0.0.0.0/80"
|
||||
`,
|
||||
wantErrSubstrings: []string{keyBlockedNetworks, "::ffff:0.0.0.0/80"},
|
||||
},
|
||||
{
|
||||
name: "mapped-95",
|
||||
yaml: signingKeyLine + `blocked_networks:
|
||||
- "::ffff:0:0/95"
|
||||
`,
|
||||
wantErrSubstrings: []string{keyBlockedNetworks},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -91,7 +91,6 @@ var (
|
||||
errNotABoolean = errors.New("not a boolean")
|
||||
errNotAStringList = errors.New("not a list of strings")
|
||||
errNotAValidCIDR = errors.New("not a valid CIDR network")
|
||||
errMappedTooShort = errors.New("in the IPv4-mapped form but shorter than /96")
|
||||
errNotAMetricsMap = errors.New("not a map of metrics settings")
|
||||
errEmptyListEntry = errors.New("list contains an empty entry")
|
||||
errEmptyEntry = errors.New("contains an empty entry")
|
||||
@@ -1249,13 +1248,6 @@ func defaultTrustedProxies() []netip.Prefix {
|
||||
// wrong type, an empty entry, a non-string entry, or an unparseable CIDR
|
||||
// aborts startup naming the key and the offending value; the default
|
||||
// (an empty list) applies only to an omitted key.
|
||||
//
|
||||
// An entry written in the IPv4-mapped form (::ffff:a.b.c.d) is kept as the
|
||||
// IPv4 range it covers: ::ffff:10.0.0.0/104 is kept as 10.0.0.0/8. Both
|
||||
// blocked_networks and trusted_proxies are matched against an IPv4-mapped
|
||||
// address as the IPv4 address it holds, so the entry as written would match
|
||||
// nothing. Such an entry shorter than /96 covers more than the IPv4
|
||||
// addresses and aborts startup.
|
||||
func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
|
||||
raw, ok := lookupValue(sc, key)
|
||||
if !ok {
|
||||
@@ -1271,10 +1263,6 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// An IPv4-mapped address is the 96 bits of ::ffff:0:0/96 followed by
|
||||
// the 32 bits of the IPv4 address it holds.
|
||||
const mappedPrefixBits = 96
|
||||
|
||||
prefixes := make([]netip.Prefix, 0, len(entries))
|
||||
|
||||
for _, entry := range entries {
|
||||
@@ -1284,16 +1272,6 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
|
||||
settingName(key), entry, errNotAValidCIDR)
|
||||
}
|
||||
|
||||
if prefix.Addr().Is4In6() && prefix.Bits() < mappedPrefixBits {
|
||||
return nil, fmt.Errorf("%s: value %q is %w",
|
||||
settingName(key), entry, errMappedTooShort)
|
||||
}
|
||||
|
||||
if prefix.Addr().Is4In6() {
|
||||
prefix = netip.PrefixFrom(prefix.Addr().Unmap(),
|
||||
prefix.Bits()-mappedPrefixBits)
|
||||
}
|
||||
|
||||
prefixes = append(prefixes, prefix)
|
||||
}
|
||||
|
||||
|
||||
@@ -83,41 +83,3 @@ func TestTrustedProxiesInvalidAbortsStartup(t *testing.T) {
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// TestTrustedProxiesMappedEntryIsIPv4Range checks that an entry written in
|
||||
// the IPv4-mapped form (::ffff:a.b.c.d) is kept as the IPv4 range it covers,
|
||||
// since the client address is matched as the IPv4 address it holds. /96, the
|
||||
// shortest such entry, covers every IPv4 address.
|
||||
func TestTrustedProxiesMappedEntryIsIPv4Range(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
c, err := configFromYAML(t, signingKeyLine+
|
||||
`trusted_proxies: ["::ffff:10.0.0.0/104", "::ffff:0:0/96"]`+"\n")
|
||||
if err != nil {
|
||||
t.Fatalf("IPv4-mapped trusted_proxies entries should load: %v", err)
|
||||
}
|
||||
|
||||
got := make([]string, len(c.TrustedProxies))
|
||||
for i, p := range c.TrustedProxies {
|
||||
got[i] = p.String()
|
||||
}
|
||||
|
||||
if joined := strings.Join(got, ","); joined != "10.0.0.0/8,0.0.0.0/0" {
|
||||
t.Errorf("TrustedProxies = %v, want 10.0.0.0/8 and 0.0.0.0/0", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTrustedProxiesMappedEntryShorterThan96AbortsStartup checks that an
|
||||
// entry in the IPv4-mapped form shorter than /96, which covers more than the
|
||||
// IPv4 addresses, aborts startup with an error naming the key and the value.
|
||||
func TestTrustedProxiesMappedEntryShorterThan96AbortsStartup(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
runAbortCases(t, []abortCase{
|
||||
{
|
||||
name: "mapped-95",
|
||||
yaml: signingKeyLine + `trusted_proxies: ["::ffff:0:0/95"]` + "\n",
|
||||
wantErrSubstrings: []string{keyTrustedProxies, "::ffff:0:0/95"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -127,7 +127,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
sourceURL := r.FormValue("url")
|
||||
|
||||
// Validate source URL
|
||||
parsed, err := parseSourceURL(sourceURL)
|
||||
parsed, err := url.Parse(sourceURL)
|
||||
if err != nil || parsed.Host == "" {
|
||||
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
|
||||
http.StatusBadRequest)
|
||||
@@ -198,25 +198,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// parseSourceURL parses the source URL entered on the generator page with its
|
||||
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
|
||||
// source URL, so that each spelling of one source URL makes one cache key and
|
||||
// one upstream request.
|
||||
func parseSourceURL(sourceURL string) (*url.URL, error) {
|
||||
parsed, err := url.Parse(sourceURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
parsed.Host = strings.ToLower(parsed.Host)
|
||||
|
||||
if parsed.Path == "" {
|
||||
parsed.Path = "/"
|
||||
}
|
||||
|
||||
return parsed, nil
|
||||
}
|
||||
|
||||
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
|
||||
// image URL made from the source URL source, is source as it was written,
|
||||
// apart from the scheme, which the token does not keep. It is not when source
|
||||
|
||||
@@ -59,17 +59,15 @@ const (
|
||||
// builtinBlockedPrefixes are internal or special-use ranges that Go's
|
||||
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
|
||||
// already cover. They are always blocked, in addition to any
|
||||
// operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6
|
||||
// addresses are read as the IPv4 address they hold before matching, so
|
||||
// these IPv4 ranges are caught in all three forms.
|
||||
// operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped
|
||||
// before matching, so these IPv4 ranges are caught in both forms.
|
||||
//
|
||||
//nolint:gochecknoglobals // immutable built-in blocklist
|
||||
var builtinBlockedPrefixes = []netip.Prefix{
|
||||
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
||||
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
||||
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
||||
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
||||
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
|
||||
netip.MustParsePrefix("100.64.0.0/10"), // RFC 6598 CGNAT / carrier-grade NAT
|
||||
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
|
||||
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
|
||||
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
|
||||
}
|
||||
|
||||
// Fetcher errors.
|
||||
@@ -475,18 +473,18 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
|
||||
return ErrUnsupportedScheme
|
||||
}
|
||||
|
||||
parsedURL, err := neturl.Parse(rawURL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
|
||||
}
|
||||
|
||||
// Hostname drops any port, and the brackets of an IPv6 address whether
|
||||
// or not a port follows them
|
||||
host := parsedURL.Hostname()
|
||||
// Parse to extract host
|
||||
host := extractHost(rawURL)
|
||||
if host == "" {
|
||||
return ErrInvalidHost
|
||||
}
|
||||
|
||||
// Remove port if present
|
||||
h, _, err := net.SplitHostPort(host)
|
||||
if err == nil {
|
||||
host = h
|
||||
}
|
||||
|
||||
// Block obvious localhost patterns
|
||||
if isLocalhost(host) {
|
||||
return ErrSSRFBlocked
|
||||
@@ -545,9 +543,6 @@ func isPrivateIP(ip net.IP) bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
|
||||
ip = ipv4CompatibleToIPv4(ip)
|
||||
|
||||
// Check for loopback
|
||||
if ip.IsLoopback() {
|
||||
return true
|
||||
@@ -599,12 +594,7 @@ func isPrivateIP(ip net.IP) bool {
|
||||
}
|
||||
|
||||
// isBlockedIP reports whether ip is refused, either by the built-in
|
||||
// internal-range check or by one of the operator-supplied prefixes. An
|
||||
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
|
||||
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
|
||||
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
|
||||
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
|
||||
// not refuse every IPv4 address.
|
||||
// internal-range check or by one of the operator-supplied prefixes.
|
||||
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
||||
if isPrivateIP(ip) {
|
||||
return true
|
||||
@@ -617,28 +607,11 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
|
||||
|
||||
addr = addr.Unmap()
|
||||
|
||||
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
|
||||
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
|
||||
// addr refuses more only for an IPv4-compatible address.
|
||||
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
|
||||
|
||||
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
|
||||
return prefix.Contains(addr) || prefix.Contains(asIPv4)
|
||||
return prefix.Contains(addr)
|
||||
})
|
||||
}
|
||||
|
||||
// ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
|
||||
// address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
|
||||
// any other address unchanged.
|
||||
func ipv4CompatibleToIPv4(ip net.IP) net.IP {
|
||||
addr, ok := netip.AddrFromSlice(ip)
|
||||
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
|
||||
return ip[12:]
|
||||
}
|
||||
|
||||
return ip
|
||||
}
|
||||
|
||||
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges
|
||||
// before connecting. New wraps dialSSRFSafe with the operator-supplied
|
||||
// blocklist; this entry point enforces the built-in ranges alone.
|
||||
|
||||
@@ -186,44 +186,6 @@ func TestIsPrivateIP(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
|
||||
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
|
||||
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
|
||||
// link-local IPv4 address is refused as that address is, and one holding a
|
||||
// public IPv4 address in the IPv4-compatible form passes.
|
||||
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
ip string
|
||||
want bool
|
||||
}{
|
||||
{"::127.0.0.1", true}, // loopback
|
||||
{"::10.0.0.1", true}, // private
|
||||
{"::169.254.169.254", true}, // link-local
|
||||
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
|
||||
{"::8.8.8.8", false}, // public
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.ip, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ip := net.ParseIP(tc.ip)
|
||||
if ip == nil {
|
||||
t.Fatalf("failed to parse IP %q", tc.ip)
|
||||
}
|
||||
|
||||
got := isPrivateIP(ip)
|
||||
t.Logf("isPrivateIP(%s) = %v", ip, got)
|
||||
|
||||
if got != tc.want {
|
||||
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -50,69 +50,3 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
|
||||
// one side 0, the side computed from the input's proportions is 1 where it
|
||||
// would round down to 0, and that Process makes that size in every fit mode
|
||||
// but FitOutside, which scales the input to cover the size and so can make
|
||||
// it longer. Each case checks the size targetDimensions gives before it
|
||||
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
|
||||
// libvips ask for about 39 GB with FitCover.
|
||||
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
|
||||
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
|
||||
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
|
||||
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
|
||||
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
|
||||
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
|
||||
}
|
||||
|
||||
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
|
||||
|
||||
for _, tt := range tests {
|
||||
for _, fit := range fits {
|
||||
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
width, height, err := targetDimensions(tt.size,
|
||||
tt.inputWidth, tt.inputHeight)
|
||||
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
|
||||
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
|
||||
width, height, err, tt.wantWidth, tt.wantHeight)
|
||||
}
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
|
||||
if fit == FitOutside {
|
||||
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
|
||||
}
|
||||
|
||||
if !ok {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
|
||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
|
||||
FitOutside)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,45 +60,3 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
|
||||
// keeps each side at 1 or more when it works both sides out again from the
|
||||
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
|
||||
// point is just under 1 and so rounded down to 0, a width libvips refuses.
|
||||
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
inputWidth, inputHeight int
|
||||
size Size
|
||||
wantWidth, wantHeight int
|
||||
}{
|
||||
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
|
||||
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
|
||||
|
||||
result, err := New(Params{}).Process(t.Context(),
|
||||
bytes.NewReader(input),
|
||||
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
|
||||
if err != nil {
|
||||
t.Fatalf("Process() error = %v", err)
|
||||
}
|
||||
|
||||
defer func() { _ = result.Content.Close() }()
|
||||
|
||||
t.Logf("Process() made %dx%d", result.Width, result.Height)
|
||||
|
||||
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
|
||||
t.Errorf("Process() made %dx%d, want %dx%d",
|
||||
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -311,10 +311,8 @@ func (p *ImageProcessor) Process(
|
||||
|
||||
// targetDimensions calculates the output dimensions for a requested size,
|
||||
// scaling proportionally when only one dimension is given and keeping the
|
||||
// original dimensions when both are zero. A side scaled proportionally is
|
||||
// rounded down but kept at 1 or more, as for a thin image it would round
|
||||
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
|
||||
// which is ErrOutputTooLarge.
|
||||
// original dimensions when both are zero. A side scaled proportionally can
|
||||
// come out above MaxOutputDimension, which is ErrOutputTooLarge.
|
||||
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||
width, height := size.Width, size.Height
|
||||
|
||||
@@ -324,10 +322,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
|
||||
return origWidth, origHeight, nil
|
||||
case width == 0:
|
||||
// Only height specified: calculate width proportionally
|
||||
width = max(1, origWidth*height/origHeight)
|
||||
width = origWidth * height / origHeight
|
||||
case height == 0:
|
||||
// Only width specified: calculate height proportionally
|
||||
height = max(1, origHeight*width/origWidth)
|
||||
height = origHeight * width / origWidth
|
||||
}
|
||||
|
||||
if width > MaxOutputDimension || height > MaxOutputDimension {
|
||||
@@ -481,16 +479,13 @@ func (p *ImageProcessor) resize(
|
||||
return cropCentreAndResize(img, width, height)
|
||||
|
||||
case FitContain:
|
||||
// Resize to fit within dimensions, maintaining aspect ratio. Scaled
|
||||
// by the smaller scale, the short side of a thin image can round
|
||||
// down to 0, which libvips refuses as a width, so each side is at
|
||||
// least 1.
|
||||
// Resize to fit within dimensions, maintaining aspect ratio
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
@@ -508,22 +503,19 @@ func (p *ImageProcessor) resize(
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
case FitOutside:
|
||||
// Resize so smallest dimension fits, may exceed target on other
|
||||
// dimension. A side scaled to 1 can come out just under 1 in
|
||||
// floating point, as 49 times 1/49 does, and round down to 0, which
|
||||
// libvips refuses as a width, so each side is at least 1.
|
||||
// Resize so smallest dimension fits, may exceed target on other dimension
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := max(scaleW, scaleH)
|
||||
newW := max(1, int(float64(imgW)*scale))
|
||||
newH := max(1, int(float64(imgH)*scale))
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
// The dimension that exceeds the target is held to the maximum too
|
||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
||||
|
||||
@@ -18,14 +18,6 @@ import (
|
||||
// write-pressure wakeups triggered by stores.
|
||||
const DefaultEvictionInterval = 5 * time.Minute
|
||||
|
||||
// reconciliationWaitFactor is how many times as long as a
|
||||
// reconciliation pass took the eviction loop waits at least, after that
|
||||
// pass ends, before it starts the next one. A pass reads a row and stats a
|
||||
// file for every cached file, so its time grows with the cache; waiting
|
||||
// ten times as long keeps reconciliation to at most a tenth of the time,
|
||||
// however large the cache is.
|
||||
const reconciliationWaitFactor = 10
|
||||
|
||||
// evictionBatchSize is how many LRU candidates of each class (variants
|
||||
// and source blobs) one eviction pass fetches from the database.
|
||||
const evictionBatchSize = 100
|
||||
@@ -446,12 +438,10 @@ func (c *Cache) notifyWritePressure() {
|
||||
}
|
||||
|
||||
// StartEviction launches the background eviction goroutine, which
|
||||
// reconciles the database accounting with the cache directories and
|
||||
// removes the expired negative cache rows at startup and again on every
|
||||
// periodic tick thereafter, reconciles the database accounting with the
|
||||
// cache directories at startup and again on the first periodic tick
|
||||
// that comes at least ten times as long after a reconciliation pass
|
||||
// ended as that pass took, and evicts to the configured limit on the given
|
||||
// periodic interval and on write-pressure notifications. On a
|
||||
// periodic tick thereafter, and evicts to the configured limit on the
|
||||
// given periodic interval and on write-pressure notifications. On a
|
||||
// disabled cache it only removes the expired negative cache rows, as
|
||||
// the negative cache is used either way. It is a no-op when already
|
||||
// started. The goroutine outlives the caller, so it runs with its own
|
||||
@@ -492,7 +482,7 @@ func (c *Cache) StopEviction(ctx context.Context) error {
|
||||
func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
||||
defer close(c.evictionDone)
|
||||
|
||||
nextReconciliation := c.runReconciliationPass(ctx)
|
||||
c.runReconciliationPass(ctx)
|
||||
c.runCleanExpired(ctx)
|
||||
c.runEvictionPass(ctx)
|
||||
|
||||
@@ -504,20 +494,16 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
// Reconciliation walks the cache directories and reads a
|
||||
// row for every file, so it runs only on the periodic
|
||||
// ticker rather than on every write-pressure wakeup, and
|
||||
// only on the first tick at or after nextReconciliation,
|
||||
// which is ten times as long after the last pass ended as
|
||||
// that pass took. Passes then never run back to back,
|
||||
// however long one takes. A file whose best-effort
|
||||
// accounting insert failed in a store stays unaccounted for
|
||||
// until the next pass, up to about one eviction interval
|
||||
// plus ten times the last pass's time.
|
||||
if !time.Now().Before(nextReconciliation) {
|
||||
nextReconciliation = c.runReconciliationPass(ctx)
|
||||
}
|
||||
|
||||
// Reconciliation walks the cache directories, so it only
|
||||
// runs on the periodic ticker rather than on every
|
||||
// write-pressure wakeup, keeping it off the per-store hot
|
||||
// path. Reusing the eviction interval itself (rather than a
|
||||
// separate, longer one) is a deliberate choice: it is the
|
||||
// simplest option that still bounds how long a store's
|
||||
// best-effort accounting insert can stay silently
|
||||
// unaccounted for to one interval, on a process that is
|
||||
// already running this loop regardless.
|
||||
c.runReconciliationPass(ctx)
|
||||
c.runCleanExpired(ctx)
|
||||
case <-c.evictionPressure:
|
||||
}
|
||||
@@ -541,25 +527,17 @@ func (c *Cache) runEvictionPass(ctx context.Context) {
|
||||
}
|
||||
|
||||
// runReconciliationPass runs one reconciliation pass, logging failures
|
||||
// instead of propagating them (the loop must keep running), and returns
|
||||
// the earliest time the next pass may start: reconciliationWaitFactor
|
||||
// times as long after this pass ended as it took. It does nothing once
|
||||
// ctx is cancelled.
|
||||
func (c *Cache) runReconciliationPass(ctx context.Context) time.Time {
|
||||
// instead of propagating them (the loop must keep running). It does
|
||||
// nothing once ctx is cancelled.
|
||||
func (c *Cache) runReconciliationPass(ctx context.Context) {
|
||||
if ctx.Err() != nil {
|
||||
return time.Now()
|
||||
return
|
||||
}
|
||||
|
||||
started := time.Now()
|
||||
|
||||
err := c.reconcileAccounting(ctx)
|
||||
if err != nil {
|
||||
c.log.Warn("cache accounting reconciliation failed", "error", err)
|
||||
}
|
||||
|
||||
ended := time.Now()
|
||||
|
||||
return ended.Add(reconciliationWaitFactor * ended.Sub(started))
|
||||
}
|
||||
|
||||
// runCleanExpired runs CleanExpired, logging a failure instead of
|
||||
@@ -578,8 +556,7 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
|
||||
// reconcileAccounting synchronizes the database size accounting with
|
||||
// the actual contents of the cache directories. It runs at startup and
|
||||
// then on the first periodic eviction tick that comes at least ten times
|
||||
// as long after the last pass ended as that pass took, off the request
|
||||
// again on every periodic eviction tick thereafter, off the request
|
||||
// hot path: it adopts variant files that predate the accounting table
|
||||
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
|
||||
// insert under transient DB contention), drops accounting rows whose
|
||||
@@ -589,10 +566,9 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
|
||||
// files left behind by crashed writes, and last checks the total cache
|
||||
// usage against the tables. It reads the tables a page at a time.
|
||||
// Running it periodically, not just once, bounds how long such drift can
|
||||
// accumulate unaccounted for on a long-running process to about one
|
||||
// eviction interval plus ten times the last pass's time.
|
||||
// Once ctx is cancelled, it stops at the next file or row and returns
|
||||
// ctx's error.
|
||||
// accumulate unaccounted for on a long-running process to one eviction
|
||||
// interval. Once ctx is cancelled, it stops at the next file or row and
|
||||
// returns ctx's error.
|
||||
func (c *Cache) reconcileAccounting(ctx context.Context) error {
|
||||
if c.disabled {
|
||||
return nil
|
||||
|
||||
@@ -918,89 +918,6 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestReconciliationPassesAreSpacedByTenTimesTheirTime makes the hook
|
||||
// called after every read of the reconciliation pass sleep 10 ms, runs
|
||||
// the eviction loop on a 10 ms interval, and records when each of those
|
||||
// calls starts and ends. A pass starts before its first call and ends
|
||||
// after its last, so the time from the first call's start to the last
|
||||
// call's end is at most the pass's time. The next pass's first call must
|
||||
// start at least ten times that long after the last call ended.
|
||||
func TestReconciliationPassesAreSpacedByTenTimesTheirTime(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cache, _ := newEvictionTestCache(t, 1<<30)
|
||||
|
||||
// A pass made here, before the loop starts, counts the reads one
|
||||
// pass makes on this empty cache, so the reads recorded below can be
|
||||
// split into passes.
|
||||
readsPerPass := 0
|
||||
cache.reconciliationReadTestHook = func(int) { readsPerPass++ }
|
||||
|
||||
err := cache.reconcileAccounting(t.Context())
|
||||
if err != nil {
|
||||
t.Fatalf("reconcileAccounting failed: %v", err)
|
||||
}
|
||||
|
||||
t.Logf("one reconciliation pass makes %d reads", readsPerPass)
|
||||
|
||||
if readsPerPass == 0 {
|
||||
t.Fatal("a reconciliation pass made no read")
|
||||
}
|
||||
|
||||
// The pass at startup and two after it.
|
||||
const passes = 3
|
||||
|
||||
type read struct{ start, end time.Time }
|
||||
|
||||
// The channel holds every read the test looks at, so none of those
|
||||
// is dropped; reads after them are, and never block the loop.
|
||||
reads := make(chan read, passes*readsPerPass)
|
||||
|
||||
cache.reconciliationReadTestHook = func(int) {
|
||||
start := time.Now()
|
||||
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
|
||||
select {
|
||||
case reads <- read{start: start, end: time.Now()}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
cache.StartEviction(10 * time.Millisecond)
|
||||
defer func() { _ = cache.StopEviction(t.Context()) }()
|
||||
|
||||
recorded := make([]read, 0, passes*readsPerPass)
|
||||
timeout := time.After(15 * time.Second)
|
||||
|
||||
for len(recorded) < passes*readsPerPass {
|
||||
select {
|
||||
case r := <-reads:
|
||||
recorded = append(recorded, r)
|
||||
case <-timeout:
|
||||
t.Fatalf("saw %d reconciliation reads in 15 s, want %d (%d passes)",
|
||||
len(recorded), passes*readsPerPass, passes)
|
||||
}
|
||||
}
|
||||
|
||||
for pass := range passes - 1 {
|
||||
first := recorded[pass*readsPerPass]
|
||||
last := recorded[(pass+1)*readsPerPass-1]
|
||||
next := recorded[(pass+1)*readsPerPass]
|
||||
|
||||
passTime := last.end.Sub(first.start)
|
||||
wait := next.start.Sub(last.end)
|
||||
|
||||
t.Logf("pass %d took at least %v, and the next pass started %v after it",
|
||||
pass+1, passTime, wait)
|
||||
|
||||
if wait < 10*passTime {
|
||||
t.Errorf("pass %d took at least %v, and the next pass started %v "+
|
||||
"after it, want at least %v", pass+1, passTime, wait, 10*passTime)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestStopEvictionInterruptsPassInProgress holds the test database's
|
||||
// only connection, so the startup reconciliation pass waits for it, and
|
||||
// checks that StopEviction stops that pass instead of waiting for the
|
||||
|
||||
@@ -29,9 +29,7 @@ var (
|
||||
const MaxDimension = imageprocessor.MaxOutputDimension
|
||||
|
||||
// ValidateDimension checks one requested width or height; name ("width" or
|
||||
// "height") appears in the error. 0 is valid: that side is then worked out from
|
||||
// the source's proportions and the other requested side, rounded down but at
|
||||
// least 1, or is the source's own side when the other side is 0 too.
|
||||
// "height") appears in the error. 0 means "original size" and is valid.
|
||||
func ValidateDimension(name string, value int) error {
|
||||
if value < 0 {
|
||||
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
|
||||
|
||||
@@ -14,12 +14,10 @@ import (
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/fx"
|
||||
"go.uber.org/fx/fxtest"
|
||||
@@ -185,57 +183,6 @@ func startImageProxy(
|
||||
return s, db.DB(), stateDir
|
||||
}
|
||||
|
||||
// startImageProxyWithBlockedNetworks starts the components pixad's fx app
|
||||
// builds, from a config with a fresh state directory and blockedNetworks as
|
||||
// blocked_networks. Unlike startImageProxy it provides no upstream fetcher, so
|
||||
// the handlers build the one pixad uses, which checks every address it
|
||||
// connects to against the built-in blocked ranges and blocked_networks. It
|
||||
// returns the server with its routes.
|
||||
func startImageProxyWithBlockedNetworks(
|
||||
t *testing.T, blockedNetworks []netip.Prefix,
|
||||
) *Server {
|
||||
t.Helper()
|
||||
|
||||
stateDir := t.TempDir()
|
||||
cfg := &config.Config{
|
||||
SigningKey: testSigningKey,
|
||||
StateDir: stateDir,
|
||||
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
|
||||
BlockedNetworks: blockedNetworks,
|
||||
// A fetch that gets past the checks and finds nothing at the address
|
||||
// fails within this time at the latest.
|
||||
UpstreamFetchTimeout: 2 * time.Second,
|
||||
// With no connection slots, the fetch would fail before connecting.
|
||||
UpstreamConnections: config.DefaultUpstreamConnections,
|
||||
DownstreamTimeout: config.DefaultDownstreamTimeout,
|
||||
}
|
||||
|
||||
var (
|
||||
h *handlers.Handlers
|
||||
mw *middleware.Middleware
|
||||
)
|
||||
|
||||
app := fxtest.New(t,
|
||||
fx.Supply(cfg),
|
||||
fx.Provide(
|
||||
globals.New,
|
||||
logger.New,
|
||||
database.New,
|
||||
healthcheck.New,
|
||||
handlers.New,
|
||||
middleware.New,
|
||||
),
|
||||
fx.Populate(&h, &mw),
|
||||
)
|
||||
app.RequireStart()
|
||||
t.Cleanup(app.RequireStop)
|
||||
|
||||
s := &Server{config: cfg, mw: mw, h: h}
|
||||
s.SetupRoutes()
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// getImage sends a GET for target to s and fails unless it answers 200.
|
||||
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
|
||||
@@ -3,20 +3,15 @@ package server
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
"sneak.berlin/go/pixa/internal/session"
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// percentSignPath is a source path holding a "%", which a URL writes %25.
|
||||
@@ -313,70 +308,6 @@ func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
|
||||
// spellings of one source URL on the generator page, requests the encrypted
|
||||
// URL made for each, and checks that the upstream receives one request and
|
||||
// that the second encrypted URL is served from the cache. One pair writes the
|
||||
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
|
||||
// an IPv6 address, whose hex digits have letter case, because the fetcher
|
||||
// would look a host name up in DNS. It has a port, as the fetcher refuses an
|
||||
// IPv6 address written without one. The other pair is a source URL with no
|
||||
// path and with the path "/", both of which fetch "/".
|
||||
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
// sourceURLs are the two spellings, after "https://"
|
||||
sourceURLs []string
|
||||
// wantRequest is the path the upstream must receive
|
||||
wantRequest string
|
||||
}{
|
||||
{"host in upper and in lower case",
|
||||
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
|
||||
"/a.png"},
|
||||
{"no path and the path written as a slash",
|
||||
[]string{upstreamHost, upstreamHost + "/"}, "/"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
var imageURLs []string
|
||||
|
||||
for _, sourceURL := range tt.sourceURLs {
|
||||
rec := postGenerate(t, s, "https://"+sourceURL)
|
||||
|
||||
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
|
||||
if rec.Code != http.StatusOK || match == nil {
|
||||
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
|
||||
rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
imageURLs = append(imageURLs, match[1])
|
||||
}
|
||||
|
||||
getImage(t, s, imageURLs[0])
|
||||
|
||||
second := getImage(t, s, imageURLs[1])
|
||||
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
|
||||
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, []string{tt.wantRequest}) {
|
||||
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
|
||||
// generator page must make no encrypted URL for, and checks that it answers
|
||||
// 400 and shows why. pixa keeps the source path decoded and fetches it
|
||||
@@ -438,228 +369,3 @@ func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// ipv6SourcePath is the source path of the image URLs signedImageURL and
|
||||
// encryptedImageURL make.
|
||||
const ipv6SourcePath = "/ipv6.png"
|
||||
|
||||
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
|
||||
// on an IPv6 source host, written with and without a port, through a
|
||||
// /v1/image/ URL with a signature and through an encrypted URL. The public
|
||||
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
|
||||
// as public, answers 200, and the upstream receives the request for
|
||||
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
|
||||
// receives no request: the test's fetcher connects every address to the test
|
||||
// upstream, so only the fetcher's check of the source URL can refuse it.
|
||||
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourceHost string
|
||||
wantStatus int
|
||||
// wantRequests are the request targets the upstream must receive
|
||||
wantRequests []string
|
||||
}{
|
||||
{"public address without a port", "[2001:db8::a]",
|
||||
http.StatusOK, []string{ipv6SourcePath}},
|
||||
{"public address with a port", "[2001:db8::a]:8443",
|
||||
http.StatusOK, []string{ipv6SourcePath}},
|
||||
{"private address without a port", "[fd00::1]",
|
||||
http.StatusForbidden, nil},
|
||||
{"private address with a port", "[fd00::1]:8443",
|
||||
http.StatusForbidden, nil},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
imageURLs := map[string]string{
|
||||
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||
}
|
||||
|
||||
for route, imageURL := range imageURLs {
|
||||
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||
|
||||
if rec.Code != tt.wantStatus {
|
||||
t.Errorf("status = %d, want %d; body %s",
|
||||
rec.Code, tt.wantStatus, rec.Body.String())
|
||||
}
|
||||
|
||||
got := upstream.requestTargets()
|
||||
t.Logf("upstream received %q", got)
|
||||
|
||||
if !slices.Equal(got, tt.wantRequests) {
|
||||
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
|
||||
// its original size as PNG, with a signature made with the test signing key,
|
||||
// which a host not on the allowlist needs. The URL has no q or fit, so it is
|
||||
// signed with the quality 85 and the fit cover.
|
||||
func signedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Helper()
|
||||
|
||||
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
|
||||
&signature.Request{
|
||||
SourceHost: sourceHost,
|
||||
SourcePath: ipv6SourcePath,
|
||||
Format: string(imgcache.FormatPNG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
}, time.Hour)
|
||||
|
||||
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
|
||||
}
|
||||
|
||||
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
|
||||
// at its original size as PNG, made with the test signing key.
|
||||
func encryptedImageURL(t *testing.T, sourceHost string) string {
|
||||
t.Helper()
|
||||
|
||||
generator, err := encurl.NewGenerator(testSigningKey)
|
||||
if err != nil {
|
||||
t.Fatalf("encurl.NewGenerator: %v", err)
|
||||
}
|
||||
|
||||
token, err := generator.Generate(&encurl.Payload{
|
||||
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
|
||||
Format: imgcache.FormatPNG,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("encurl Generate: %v", err)
|
||||
}
|
||||
|
||||
return "/v1/e/" + token + "/img.png"
|
||||
}
|
||||
|
||||
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
|
||||
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
|
||||
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
|
||||
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
|
||||
// through an encrypted URL. Each answers 403, and the upstream receives no
|
||||
// request: the test's fetcher connects every address to the test upstream, so
|
||||
// only the fetcher's check of the source URL can refuse it.
|
||||
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
upstream := startRecordingUpstream(t)
|
||||
s, _, _ := startImageProxy(t, upstream.Server)
|
||||
|
||||
sourceHosts := []string{
|
||||
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
|
||||
"[::10.0.0.1]", // IPv4-compatible form of a private address
|
||||
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
|
||||
}
|
||||
|
||||
for _, sourceHost := range sourceHosts {
|
||||
imageURLs := []string{
|
||||
signedImageURL(t, sourceHost),
|
||||
encryptedImageURL(t, sourceHost),
|
||||
}
|
||||
|
||||
for _, imageURL := range imageURLs {
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
|
||||
|
||||
if rec.Code != http.StatusForbidden {
|
||||
t.Errorf("source host %s: status = %d, want %d; body %s",
|
||||
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if got := upstream.requestTargets(); len(got) != 0 {
|
||||
t.Errorf("upstream received %q, want no request", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageRoutesCheckIPv4AddressInIPv6SourceHost requests ipv6SourcePath on
|
||||
// source hosts that hold an IPv4 address, through a /v1/image/ URL with a
|
||||
// signature and through an encrypted URL, with the fetcher pixad uses.
|
||||
// [64:ff9b:1::a00:1], in the local-use NAT64 range (64:ff9b:1::/48, RFC 8215),
|
||||
// answers 403. [::203.0.113.5] holds the public address 203.0.113.5 in the
|
||||
// IPv4-compatible form (::/96) and answers 403 when blocked_networks lists
|
||||
// 203.0.113.0/24 or ::/96. [::ffff:203.0.113.5] holds it in the IPv4-mapped
|
||||
// form (::ffff:0:0/96) and answers 403 when blocked_networks lists
|
||||
// 203.0.113.0/24. Go keeps the plain address 203.0.113.5 in the mapped form
|
||||
// too, and an IPv6 range must not refuse it: with ::/0 listed it is not
|
||||
// refused. Neither is [::203.0.113.5] with blocked_networks empty. Where a
|
||||
// host is not refused, the fetcher tries to connect to it, finds nothing
|
||||
// there, and the request fails with another status.
|
||||
func TestImageRoutesCheckIPv4AddressInIPv6SourceHost(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const (
|
||||
ipv4CompatibleHost = "[::203.0.113.5]"
|
||||
ipv4MappedHost = "[::ffff:203.0.113.5]"
|
||||
)
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourceHost string
|
||||
blockedNetworks []netip.Prefix
|
||||
wantForbidden bool
|
||||
}{
|
||||
{"local-use NAT64 form of 10.0.0.1", "[64:ff9b:1::a00:1]", nil, true},
|
||||
{"IPv4-compatible form of an address in blocked_networks",
|
||||
ipv4CompatibleHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||
{"IPv4-compatible form with ::/96 in blocked_networks",
|
||||
ipv4CompatibleHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("::/96")}, true},
|
||||
{"IPv4-mapped form of an address in blocked_networks",
|
||||
ipv4MappedHost,
|
||||
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
|
||||
{"plain IPv4 address with ::/0 in blocked_networks",
|
||||
"203.0.113.5",
|
||||
[]netip.Prefix{netip.MustParsePrefix("::/0")}, false},
|
||||
{"IPv4-compatible form of an address with blocked_networks empty",
|
||||
ipv4CompatibleHost, nil, false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
imageURLs := map[string]string{
|
||||
"image URL with a signature": signedImageURL(t, tt.sourceHost),
|
||||
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
|
||||
}
|
||||
|
||||
for route, imageURL := range imageURLs {
|
||||
t.Run(tt.name+" "+route, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
s := startImageProxyWithBlockedNetworks(t, tt.blockedNetworks)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
s.ServeHTTP(rec, httptest.NewRequestWithContext(
|
||||
t.Context(), http.MethodGet, imageURL, nil))
|
||||
t.Logf("GET %s: %d", imageURL, rec.Code)
|
||||
|
||||
forbidden := rec.Code == http.StatusForbidden
|
||||
if tt.wantForbidden && !forbidden {
|
||||
t.Errorf("status = %d, want %d; body %s",
|
||||
rec.Code, http.StatusForbidden, rec.Body.String())
|
||||
}
|
||||
|
||||
if !tt.wantForbidden && forbidden {
|
||||
t.Errorf("status = %d, want any other status; body %s",
|
||||
rec.Code, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+3
-2
@@ -19,8 +19,9 @@
|
||||
# CGO image libraries (pkg-config, vips
|
||||
# with its JPEG XL support, libheif) for
|
||||
# the govips bindings instead of Node: to
|
||||
# compile pixa outside Docker, as
|
||||
# `make build` does
|
||||
# compile pixa, in the Dockerfile's test
|
||||
# phase and build stage, which format
|
||||
# nothing
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
+4
-5
@@ -2,12 +2,11 @@
|
||||
# script/cibuild: run the CI build. It bootstraps first: a CI runner
|
||||
# checks out and runs this and nothing else, and script/fmt-check runs
|
||||
# the formatter on the host, which a pristine checkout cannot do.
|
||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
||||
# line, since a cached gate phase is a check that did not run; unlike
|
||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
||||
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
||||
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
||||
# before every step that lints, tests or compiles; a stage without that
|
||||
# line is served from cache on an unchanged tree, with no warning.
|
||||
# first; a stage without that line is served from cache on an unchanged
|
||||
# tree, with no warning.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
|
||||
+4
-5
@@ -1,12 +1,11 @@
|
||||
#!/bin/sh
|
||||
# script/docker: build the Docker image tagged with the project name.
|
||||
# Identical in all repos; the tag comes from script/projectname.
|
||||
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
|
||||
# line, since a cached gate phase is a check that did not run; unlike
|
||||
# --no-cache, it keeps the layers above that line and the cache mounts.
|
||||
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
|
||||
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
|
||||
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
|
||||
# before every step that lints, tests or compiles; a stage without that
|
||||
# line is served from cache on an unchanged tree, with no warning.
|
||||
# first; a stage without that line is served from cache on an unchanged
|
||||
# tree, with no warning.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
|
||||
+5
-7
@@ -5,13 +5,11 @@
|
||||
# make its answer untrustworthy.
|
||||
#
|
||||
# The phase is not the last stage in the file, so it is built only when
|
||||
# --target names it. A new CACHEBUST value reruns every RUN step after
|
||||
# the ARG CACHEBUST line, since a cached lint layer is a lint that did
|
||||
# not run; unlike --no-cache, it keeps the layers above that line and
|
||||
# the cache mounts. This relies on every stage of the Dockerfile
|
||||
# declaring ARG CACHEBUST before every step that lints, tests or
|
||||
# compiles; a stage without that line is served from cache on an
|
||||
# unchanged tree, with no warning.
|
||||
# --target names it. A new CACHEBUST value reruns every RUN step, since
|
||||
# a cached lint layer is a lint that did not run; unlike --no-cache, it
|
||||
# keeps Go's build cache. This relies on every stage of the Dockerfile
|
||||
# declaring ARG CACHEBUST first; a stage without that line is served
|
||||
# from cache on an unchanged tree, with no warning.
|
||||
# --output type=cacheonly writes no image, since nothing uses one.
|
||||
set -eu
|
||||
|
||||
|
||||
+5
-6
@@ -3,12 +3,11 @@
|
||||
# and this builds that phase alone, on the same terms as script/lint:
|
||||
# --target because a phase that is not the last stage is built only when
|
||||
# named, and a new CACHEBUST value because a cached test layer is a test
|
||||
# that did not run; unlike --no-cache, it keeps the layers above the ARG
|
||||
# CACHEBUST line and the cache mounts. This relies on every stage of the
|
||||
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests
|
||||
# or compiles; a stage without that line is served from cache on an
|
||||
# unchanged tree, with no warning. --output type=cacheonly writes no
|
||||
# image, since nothing uses one.
|
||||
# that did not run; unlike --no-cache, it keeps Go's build cache. This
|
||||
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
|
||||
# a stage without that line is served from cache on an unchanged tree,
|
||||
# with no warning. --output type=cacheonly writes no image, since
|
||||
# nothing uses one.
|
||||
set -eu
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
|
||||
|
||||
Reference in New Issue
Block a user