3 Commits
Author SHA1 Message Date
clawbot 6572b3fe08 Test the 503 for a request waiting on a transcode stopped at a deadline
check / check (push) Waiting to run
A request that waited for another request's transcode, stopped at that
request's deadline, must get ErrProcessingTimedOut, which both image
routes answer 503. Declares that error so the tests build. Also tests
that Process, past its deadline before it starts, never reads its input.

Model: opus-5-5
2026-10-15 12:03:53 +00:00
clawbot 1ba4c8d39f Stop a transcode whose request has ended before it encodes (closes #243)
Process looked at its context only while it waited for a processing
slot, so a transcode went on decoding and encoding, holding its slot,
however long after its request's downstream_timeout that took.
acquireSlot now gives the slot back and returns the context's error
when the context has ended by the time it has a slot, and encode
returns that error without beginning. libvips cannot stop an encode
once it has begun, so that encode keeps its slot until it finishes.
README.md gives how long that can take per output format at the 8192
limits and no longer says the work stops when downstream_timeout ends;
TODO.md is corrected the same way.

Model: opus-5-5
2026-10-15 12:00:34 +00:00
clawbot 74ce122dc4 Test that Process stops once its deadline has passed
Process looks at its context only while it waits for a processing
slot, so a request whose deadline has passed, before Process starts or
while it reads its input, still has its image decoded and encoded while
it holds the slot. The new test fails until Process returns
context.DeadlineExceeded in both cases and gives the slot back
(#243).

Model: opus-5-5
2026-10-15 12:00:34 +00:00
60 changed files with 560 additions and 4688 deletions
+9 -18
View File
@@ -13,21 +13,12 @@
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and # `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context. # deletes the package directory from the context.
# .git is sent without its config. Without a VERSION build argument the # Unlike the standard file, which leaves out all of .git, pixa sends
# stage that compiles runs `git describe --tags --always` on .git, which # .git without its config. Without a VERSION build argument the stage
# does not need .git/config; that file can hold a credential, such as a # that compiles runs `git describe --tags --always` on .git, which does
# not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there. # password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory .git/config
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent. # Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root. # Anchored because it occurs once where agents run at the repo root.
@@ -51,9 +42,7 @@
**/[iI][dD]_[rR][sS][aA] **/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA] **/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA] **/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519 **/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in. # Dependencies: restored inside the image, never copied in.
**/node_modules **/node_modules
@@ -71,8 +60,10 @@
**/.vscode **/.vscode
**/*.sublime-* **/*.sublime-*
# pixa's own entries. On the host, `make build` writes bin/pixad, and the # pixa's own entries. Nothing in the build reads .gitignore. On the
# example config keeps its state directory in data/. # host, `make build` writes bin/pixad, and the example config keeps its
# state directory in data/.
.gitignore
/bin /bin
/data /data
-3
View File
@@ -13,6 +13,3 @@ indent_style = tab
[*.go] [*.go]
indent_style = tab indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
+3 -9
View File
@@ -1,21 +1,15 @@
name: check name: check
on: [push] on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# script/cibuild needs no token, so none is left in .git/config. # The default clone is shallow and has no tags, so the
# version the build takes from `git describe` would be a
# bare commit; this fetches the whole history with its tags.
with: with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0 fetch-depth: 0
- run: script/cibuild - run: script/cibuild
- run: script/docker-smoke - run: script/docker-smoke
+6 -37
View File
@@ -17,46 +17,15 @@ Thumbs.db
# entry and must not be given a `**/` prefix on the way into one. # entry and must not be given a `**/` prefix on the way into one.
.claude/ .claude/
# Node # Environment / secrets
node_modules/ .env
.env.*
# Secrets. Unanchored like every entry above, so each matches at every *.pem
# depth. Matching is case-sensitive on Linux, so names use character *.key
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Go
*.log
*.out
*.test
# Dependencies # Dependencies
vendor/ vendor/
node_modules/
# Build output # Build output
/bin/ /bin/
-3
View File
@@ -17,7 +17,6 @@ linters:
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
@@ -25,8 +24,6 @@ linters:
# silenced by disabling that name, not by enabling the successor. # silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5 - wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2 - gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings: settings:
lll: lll:
line-length: 88 line-length: 88
+5
View File
@@ -1,2 +1,7 @@
node_modules/ node_modules/
yarn.lock yarn.lock
# A byte-for-byte copy of the one in sneak/prompts.
REPO_POLICIES.md
vendor/
View File
+30 -48
View File
@@ -1,12 +1,7 @@
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# Lint phase. script/lint builds it alone. The linter is run directly: # Lint phase. script/lint builds it alone. The linter is run directly:
# `make lint` and script/lint are themselves a docker build. # `make lint` and script/lint are themselves a docker build.
# golangci/golangci-lint:v2.14.0, 2026-10-15 # golangci/golangci-lint:v2.12.2, 2026-10-04
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
ARG CACHEBUST
# The linter compiles every package, and govips needs the libvips # The linter compiles every package, and govips needs the libvips
# headers for that. REPO_POLICIES.md has the lint phase install them # headers for that. REPO_POLICIES.md has the lint phase install them
@@ -19,20 +14,17 @@ WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN golangci-lint run --config .golangci.yml ./...
golangci-lint run --config .golangci.yml ./...
# Test phase. script/test builds it alone. -race needs cgo and so a C # Test phase. script/test builds it alone.
# compiler, which the Debian Go image ships and the alpine one does not. # golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
# golang:1.25.4-trixie, 2026-10-15 FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
ARG CACHEBUST
WORKDIR /src WORKDIR /src
# script/bootstrap --cgo installs the build dependencies (the libvips # script/bootstrap --cgo installs the build dependencies (a C compiler,
# and libheif headers, and libvips' JPEG XL support, which the tests # the libvips and libheif headers, and libvips' JPEG XL support, which
# need) and downloads the Go modules. # the tests need) and downloads the Go modules.
COPY script/ ./script/ COPY script/ ./script/
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN script/bootstrap --cgo RUN script/bootstrap --cgo
@@ -40,14 +32,11 @@ RUN script/bootstrap --cgo
COPY . . COPY . .
# Without -v first; on a failure, again with -v for the details, and # Without -v first; on a failure, again with -v for the details, and
# the step fails even if the second run passes. -count=1 because the # the step fails even if the second run passes. -parallel 4: by default
# mounted cache also holds test results, which go test would otherwise # a package runs as many of its tests at once as the host has CPUs, and
# replay. -parallel 4: by default a package runs as many of its tests at # on a busy host they then wait so long to be scheduled that a test
# once as the host has CPUs, and on a busy host they then wait so long # that times a request can see it return a second late.
# to be scheduled that a test that times a request can see it return a RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
# second late.
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; } go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
@@ -56,44 +45,38 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
# test passed. # test passed.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too # golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
WORKDIR /src WORKDIR /src
# Build dependencies, git among them, and Go modules, as in the test # Build dependencies and Go modules, as in the test phase
# phase
COPY script/ ./script/ COPY script/ ./script/
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN script/bootstrap --cgo RUN script/bootstrap --cgo
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
# Copy source code # Copy source code
COPY . . COPY . .
# The VERSION build arg when one is given, otherwise # VERSION is declared here, not earlier: a new value reruns only the
# `git describe --tags --always` on the .git in the build context: the # build, not script/bootstrap. Given none, the version is
# tag on a tagged commit, tag-N-gHASH after one, the short commit when # `git describe --tags --always` of the .git in the build context (git
# no tag is reachable. With .git present, a version that is still empty, # comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
# dev or unknown fails the build: git is missing or could not read the # after one, the short commit when no tag is reachable. A context that
# checkout. A context without .git, as from a source tarball, stamps an # carries .git and still yields no version fails the build; one without
# empty version. CGO stays enabled for govips; -trimpath keeps build # .git, as from a source tarball, stamps an empty version. CGO stays
# paths out of the binary, and -s -w leave out the symbol table and # enabled for govips; -trimpath keeps build paths out of the binary, and
# debug information. # -s -w leave out the symbol table and debug information.
ARG VERSION ARG VERSION
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN version="${VERSION:-$(git describe --tags --always)}"; \
VERSION="${VERSION:-$(git describe --tags --always)}"; \ if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
if [ -e .git ]; then \ [ "$version" = unknown ]; }; then \
case "$VERSION" in ""|dev|unknown) \ echo "the build context carries .git but yields no version" >&2; \
echo "version is '$VERSION' although .git is present" >&2; \ exit 1; \
exit 1 ;; \
esac; \
fi; \ fi; \
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \ CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \ -ldflags "-s -w -X main.Version=${version}" \
-o /pixad ./cmd/pixad -o /pixad ./cmd/pixad
# Runtime stage, and the last one: a plain `docker build .` builds this # Runtime stage, and the last one: a plain `docker build .` builds this
@@ -102,7 +85,6 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
# the libvips and musl it was built with. # the libvips and musl it was built with.
# alpine:3.22, 2026-10-08 # alpine:3.22, 2026-10-08
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8 FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
ARG CACHEBUST
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL # Install runtime dependencies only. vips-jxl is libvips' JPEG XL
# support, without which pixad does not start. # support, without which pixad does not start.
+41 -177
View File
@@ -71,8 +71,7 @@ The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
database and the disk cache: database and the disk cache:
- It needs a persistent volume: without one, every restart starts with an empty - It needs a persistent volume: without one, every restart starts with an empty
cache. Losing the directory also ends every login session (see Encrypted cache. In the container, the startup script gives the directory to the user
URLs). In the container, the startup script gives the directory to the user
pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user pixa runs as (uid 65532) and sets its mode to `750`; outside it, that user
must be able to write the directory. must be able to write the directory.
- `cache_max_bytes` limits the source and transformed images together. The - `cache_max_bytes` limits the source and transformed images together. The
@@ -124,41 +123,9 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
comma-separated; default none. Every image on a listed host is served to comma-separated; default none. Every image on a listed host is served to
anyone without a signature, so list only hosts you control, where nobody anyone without a signature, so list only hosts you control, where nobody
else can put images. else can put images.
- `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it. Set - `PIXA_CACHE_MAX_BYTES`: disk cache limit in bytes; `0` disables it;
it: the volume is a directory on the host's filesystem, shared with the default 75% of (free space + what the cache holds)
host and every other app. The default is 75% of the sum of that
filesystem's free space and what the cache holds, and at least 500 MiB, so
the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below - the rest are in the table under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait
for processing. 4.2 GiB is the most resident memory was measured to grow while
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
largest it accepts, saving it at full size to each output format in turn, one
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use.
- **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
written, so a listing of `/tmp` does not show it, but it keeps its space, on
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
well as written to that file, and the 4.2 GiB above includes that memory. For
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
does not include these files; where `/tmp` is a tmpfs, they are memory on top
of it.
- **Health check:** the image's `HEALTHCHECK` requests - **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds `/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
after a deploy and marks the deploy failed unless it is `healthy`. The probe after a deploy and marks the deploy failed unless it is `healthy`. The probe
@@ -193,14 +160,6 @@ name.
Multiple source paths may reference the same content blob; the database tracks Multiple source paths may reference the same content blob; the database tracks
references rather than using filesystem refcounting. references rather than using filesystem refcounting.
Source and transformed image files, and the `.meta` file beside each transformed
image, are written under a temporary name, synced to disk, and only then renamed
into place, so a crash leaves no file under its own name with part of its data.
When pixa starts, and every five minutes after, it removes each cached source or
transformed image whose file is empty, or not the size the database records for
it, with a warning naming the file; the next request for it fetches the source
or converts the image again.
pixa's target is 1-5k r/s, which has not been measured at that rate (see Load pixa's target is 1-5k r/s, which has not been measured at that rate (see Load
Test). Toward it, pixa keeps in memory the content types of the 10,000 Test). Toward it, pixa keeps in memory the content types of the 10,000
transformed images most recently cached or served, so a cache hit on one of them transformed images most recently cached or served, so a cache hit on one of them
@@ -225,11 +184,7 @@ path under `/v1/` answers 200, in maintenance mode too.
session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400 session it answers 303 to `/`. Answers: 200 with the page showing the URL; 400
with the page naming a field that is not valid; 500 when the URL cannot be with the page naming a field that is not valid; 500 when the URL cannot be
made. made.
- `POST /logout` — log out with the generator page's Logout button: end every - `GET /logout` — end the login session. Needs: nothing. Answers: 303 to `/`.
login session, in every browser (see Encrypted URLs), and remove the session
cookie. Needs: a login session and the generator page's form (below); without
a login session it only removes the cookie. Answers: 303 to `/`; 500 when the
logout cannot be saved in the database.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or - `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized `/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
and converted (below). Needs: a signature, unless the host is allowlisted (see and converted (below). Needs: a signature, unless the host is allowlisted (see
@@ -277,7 +232,7 @@ the request carries the same ID as `request_id`, and so do the lines it logs
when it fetches, converts and serves an image; the fetch sends it to the when it fetches, converts and serves an image; the fetch sends it to the
upstream host as `X-Request-ID`. upstream host as `X-Request-ID`.
The `POST` routes accept only a form that pixa's own page served: the page puts Both `POST` routes accept only a form that pixa's own page served: the page puts
a token in the form and sets a cookie to match, and a request without both is a token in the form and sets a cookie to match, and a request without both is
refused with 403, so another site cannot submit the form from a visitor's refused with 403, so another site cannot submit the form from a visitor's
browser. The login and generator pages are meant to be opened over HTTPS: while browser. The login and generator pages are meant to be opened over HTTPS: while
@@ -316,35 +271,6 @@ once, is refused with 400.
both optional (values under Signature Specification). Both are part of what is both optional (values under Signature Specification). Both are part of what is
cached, so each value of either is a separate cached image. cached, so each value of either is a separate cached image.
A source URL's query follows its path after `%3F`, the encoded `?`:
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
percent-escapes in the image URL's path once, then reads the host up to the
first `/`, the size and format after the last `/`, and between them the source
path up to the first `?` and the source query after it. So the `?` before the
query, any `?` in the query, and every `%` in the source path or query must be
percent-encoded, as `%3F` and `%25`. Any other character may be written as
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
the source path nor the source query may hold a `#`, and an image URL that
writes one as `%23` is refused with 400. For example,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 WebP is
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
signed as shown under Signature Specification.
The source path is the path with its percent-escapes decoded, and pixa fetches
it percent-encoded again: every character other than letters, digits, `-`, `.`,
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
percent-escape. So `my%20cat.jpg` in the image URL is the source path
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
`/100%25.png`. The source query is the source URL's query as that URL writes it,
its percent-escapes included, and is sent as it is, except that the characters a
browser percent-encodes in a query (a space, `"`, `'`, `<`, `>`, control
characters and characters outside ASCII) and a `%` that does not begin a
percent-escape are percent-encoded. pixa signs and caches the query in that
form, so `img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are
the same source query, `name=my%20cat`.
A size with one side `0` has that side scaled to keep the source's proportions, A size with one side `0` has that side scaled to keep the source's proportions,
and is refused with 400 when that side would come out above 8192, the most and is refused with 400 when that side would come out above 8192, the most
either side may be; so is a size with `fit=outside`, which scales the image to either side may be; so is a size with `fit=outside`, which scales the image to
@@ -393,12 +319,9 @@ fetch sends the first request's ID upstream, and the lines logged for the fetch
and the transcode carry that ID. and the transcode carry that ID.
A transcode whose request's `downstream_timeout` has ended (for a shared one, A transcode whose request's `downstream_timeout` has ended (for a shared one,
the first request's) stops waiting for a processing slot, or, once it has one, the first request's) stops when it gets its processing slot or before it begins
gives it back without beginning to encode. A request waiting for a shared encoding, and gives the slot back. libvips cannot stop an encode once it has
transcode that stops this way is answered 503 with the error begun, so an encode keeps its slot until it finishes, which can be after
`server busy, try again later`, though its own `downstream_timeout` may not have
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
encode keeps its slot until it finishes, which can be after
`downstream_timeout`. The limit of 8192 on each side of the source and of the `downstream_timeout`. The limit of 8192 on each side of the source and of the
output bounds how long that takes. With one libvips thread, as pixad runs it, output bounds how long that takes. With one libvips thread, as pixad runs it,
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG, encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
@@ -444,28 +367,14 @@ nor change what it asks for.
1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see 1. Open `/` in a browser over HTTPS (or over plain HTTP while `debug` is on, see
Routes) and log in with the signing key (`signing_key`). The login session Routes) and log in with the signing key (`signing_key`). The login session
lasts 30 days, or until someone logs out with the generator page's Logout lasts 30 days, or until `/logout`.
button. Every login is made with the one signing key, so logging out ends
every login session, in every browser, a copy of the session cookie included.
pixa keeps a random value in its database, which it makes when the database
has none and replaces at each logout. A login session carries the value
current at its login and is refused once the value has changed, also after a
restart (see Deployment). The check for a logout compares no time, so a login
after a logout works whatever the host's clock says; the 30 days are still
measured by the host's clock.
2. On the generator page, give the source image's URL, the width and height, the 2. On the generator page, give the source image's URL, the width and height, the
format, quality and fit, and how long the URL lasts, then submit the form format, quality and fit, and how long the URL lasts, then submit the form
(`POST /generate`). The format is JPEG XL unless another is chosen; a form (`POST /generate`). The format is JPEG XL unless another is chosen; a form
sent with an empty format, or none, also makes a JPEG XL URL. Width and sent with an empty format, or none, also makes a JPEG XL URL. Width and
height both empty or `0` keep the original size; if only one of them is empty height both empty or `0` keep the original size; if only one of them is empty
or `0`, that side is scaled to keep the image's proportions. A URL whose or `0`, that side is scaled to keep the image's proportions. A URL whose
scaled side would come out above 8192 answers 400, and so does a source URL scaled side would come out above 8192 answers 400.
whose path holds a `..` segment. So does a source URL that the image URL
would fetch written another way, and the page then shows the URL pixa would
fetch: one with a user name and password, which pixa does not send, or one
whose path holds `%2F`, fetched as `/`, or `(`, fetched as `%28`, as pixa
keeps the source path decoded and fetches it percent-encoded again (see
Routes).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when 3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on, and the URL starts it expires. `<host>` is the host the page was opened on, and the URL starts
with `http` instead while `debug` is on. The name after the token is ignored with `http` instead while `debug` is on. The name after the token is ignored
@@ -473,11 +382,9 @@ nor change what it asks for.
`jxl` for a form with no format. `jxl` for a form with no format.
The token holds the source's host, path and query and the size, format, quality, The token holds the source's host, path and query and the size, format, quality,
fit and expiry, encrypted with a key derived from `signing_key`. The host is fit and expiry, encrypted with a key derived from `signing_key`. The source
kept in lower case and an empty path as `/`, as for `/v1/image/`, so URL's scheme is not kept: the image is fetched like any other (see Routes), and
`https://CDN.Example.com` and `https://cdn.example.com/` are one source image, the blocked networks still apply.
fetched and cached once. The source URL's scheme is not kept: the image is
fetched like any other (see Routes), and the blocked networks still apply.
How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never. How long the URL lasts is chosen on the page, from 1 minute to 1 year, or never.
The expiry is fixed in the token when the URL is made and cannot be changed or The expiry is fixed in the token when the URL is made and cannot be changed or
@@ -529,20 +436,13 @@ Where:
- `host` — source host as the URL writes it up to the first `/`, in lower case, - `host` — source host as the URL writes it up to the first `/`, in lower case,
keeping an IPv6 address's brackets and any `:` and port after it (e.g. keeping an IPv6 address's brackets and any `:` and port after it (e.g.
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`) `cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
- `path` — source path, with its percent-escapes decoded as under Routes (e.g. - `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
`/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is - `query` — source query string, empty string if none, with `%` and `:` written
signed as `/photos/a%3Ab.jpg` as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
- `query` — source query string as under Routes, empty string if none, with `%` `/a` with query `b:` are signed differently
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no - `width` — requested width in pixels, `0` for original
query and path `/a` with query `b:` are signed differently - `height` — requested height in pixels, `0` for original
- `width` — requested width in pixels, or `0`: the width is then worked out from
the source's proportions and the requested height, rounded down but at least
1, or is the source's own width when the height is `0` too (the `orig` size)
- `height` — requested height in pixels, or `0`: the height is then worked out
from the source's proportions and the requested width, rounded down but at
least 1, or is the source's own height when the width is `0` too (the `orig`
size)
- `format` — output format, one of those listed under Routes, with `original` - `format` — output format, one of those listed under Routes, with `original`
signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no signed as `orig`, `jpg` as `jpeg`, and no format as `jxl`, so a URL with no
format has the signature of the same URL ending in `.jxl`; `auto` is signed as format has the signature of the same URL ending in `.jxl`; `auto` is signed as
@@ -555,10 +455,7 @@ Where:
with 400 with 400
- `fit` — the URL's `fit` query parameter (cover, contain, fill, inside, - `fit` — the URL's `fit` query parameter (cover, contain, fill, inside,
outside), or `cover` when the URL has no `fit`; a request whose `fit` is outside), or `cover` when the URL has no `fit`; a request whose `fit` is
anything else, an empty `fit=` included, is refused with 400. `outside` scales anything else, an empty `fit=` included, is refused with 400
the image to cover the requested size, keeping its proportions, so it can come
out larger than that size on one side: an 8192x1 source at `100x0`, which is
100x1 by its proportions, comes out at 8192x1
The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of The URL's `sig` is the HMAC-SHA256 result in base64url (the URL-safe alphabet of
RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa RFC 4648) with the trailing `=` padding kept, 44 characters in all. pixa
@@ -582,17 +479,6 @@ For the same image at quality 40 with fit `contain`, the input ends in
and the URL is and the URL is
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`. `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
For a source URL with a query,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
800x600 WebP with the same key and expiration and the default quality and fit:
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
`%25` (see Routes):
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
**Allowlist patterns:** **Allowlist patterns:**
- **Exact match**: `cdn.example.com` — matches only that host - **Exact match**: `cdn.example.com` — matches only that host
@@ -608,11 +494,7 @@ pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
`CDN.Example.com` is matched, fetched, cached and counted against `CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has `upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
even when the host is allowlisted. The host must be a host name or an IPv4 even when the host is allowlisted. A host holding `@` is refused with 400.
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
brackets, either one optionally followed by `:` and a port. Any other host, such
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
refused with 400.
### Configuration ### Configuration
@@ -656,7 +538,7 @@ tell. With no file, pixa uses the environment and the defaults.
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` | | `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` | | `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` | | `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 | | `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs |
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` | | `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` | | `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB | | `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
@@ -694,10 +576,7 @@ Key settings in more detail:
and generator pages. Default: empty and generator pages. Default: empty
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added - `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, added
to the always-enforced built-in ranges (loopback, private, link-local, CGNAT, to the always-enforced built-in ranges (loopback, private, link-local, CGNAT,
benchmark, NAT64, and the like); an invalid CIDR aborts startup. An IPv6 benchmark, NAT64, and the like); an invalid CIDR aborts startup
address in the IPv4-mapped form (`::ffff:a.b.c.d`) is matched only as the IPv4
address it holds, so an IPv6 range such as `::/0` refuses neither it nor a
plain IPv4 address
- `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of - `trusted_proxies` — list of CIDR ranges of the reverse proxies in front of
pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one pixa. `X-Forwarded-For` is believed only when the direct peer falls inside one
of these ranges; the logged and login-recorded client address is then the of these ranges; the logged and login-recorded client address is then the
@@ -728,14 +607,11 @@ Key settings in more detail:
- `signing_key` — HMAC secret for URL signatures - `signing_key` — HMAC secret for URL signatures
- `db_url` — the SQLite database to open; omitted, it is - `db_url` — the SQLite database to open; omitted, it is
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the `file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or database in WAL mode. pixa opens one connection to it, so its own reads and
`#` in the name is written as `%25`, `%3F` or `%23`. For a `state_dir` of `/` writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to any
or one starting with `//` it starts `file:////`, so SQLite does not read the `db_url`, so a write that finds another program writing to the file waits up
first name in the path as a host. pixa opens one connection to it, so its own to five seconds for it instead of failing. WAL mode comes only from the URL:
reads and writes run one at a time. pixa adds `_pragma=busy_timeout(5000)` to keep `_pragma=journal_mode(WAL)` in one you set
any `db_url`, so a write that finds another program writing to the file waits
up to five seconds for it instead of failing. WAL mode comes only from the
URL: keep `_pragma=journal_mode(WAL)` in one you set
- `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk - `cache_max_bytes` — disk cache size limit in bytes; `0` disables the disk
cache entirely; omitted defaults to 75% of the sum of the free space on the cache entirely; omitted defaults to 75% of the sum of the free space on the
filesystem containing `<state_dir>/cache/` and the bytes of source and filesystem containing `<state_dir>/cache/` and the bytes of source and
@@ -748,12 +624,10 @@ Key settings in more detail:
does, and `downstream_timeout` has not ended first, the request is answered does, and `downstream_timeout` has not ended first, the request is answered
503 with the error `server busy, try again later` 503 with the error `server busy, try again later`
- `max_concurrent_processing` — the most images decoded and encoded at once; - `max_concurrent_processing` — the most images decoded and encoded at once;
default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's default the number of CPUs pixa can use (`GOMAXPROCS`), which follows a
CPU limit), rounded down, and at least 1. A value that is not a whole number container's CPU limit. A request that finds all of them in use waits up to 10
of at least 1 aborts startup. A request that finds all of them in use waits up seconds for one to free up; if none does, and `downstream_timeout` has not
to 10 seconds for one to free up; if none does, and `downstream_timeout` has ended first, it is answered 503 the same way
not ended first, it is answered 503 the same way. Each one can take a lot of
memory: see "Running under upaas"
- `image_requests_per_minute` — how many requests one client may send to the - `image_requests_per_minute` — how many requests one client may send to the
image routes per minute (see the limit on them under Routes); default `600`. image routes per minute (see the limit on them under Routes); default `600`.
`0` turns the limit off, and a negative or non-integer value aborts startup. `0` turns the limit off, and a negative or non-integer value aborts startup.
@@ -780,12 +654,7 @@ See `configs/config.example.yml` for all options with defaults.
- **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics - **Metrics**: Prometheus, at `/metrics`: generic HTTP request metrics
(duration, response size, requests in flight) and the Go runtime and process (duration, response size, requests in flight) and the Go runtime and process
metrics; requests are measured and `/metrics` is served only when metrics; requests are measured and `/metrics` is served only when
`metrics.username` and `metrics.password` are set. A request's `handler` label `metrics.username` and `metrics.password` are set
is the pattern of the route it matches, such as `/v1/image/*`, never its path.
Requests that match no route share the label `unmatched`: those the router
answers 404 or 405, and CORS preflights. A request's `method` label is its
method when that is `GET`, `HEAD`, `POST`, `PUT`, `PATCH`, `DELETE`,
`OPTIONS`, `CONNECT` or `TRACE`, and `other` for any other method
- **Logging**: stdlib slog - **Logging**: stdlib slog
## Entrypoints ## Entrypoints
@@ -804,9 +673,9 @@ provide:
install-precommit) install-precommit)
- `script/projectname` — output the project name ("pixa") - `script/projectname` — output the project name ("pixa")
- `script/test` — run the test suite: build the `test` phase of the - `script/test` — run the test suite: build the `test` phase of the
`Dockerfile`, writing no image `Dockerfile`, tagged `pixa-test`
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`, - `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
writing no image; the linter never runs on the host tagged `pixa-lint`; the linter never runs on the host
- `script/fmt` — format the Go code with gofmt and the markdown with prettier - `script/fmt` — format the Go code with gofmt and the markdown with prettier
(writes) (writes)
- `script/fmt-check` — check the same formatting (read-only), on the host - `script/fmt-check` — check the same formatting (read-only), on the host
@@ -824,20 +693,15 @@ provide:
- `script/install-precommit` — install the git pre-commit hook that runs - `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit` `script/precommit`
Every `docker build` in these scripts passes a new value of the `CACHEBUST` Every `docker build` in these scripts passes `--no-cache`, so the lint and test
build argument, which every stage of the `Dockerfile` declares first, so every phases run on every build instead of coming from the build cache.
`RUN` step, the lint and test phases included, runs on every build instead of
coming from the build cache. Go's build cache is kept from build to build: the
steps that compile mount it, so a package whose inputs are unchanged is not
compiled again, and both `go test` runs pass `-count=1` so that no stored test
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
libvips and libheif; the image pixa ships is built and runs on Alpine.
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`, `script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
`script/setup` and `script/install-precommit` are the standard copies from `script/setup` and `script/install-precommit` are the standard copies from
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are `sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
the standard copies with pixa's `gofmt` step kept before prettier. prettier the standard copies with pixa's `gofmt` step kept before prettier. prettier
formats the markdown only, not the HTML templates, as it cannot parse a Go formats the markdown only: not the HTML templates, as it cannot parse a Go
template action inside a tag. template action inside a tag, and not `REPO_POLICIES.md` (see
`.prettierignore`), a copy of the one in `sneak/prompts`.
## Load Test ## Load Test
+103 -256
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-15 last_modified: 2026-09-08
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -54,39 +54,34 @@ style conventions are in separate documents:
`cibuild`. `script/bootstrap` installs all dependencies idempotently and `cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the (detected in that order; apt runs noninteractive). For node it uses the
installed node only when its major version is the pinned one; otherwise (node installed node if present; otherwise it installs a PINNED node version via
missing, or another major, such as the node 24 the CI runner image ships) it nvm, first installing nvm itself if missing — from a hash-verified GitHub
installs the PINNED node version via nvm and installs yarn and the packages release archive (never `curl | sh`), with bash installed as an explicit
under it. nvm itself is installed first if missing, from a hash-verified prerequisite since nvm requires bash. yarn is then pinned via
GitHub release archive (never `curl | sh`), with bash installed as an explicit `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
prerequisite since nvm requires bash. Only the major version is compared always exact versions. `script/cibuild` runs the CI build: it changes to the
because the `Dockerfile` stages start from a node image whose exact version is repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned with the version; the Gitea workflow calls it. **`script/cibuild` runs
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs `script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**: when containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm it leaves neither on the `script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs `PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
under another node. The host entrypoints that need yarn — `script/fmt` and entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when source nvm for the pinned node version before invoking it, exactly as
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`, `script/bootstrap`'s own install step does. A runner carrying nothing but
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the docker and git then gets through `script/check`. Four further scripts are our
pinned node. A runner carrying nothing but docker and git then gets through own extensions to the standard: `script/check` runs `script/test`,
`script/check`. Four further scripts are our own extensions to the standard: `script/lint` and `script/fmt-check`; `script/precommit` is what the git
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`; pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
`script/precommit` is what the git pre-commit hook runs, and it calls installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/check`; `script/install-precommit` installs the git pre-commit hook `script/projectname` (literally that filename) simply outputs the project's
(the `make hooks` target shims to it); and `script/projectname` (literally name. Scripts that need the name call `script/projectname` — e.g.
that filename) simply outputs the project's name. Scripts that need the name `script/docker` assembles its image tag from it — so those scripts stay
call `script/projectname` — e.g. `script/docker` assembles its image tag from byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
it — so those scripts stay byte-identical across all repos. Repo-type-specific `go mod tidy` verification in Go repos) belong in `script/precommit`, not in
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in the hook itself. Model scripts are at
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the must document the provided scripts in an **Entrypoints** section (see the
README requirements below). README requirements below).
@@ -109,24 +104,18 @@ style conventions are in separate documents:
`lint` phase and a `test` phase, with the final stage depending on both so the `lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image. brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and Dockerfiles install development prerequisites by running `script/bootstrap`
install what those images lack either inline, as the canonical Go `Dockerfile` rather than duplicating installs inline; COPY `script/` and the dependency
below does for `git`, or by running `script/bootstrap`, as the `prompts` manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
repo's own `Dockerfile` does for its yarn packages. The development running it.
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else: and nothing else:
```sh ```sh
docker build --build-arg CACHEBUST="$(date +%s)-$$" \ docker build --no-cache --target lint -t "$(script/projectname)-lint" .
--target lint --output type=cacheonly . docker build --no-cache --target test -t "$(script/projectname)-test" .
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target test --output type=cacheonly .
``` ```
**A stage that is not the last one in the file is built only when the final **A stage that is not the last one in the file is built only when the final
@@ -136,15 +125,10 @@ style conventions are in separate documents:
plain `docker build .` builds the last stage alone and exits 0 having linted plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing. and tested nothing.
**The gate builds write no image.** With `--output type=cacheonly` the phase **Every `docker build` in `script/` is tagged**, here and in
runs and a failing step fails the build, but the result is not exported. `script/cibuild` and `script/docker`. An untagged build leaves a dangling
Nothing uses those images, and writing one out is slow: a Go test phase's image behind on every invocation, on every developer host and every CI
image holds the toolchain. A build given neither `--output` nor `-t` writes runner; a tagged one replaces the previous image.
an untagged image and leaves it dangling, on every developer host and every
CI runner. `script/cibuild` and `script/docker` build the image that ships
and tag it, so each build replaces the previous image; each assigns the tag
on its own line before the build, so `set -e` stops it where
`script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`, Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are `eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -159,107 +143,66 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own golangci-lint cache, its org, in both directions. A container has its own cache, its own `TMPDIR` and
own `TMPDIR` and a digest-pinned binary, so neither is reachable. a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - **Any build that runs checks is built with `--no-cache`.** Docker invalidates
invalidates a `COPY` layer only when the copied content changes, so on an a `COPY` layer only when the copied content changes, so on an unchanged tree
unchanged tree the check `RUN` is served from cache, nothing executes, and the the check `RUN` is served from cache, nothing executes, and the build still
build still exits 0. Every `docker build` in `script/` therefore passes exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process `script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
id, so no two builds share a value, not even the three in one four, and there is no fifth — `script/check` runs the two gate phases and
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and `script/fmt-check`, and builds no image of its own. A bare `docker build .` is
`script/docker` are the four, and there is no fifth — `script/check` runs the not evidence that anything ran: a sub-second build reporting success is a
two gate phases and `script/fmt-check`, and builds no image of its own. Every cache hit, not a result. Never invalidate by pruning — `docker builder prune`
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after and friends destroy a build cache shared with every other build on the host.
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
are compared by content and need nothing. A stage without the `ARG` line is
served from cache on an unchanged tree, so a repository re-vendors the scripts
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
it reruns every step too, but it also empties every cache mount, and Go's
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
packages survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every
other build on the host. When a check is added or changed, prove it works by
planting a defect it must catch and watching the run fail on it, then revert
the defect. A green run alone shows neither that the check ran nor that it
covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.** - **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile, hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo and the test phase is based on the Go image. The canonical Go repo
`Dockerfile`: `Dockerfile`:
```dockerfile ```dockerfile
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# Lint phase # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN golangci-lint run --config .golangci.yml ./...
golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go # Test phase
# image ships and the alpine one does not. -count=1 because the mounted # golang:1.x-alpine, YYYY-MM-DD
# cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN go test -timeout 90s -race -cover ./... || \
go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies # Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run # are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed. # unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
# The VERSION build arg when one is given, otherwise ARG VERSION=dev
# `git describe --tags --always` on the .git in the build context. With RUN CGO_ENABLED=0 go build -trimpath \
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN --mount=type=cache,target=/root/.cache/go-build \
VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \ -ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/ -o /app ./cmd/app/
# Runtime stage, and the last one # Runtime stage, and the last one
FROM alpine@sha256:... FROM alpine@sha256:...
ARG CACHEBUST
COPY --from=builder /app /usr/local/bin/app COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"] ENTRYPOINT ["app"]
``` ```
@@ -274,88 +217,23 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain same two copies. A plain `docker build .` builds the last stage's chain
and nothing else. and nothing else.
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
it for root), and only on the steps that compile: `golangci-lint run`,
both `go test` runs and `go build`. It keeps compiled packages between
check builds, and Go reuses one only when all its inputs are unchanged, so
it saves the compile without skipping a check. Nothing else is kept:
golangci-lint's own cache would replay stored findings in place of a lint
run, and Go builds from whatever files a kept module cache holds without
checking them again (only `go mod verify` does), so a file changed there
would go unnoticed. Go's build cache also holds test results, which is why
both `go test` runs carry `-count=1`.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting, install them - If the project requires CGO or system libraries for linting (e.g.
in the lint phase. The `golangci/golangci-lint` image is Debian-based and `vips-dev`), install them in the lint phase with `apk add`.
has no `apk`, so install with `apt-get` under the Debian package name - `ARG VERSION=dev` is declared in the stage that compiles and supplied by
(`libvips-dev`, where alpine says `vips-dev`), and delete the package `script/docker` and `script/cibuild`; no stage may call `git describe`.
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
A checkout whose `.git` is a file (a linked worktree, or a repository
checked out as a submodule) is the exception: that file points to a git
directory outside the build context, so the build cannot read the version
and a plain `docker build .` fails; pass the version with
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step, runs `script/cibuild` on push, and checks out the repo as its only other step.
with `persist-credentials: false`: `script/cibuild` needs no token, and That script bootstraps, runs the gate phases, and then builds the image, so a
without it the checkout leaves the job's token in `.git/config` for every successful run means every check passed; a bare `docker build .` does not
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
tags `git describe` needs: by default it clones shallow with no tags, and a
tagged repository's CI build would stamp a bare short commit id. The
workflow's `concurrency` block groups runs by workflow and branch
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
so a new push cancels the older run on the same branch, queued or running, and
no other: runs for replaced commits do not hold up the shared runner.
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The carry the same guarantee, because its gate phases may come from the cache. The
image build gets a `CACHEBUST` value of its own and so runs the gate phases a image build is uncached and so runs the gate phases a second time. That is the
second time. That is the price of the rule above, and it is worth paying: the price of the rule above, and it is worth paying: the image that ships is built
image that ships is built from a run of its own gates rather than from a cache from a run of its own gates rather than from a cache entry.
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
shared runner after 20 minutes. That allows for the three Docker builds
described above (the test phase, the lint phase, then the image), each held to
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
limited to `main` by a `branches` list under `on: push` cannot be checked by
review: to try a change to it, add the feature branch to that list and push,
then remove the branch from the list again before merging. Keep any job in it
that publishes behind `if: github.ref_name == 'main'`, so the run from the
feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -408,18 +286,17 @@ style conventions are in separate documents:
``` ```
`-count=1` is required on both invocations: it defeats Go's test _result_ `-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the cache, so the target cannot report a pass it did not earn, and the rerun
tests. It leaves the build cache alone, so it costs the runtime of the suite reproduces a failure instead of replaying it. It leaves the build cache
and no recompilation. alone, so it costs the runtime of the suite and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a Note that this is a second, independent cache, stacked below the Docker
passing result in its cache directory (`GOCACHE`), and when the same tests layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
run again on unchanged code it prints that result, marked `(cached)`, addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
without running them. The directory lasts from one run to the next on a it does not guarantee `go test` inside that step does any work, because the
developer's machine, where this target runs, and in the `test` phase of the `GOCACHE` baked into earlier image layers survives into the re-executed
`Dockerfile`, which mounts it to keep compiled packages between check step. They are two separate defects requiring two separate fixes, and a fix
builds, so the phase's `go test` runs carry `-count=1` too. A new for one must not be recorded as covering the other.
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
Python example: Python example:
@@ -447,12 +324,9 @@ style conventions are in separate documents:
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`), editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
`node_modules/`, and the repo's own build outputs. Fetch the standard language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
`.gitignore` from from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up setting up a new repo. These patterns are written to `.gitignore`'s own
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified. are not a `.dockerignore` and must not be transplanted into one unmodified.
@@ -466,7 +340,7 @@ style conventions are in separate documents:
— which is more dangerous than a short file with no secret patterns at all, — which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built root-anchored entries unprefixed: `.git`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is `cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
@@ -491,24 +365,21 @@ style conventions are in separate documents:
directory, so a repo running agents in subdirectories still ships directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there. `services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that - **Excluding `.git` means `git describe` cannot run inside any build stage, and
`git describe --tags --always` gives**, derived from the `.git` in the build it fails quietly there.** In a build stage there is no repository, so
context as the canonical `Dockerfile` above shows. Without its failure check, `git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty the binary reports no version at all, and the build still exits 0. Compute the
and the build would still exit 0. `script/docker` and `script/cibuild` pass version on the host and thread it in as a build arg. `script/docker` and
the version they compute on the host; it takes precedence. They do this `script/cibuild` do this, byte-identically across repos:
byte-identically across repos:
```sh ```sh
# The version and the tag each get their own line: a failing command # Own line: a failing command substitution inside an argument does not
# substitution inside an argument does not trip `set -e`, so the inline # trip `set -e`, so the inline form degrades to an empty constant.
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$(script/projectname)" docker build --no-cache \
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$(script/projectname)" .
``` ```
`--always` makes an untagged repo yield an abbreviated commit hash rather `--always` makes an untagged repo yield an abbreviated commit hash rather
@@ -516,12 +387,12 @@ style conventions are in separate documents:
fallback is applied — a live check that fires on a build from an export with fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so the canonical checkout action clones shallow and fetches no tags, so a repo that embeds a
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step. tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the - **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build patterns.** Plant files at the root _and_ at least two directories deep, build
@@ -555,18 +426,12 @@ style conventions are in separate documents:
`test-support` depguard rule, where a repo names its own test-support packages `test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
image image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`, (`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go` which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
directive must not name a newer Go minor version than the one golangci-lint only pin, since no repo installs golangci-lint on the host: bumping the
was built with, or golangci-lint refuses to lint it: this release lints version means changing it and nothing else.
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by - **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
@@ -588,18 +453,8 @@ style conventions are in separate documents:
function defined and never invoked has the same exit status and the same function defined and never invoked has the same exit status and the same
empty output as one that worked. empty output as one that worked.
Node is the exception to the whole-token comparison: the canonical
`script/bootstrap` compares only its major version, because the node 22
alpine image the `Dockerfile` stages start from is not the exact pin, and
nvm cannot install a prebuilt node on alpine.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`. Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference - When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD). with the version and date (YYYY-MM-DD).
@@ -709,17 +564,13 @@ style conventions are in separate documents:
Never edit existing migrations after release. Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation - All repos should have an `.editorconfig` enforcing the project's indentation
settings: the standard file from settings.
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`, only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
and language-specific config). Everything else goes in a subdirectory. language-specific config). Everything else goes in a subdirectory. Canonical
Canonical subdirectory names: subdirectory names:
- `bin/` — executable scripts and tools - `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in body is a single call into `internal/` or `pkg/`, no project logic in
@@ -750,7 +601,3 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml` - Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore` - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml` - Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+16 -224
View File
@@ -18,7 +18,7 @@
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
`next`, and `main` receives only the milestone PR that the owner merges. `next` `next`, and `main` receives only the milestone PR that the owner merges. `next`
is at the canonical `golangci-lint` v2.14.0 config and is green. Recent work is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
internal/signature packages. The gosec findings from the 2026-07-06 survey are internal/signature packages. The gosec findings from the 2026-07-06 survey are
resolved. The disk cache is now size-bounded with LRU eviction resolved. The disk cache is now size-bounded with LRU eviction
@@ -30,229 +30,6 @@ P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
addresses refuses the local-use NAT64 range (`64:ff9b:1::/48`, RFC 8215) as it
refuses `64:ff9b::/96`, and refuses an IPv6 address in the IPv4-compatible
form (`::/96`) when `blocked_networks` contains either the address as written
or the IPv4 address it holds (closes #299). A source host such as
`[64:ff9b:1::a00:1]` was fetched, which on a network that runs NAT64 on that
range reaches the IPv4 address it holds, here `10.0.0.1`; and with
`203.0.113.0/24` in `blocked_networks`, `[::ffff:203.0.113.5]` was refused but
`[::203.0.113.5]` was fetched. Both now answer 403. The mapped form
(`::ffff:a.b.c.d`) is still matched only as the IPv4 address it holds: Go
keeps a plain IPv4 address in that form, and an IPv6 entry such as `::/0` must
not refuse every IPv4 address.
- 2026-10-16 P1 security bug: the check that keeps pixa from fetching internal
addresses judges an IPv6 address in the IPv4-compatible form (`::/96`), such
as `::127.0.0.1` or `::10.0.0.1`, by the IPv4 address in its last four bytes
(closes #297). It read only the mapped form (`::ffff:a.b.c.d`) as IPv4, so a
source host such as `[::127.0.0.1]` was checked as a public IPv6 address and
fetched, which on a network that routes it reaches the internal IPv4 host; it
is now refused with 403. The NAT64 form (`64:ff9b::/96`) was already refused
as a whole, as one of the built-in blocked ranges, and still is.
- 2026-10-16 a source host that is an IPv6 address in brackets with no port,
such as `[2001:db8::a]`, is fetched like one with a port, through `/v1/image/`
and through an encrypted URL (closes #295). The fetcher's check of the source
URL kept the brackets when no port followed them, could not look the host up,
and the request answered 500. It now checks the address without brackets, so a
private one is refused with 403, as with a port.
- 2026-10-16 a `state_dir` whose name holds `%`, `?` or `#` keeps its database
at `state.sqlite3` in that directory (closes #290): the `db_url` derived from
it put the name into a `file:` URL as it was, so SQLite read `%41` as `A`, a
`?` or `#` ended the path early, and the journal mode parameter was lost. The
derived `db_url` is now built with Go's `net/url`, which writes those
characters as `%25`, `%3F` and `%23`. For a `state_dir` of `/` or one starting
with `//` it starts `file:////`, so SQLite no longer reads the first name in
the path as a host.
- 2026-10-16 a size with one side `0` keeps the side computed from the source's
proportions at 1 or more (closes #283): it was rounded down, so for a thin
source it came out at 0, and libvips then made the size square. An 8192x1
source at `100x0` came out at 100x100, and at `1300x0` with `fit=cover` made
libvips ask for about 39 GB at once; an 8192x2 source at `100x0` answered 500.
`fit=contain` and `fit=inside` work both sides out again from the smaller of
the two scales and `fit=outside` from the larger, and each keeps both sides at
1 or more too, so a 1x8192 source at `0x100` with `fit=contain`, and a 49x8192
source at `0x100` or a 49x49 source at `1x1` with `fit=outside`, no longer
answer 500. An 8192x1 source at `1300x0` now comes out at 1300x1, and the
process running it alone peaked at about 34 MiB resident. With `fit=outside`
the image is scaled to cover the size, so that source at `100x0` comes out at
8192x1. `README.md` now says this under Signature Specification, and says how
a `0` side is worked out where it said `0` meant the original.
- 2026-10-16 the generator page lower-cases the source host and writes an empty
source path as `/` before it makes the encrypted URL, as `/v1/image/` does
(closes #293). A host written in upper and in lower case, or a source URL with
no path and with the path `/`, made two cache keys and two upstream requests
for one image.
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
source host, path and query, not joined as text (closes #288). The source
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
`/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png`
fetch different paths where both fetched `/my%20cat.png` under two cache keys.
The source query stays as the source URL writes it; the characters a browser
percent-encodes in a query, such as `'` or a space, which was sent raw and
answered 502, are percent-encoded when the image URL is read, so the query is
signed and cached as it is sent. The generator page answers 400, showing the
URL pixa would fetch, for a source URL its image URL would fetch written
another way, such as with `%2F` in the path, whatever else the path holds, or
with a user name and password. It also answers 400 for a source path holding a
`..` segment, as `/v1/image/` refuses one.
- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips
build a huge image (closes #282): libvips' thumbnail scaled the whole source
to cover the requested size and copied that into memory before it cut out the
centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and
an 8192x1 source answered 500. The image processor now first cuts the centre
of the source, widened to whole source pixels, and thumbnail scales and cuts
only that, by the same amount on both sides as before. What it copies into
memory is then larger than the output by less than one scaled source pixel on
each side. With the runtime image's libvips and JPEG output at 8192x8192,
resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for
the 8192x1 one. The output can sit a fraction of an output pixel from where it
was.
- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`,
written as `%23`, is refused with 400 (closes #286): the source URL built from
it ended at the `#` when fetched, so each text after it fetched the same
upstream image, without a signature for an allowlisted host, and cached it
under keys of its own. `README.md` says neither may hold a `#`.
- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
brackets, either one optionally followed by `:` and a port, and any other host
is refused with 400 (closes #275): only `@` was refused, so a `?` written as
`%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as
`cdn.example.com` with a query. Such a URL passed the allowlist, and each text
after the `?` fetched the host's root path and cached the answer under keys of
its own. A `#`, `%`, `\` or space written percent-encoded is refused the same
way, and so is an IPv6 address without brackets, which needed a signature
before. Underscores are kept because `allowlist_hosts` accepts host names
holding them.
- 2026-10-15 the README says how a source URL with a query is written into an
image URL, and the image route accepts it (closes #255): the query follows the
source path after `%3F`. pixa now reads the image URL's path decoded once, so
the `?` before the query, any `?` in it and every `%` in the source path or
query must be percent-encoded, and any other character may be. The route used
to parse chi's wildcard, which holds the path as the client wrote it whenever
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
writes Go's own encoding of the path. The README has an example with a query
and its signature. The path traversal check no longer reads the source query,
and keeps a `%` that does not start a percent-escape as it is instead of
refusing the URL, so a source path or query holding `%25` is accepted, while a
double-encoded `..` next to such a `%` is still refused.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
`.prettierignore`, `.prettierrc`, `REPO_POLICIES.md` and the standard scripts
are its copies. pixa's own entries follow the standard part: `/bin`, `/data`
and the local config files in `.dockerignore`; Go's `*.log`, `*.out` and
`*.test`, `vendor/`, the binaries, `/data/`, `*.sqlite3` and the local config
files in `.gitignore`. The workflow keeps its `script/docker-smoke` step while
https://git.eeqj.de/sneak/pixa/issues/203 is open. The scripts pass a new
`CACHEBUST` build argument in place of `--no-cache`, and every `Dockerfile`
stage declares it first, so every `RUN` step still runs on every build while
Go's build cache, mounted on the steps that compile, is kept; both `go test`
runs keep `-count=1`. The lint and test builds write no image. The lint phase
runs golangci-lint v2.14.0, the test phase runs on the Debian Go image, and
the build stage marks `/src` safe for git. `script/bootstrap`, `script/fmt`
and `script/fmt-check` use the node on the `PATH` only when it has the pinned
major version. Of golangci-lint v2.14.0's two findings, an unused `nolint`
directive is removed, and gosec's new path check is marked as not applying to
the fixed config file locations. The agent guidance moves unchanged from
`CLAUDE.md` to a root `AGENTS.md`, where `REPO_POLICIES.md` keeps it.
- 2026-10-15 the request metrics record any method that is not a standard one as
`other` (closes #274): their `method` label was each request's method as sent,
and Go's HTTP server accepts any method name, so a client sending made-up
methods added series to the Prometheus registry that were never removed. The
label is now the request's method only for `GET`, `HEAD`, `POST`, `PUT`,
`PATCH`, `DELETE`, `OPTIONS`, `CONNECT` and `TRACE`. The metrics library reads
the method from the request it is handed, so the metrics middleware hands it a
copy of the request carrying the label; the handlers still see the method the
client sent.
- 2026-10-15 logging out ends every login session (closes #256): a login session
was a cookie with nothing kept on the server, so `/logout` removed it from one
browser and a copy of it kept working for its 30 days. Every login is made
with the one signing key, so logging out now ends them all: the session
manager keeps a random value, made with `crypto/rand` when the new `logout`
table has none and replaced by each logout, saves it there so that it outlasts
a restart, puts the value at login in each session, and refuses a session
carrying another value. The check for a logout compares no time, so a login
after a logout works whatever the host's clock does; a session's 30 days are
still checked against the clock. Losing the state directory ends every login
session, and no logout can make an ended one work again. Logging out is now
`POST /logout` from a form on the generator page, behind the CSRF check the
other forms have, so a link or image elsewhere on the same site cannot end the
sessions. A logout without a valid login session ends none and only removes
the cookie, as anyone can take a CSRF token from the login page. A logout that
cannot be saved answers 500. `session.NewManager`, which the tests use, keeps
the value in memory only; pixad uses `session.NewManagerWithDatabase`.
- 2026-10-15 a cached file left empty or partial is no longer served, or fetched
again on every request (closes #252). Source and transformed image files, and
each transformed image's `.meta` file, are synced to disk before they are
renamed into place, and their directory after; the `.meta` file is written
before the image, and a failed directory sync removes the renamed file. The
reconciliation pass, at startup and every five minutes, removes a cached
source or transformed image whose file is empty, or not the size the database
records for it, with its rows and a warning naming the file, so the next
request fetches the source or converts the image again; requests compare no
sizes. Storing a source over a file of another size replaces that file. A
source that fails to decode is not removed for that alone: a damaged upstream
image would then be fetched on every request.
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
ended, unless it has begun encoding (closes #243): the image processor looked
at its context only while it waited for a processing slot, so an image was
decoded and encoded, holding the slot, however long after its request's
deadline that took. `Process` now returns the context's error, and gives the
slot back, when the context has ended by the time it has the slot or before it
encodes. A request that waited for a shared transcode stopped at the first
request's deadline is answered 503 `server busy, try again later`, not 500,
though its own `downstream_timeout` may not have ended. govips gives pixa no
way to stop libvips once an encode has begun, so that encode keeps its slot
until it finishes; the limit of 8192 on each side of the source and of the
output bounds how long that takes. With one libvips thread, 8192x8192 random
pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as GIF, 63
as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha channel,
plus up to about 11 seconds to decode the source; `README.md` gives these
times and no longer says the work stops when `downstream_timeout` ends. The
settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot past
the default `downstream_timeout` of 60 seconds, are unchanged.
- 2026-10-15 "Running under upaas" in `README.md` tells the operator to set
`PIXA_CACHE_MAX_BYTES` (closes #251): it listed the variable as optional, but
under upaas the volume is a directory on the host's filesystem. The default,
75% of the sum of that filesystem's free space and what the cache holds, and
at least 500 MiB, let one pixa take three quarters or more of the space the
host and every other app had left, and all of it when that was 500 MiB or
less. The section now says so. The default is unchanged.
- 2026-10-15 the request metrics are labelled by route, not by path (closes
#254): their `handler` label was each request's path, so every new path, a 404
or an encrypted URL, added series to the Prometheus registry that were never
removed, and `/metrics` listed every encrypted URL's token. The label is now
the pattern of the chi route the request matches, such as `/v1/image/*`, or
`unmatched` for a request that matches none: one the router answers 404 or
405, or a CORS preflight. The metrics library takes the label before the
request is routed, so the metrics middleware asks the router which route
matches.
- 2026-10-15 expired negative cache rows are removed on a schedule (closes
#244): the row of a failed fetch, which makes pixa answer 502 for that source
URL for 5 minutes, was deleted only when the same URL was requested again
after that, so a client asking for a new missing path each time added rows
that stayed for good. The eviction goroutine now calls `CleanExpired` at
startup and on each periodic tick, every 5 minutes in pixad, so a row stays at
most about 10 minutes. `CleanExpired` deletes at most 1000 rows per statement,
so a request waits behind 1000 deletions at most, however many rows have
expired. The goroutine runs with the disk cache disabled
(`cache_max_bytes: 0`) too, where it does nothing else, as the negative cache
is used either way.
- 2026-10-15 a cached variant is served while its source URL is in the negative - 2026-10-15 a cached variant is served while its source URL is in the negative
cache (closes #245): `Get` checked the negative cache before the variant cache (closes #245): `Get` checked the negative cache before the variant
cache, so once a fetch of the source failed, as one does when another variant cache, so once a fetch of the source failed, as one does when another variant
@@ -282,6 +59,21 @@ P2: security: per-origin rate limiting
and the query now have each `%` written as `%25` and each `:` as `%3A`; a path and the query now have each `%` written as `%25` and each `:` as `%3A`; a path
and query holding neither are signed and keyed as before. Signature and query holding neither are signed and keyed as before. Signature
Specification in `README.md` says so. Specification in `README.md` says so.
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
ended, unless it has begun encoding (closes #243): the image processor looked
at its context only while it waited for a processing slot, so an image was
decoded and encoded, holding the slot, however long after its request's
deadline that took. `Process` now returns the context's error, and gives the
slot back, when the context has ended by the time it has the slot or before it
encodes. libvips cannot stop an encode once it has begun, so that encode keeps
its slot until it finishes; the limit of 8192 on each side of the source and
of the output bounds how long that takes. With one libvips thread, 8192x8192
random pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as
GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha
channel, plus up to about 11 seconds to decode the source; `README.md` gives
these times and no longer says the work stops when `downstream_timeout` ends.
The settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot
past the default `downstream_timeout` of 60 seconds, are unchanged.
- 2026-10-15 the example config allowlists no host (closes #249): it allowlisted - 2026-10-15 the example config allowlists no host (closes #249): it allowlisted
`github.com` and `user-images.githubusercontent.com`, where anyone can put an `github.com` and `user-images.githubusercontent.com`, where anyone can put an
image, so a deployment set up as Getting Started says served those images to image, so a deployment set up as Getting Started says served those images to
+4 -5
View File
@@ -108,11 +108,10 @@ upstream_connections_per_host: 20
# downstream_timeout has ended first. # downstream_timeout has ended first.
upstream_connections: 64 upstream_connections: 64
# Maximum number of images decoded and encoded at once (default: half the # Maximum number of images decoded and encoded at once (default: the
# number of CPUs pixa can use, at least 1; the number of CPUs follows a # number of CPUs pixa can use, which follows a container's CPU limit). A
# container's CPU limit). A request that finds none free waits up to 10 # request that finds none free waits up to 10 seconds for one, and if none
# seconds for one, and if none frees up it is answered 503, unless # frees up it is answered 503, unless downstream_timeout has ended first.
# downstream_timeout has ended first.
# max_concurrent_processing: 4 # max_concurrent_processing: 4
# How many requests to the image routes (/v1/image/ and /v1/e/) one client # How many requests to the image routes (/v1/image/ and /v1/e/) one client
@@ -1,7 +1,6 @@
package config package config
import ( import (
"fmt"
"runtime" "runtime"
"testing" "testing"
) )
@@ -13,8 +12,8 @@ const (
) )
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted // TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
// max_concurrent_processing is half the CPUs Go uses, and at least 1, and // max_concurrent_processing is the number of CPUs Go uses and an omitted
// an omitted upstream_connections is 64. // upstream_connections is 64.
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) { func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Parallel() t.Parallel()
@@ -23,9 +22,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Fatalf("minimal config should be valid, got error: %v", err) t.Fatalf("minimal config should be valid, got error: %v", err)
} }
if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) { if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs", t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU",
c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1)) c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0))
} }
if c.UpstreamConnections != 64 { if c.UpstreamConnections != 64 {
@@ -33,36 +32,6 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
} }
} }
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
// that an omitted max_concurrent_processing is half the CPUs Go uses,
// rounded down, and 1 when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
tc.cpus, c.MaxConcurrentProcessing)
if c.MaxConcurrentProcessing != tc.want {
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
tc.cpus, c.MaxConcurrentProcessing, tc.want)
}
}
}
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the // TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
// two limits are used as given. // two limits are used as given.
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) { func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
@@ -191,19 +160,3 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
}) })
} }
} }
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
// least 1, an empty one included, aborts startup naming the variable and
// the key instead of taking the default.
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
t.Setenv(testMaxConcurrentProcessingVar, value)
_, err := configFromYAML(t, signingKeyLine)
wantStartupError(t, err,
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
})
}
}
+5 -35
View File
@@ -37,14 +37,6 @@ const (
DefaultImageRequestsPerMinute = 600 DefaultImageRequestsPerMinute = 600
) )
// DefaultMaxConcurrentProcessing returns the most images processed at once
// when max_concurrent_processing is omitted: half the CPUs Go uses
// (runtime.GOMAXPROCS(0), which follows a container's CPU limit), rounded
// down, and at least 1.
func DefaultMaxConcurrentProcessing() int {
return max(runtime.GOMAXPROCS(0)/2, 1) //nolint:mnd // half the CPUs
}
// Configuration key names. // Configuration key names.
const ( const (
keyDebug = "debug" keyDebug = "debug"
@@ -305,8 +297,10 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost), keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
UpstreamConnections: loader.intVal( UpstreamConnections: loader.intVal(
keyUpstreamConnections, DefaultUpstreamConnections), keyUpstreamConnections, DefaultUpstreamConnections),
// Decoding and encoding are CPU-bound, so the default is one image
// per CPU Go uses, which follows a container's CPU limit.
MaxConcurrentProcessing: loader.intVal( MaxConcurrentProcessing: loader.intVal(
keyMaxConcurrentProcessing, DefaultMaxConcurrentProcessing()), keyMaxConcurrentProcessing, runtime.GOMAXPROCS(0)),
UpstreamFetchTimeout: loader.durationVal( UpstreamFetchTimeout: loader.durationVal(
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout), keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
UpstreamMaxResponseSize: loader.int64Val( UpstreamMaxResponseSize: loader.int64Val(
@@ -338,7 +332,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
settingName(keyDBURL), errValueEmpty) settingName(keyDBURL), errValueEmpty)
} }
c.DBURL = defaultDBURL(c.StateDir) // The driver sets the journal mode only through a _pragma parameter.
c.DBURL = fmt.Sprintf("file:%s/state.sqlite3?_pragma=journal_mode(WAL)", c.StateDir)
} }
if loader.err != nil { if loader.err != nil {
@@ -353,30 +348,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
return c, nil return c, nil
} }
// defaultDBURL returns the db_url used when db_url is omitted: the file
// state.sqlite3 in stateDir, in WAL mode, which the driver sets only
// through a _pragma parameter. String() percent-encodes any %, ? or # in
// the directory name, which SQLite decodes back, so the URL names exactly
// this file. OmitHost writes file:/var/lib/pixa/... instead of
// file:///var/lib/pixa/..., and keeps a relative stateDir from being read
// as a host name. The path written into the URL is stateDir followed by
// /state.sqlite3; when it starts with //, as it does for a stateDir of /
// or one starting with //, the empty host is kept, so the URL starts
// file://// and SQLite does not read the first name in the path as a
// host.
func defaultDBURL(stateDir string) string {
dbPath := stateDir + "/state.sqlite3"
dbURL := url.URL{
Scheme: "file",
OmitHost: !strings.HasPrefix(dbPath, "//"),
Path: dbPath,
RawQuery: "_pragma=journal_mode(WAL)",
}
return dbURL.String()
}
// validateKnownKeys rejects configuration files containing keys the // validateKnownKeys rejects configuration files containing keys the
// application does not understand, so typos fail at startup instead of // application does not understand, so typos fail at startup instead of
// being silently ignored, and rejects keys that are explicitly set to // being silently ignored, and rejects keys that are explicitly set to
@@ -852,7 +823,6 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
// one whose path runs through a file, such as under a HOME of // one whose path runs through a file, such as under a HOME of
// /dev/null. One that cannot be read or does not parse is a // /dev/null. One that cannot be read or does not parse is a
// fatal startup error. // fatal startup error.
//nolint:gosec // G703: a fixed config location; HOME is the operator's
_, statErr := os.Stat(cleanPath) _, statErr := os.Stat(cleanPath)
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) { if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
continue continue
@@ -135,104 +135,6 @@ func TestDefaultDBURLOpensTheDatabaseInWALMode(t *testing.T) {
} }
} }
// TestDerivedDBURLOpensStateDirHoldingURLCharacters opens the db_url
// derived from a state_dir whose name holds %41, ? and #, and checks that
// the database is state.sqlite3 in that directory, in WAL mode. Put into
// a file: URL as it is, the name would lose its meaning: %41 is read as
// A, and the path ends at the ?, which also loses the journal mode.
func TestDerivedDBURLOpensStateDirHoldingURLCharacters(t *testing.T) {
t.Parallel()
stateDir := filepath.Join(t.TempDir(), "a%41b?c#d")
err := os.Mkdir(stateDir, 0o750)
if err != nil {
t.Fatalf("failed to create %q: %v", stateDir, err)
}
// Quoted so YAML takes the whole name as it is.
c, err := configFromYAML(t, signingKeyLine+"state_dir: '"+stateDir+"'\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(stateDir, "state.sqlite3"))
}
// TestDerivedDBURLOpensStateDirStartingWithTwoSlashes opens the db_url
// derived from a state_dir that starts with //, and checks that the
// database is state.sqlite3 in that directory, in WAL mode. Written right
// after file:, such a name gives a URL starting file://, and SQLite reads
// the first directory name as a host and refuses to open it.
func TestDerivedDBURLOpensStateDirStartingWithTwoSlashes(t *testing.T) {
t.Parallel()
dir := t.TempDir()
stateDir := "/" + dir
c, err := configFromYAML(t, signingKeyLine+"state_dir: "+stateDir+"\n")
if err != nil {
t.Fatalf("config with state_dir %q should be valid, got: %v", stateDir, err)
}
t.Logf("state_dir %q gives db_url %q", stateDir, c.DBURL)
checkDBURLOpensFileInWALMode(t, c.DBURL, filepath.Join(dir, "state.sqlite3"))
}
// TestDerivedDBURLForRootStateDir checks the db_url derived from a
// state_dir of /. The path written into the URL is then
// //state.sqlite3, and written right after file: it gives a URL starting
// file://, where SQLite reads state.sqlite3 as a host and refuses to open
// it. The test only checks the URL, since it cannot write to /.
func TestDerivedDBURLForRootStateDir(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+"state_dir: /\n")
if err != nil {
t.Fatalf("config with state_dir / should be valid, got: %v", err)
}
t.Logf("state_dir / gives db_url %q", c.DBURL)
wantDBURL := "file:////state.sqlite3?_pragma=journal_mode(WAL)"
if c.DBURL != wantDBURL {
t.Errorf("DBURL = %q, want %q", c.DBURL, wantDBURL)
}
}
// checkDBURLOpensFileInWALMode opens dbURL with the SQLite driver pixad
// uses, and checks that the database is in WAL mode and is the file
// dbFile.
func checkDBURLOpensFileInWALMode(t *testing.T, dbURL, dbFile string) {
t.Helper()
db, err := sql.Open("sqlite", dbURL)
if err != nil {
t.Fatalf("failed to open %q: %v", dbURL, err)
}
t.Cleanup(func() { _ = db.Close() })
var journalMode string
err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&journalMode)
if err != nil {
t.Fatalf("failed to read the journal mode of %q: %v", dbURL, err)
}
if journalMode != "wal" {
t.Errorf("journal mode of %q = %q, want wal", dbURL, journalMode)
}
_, err = os.Stat(dbFile)
if err != nil {
t.Errorf("opening %q did not create %q: %v", dbURL, dbFile, err)
}
}
func TestExplicitValidValuesAreUsed(t *testing.T) { func TestExplicitValidValuesAreUsed(t *testing.T) {
t.Parallel() t.Parallel()
-8
View File
@@ -176,11 +176,3 @@ CREATE TABLE IF NOT EXISTS cache_stats (
last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP last_updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
); );
INSERT OR IGNORE INTO cache_stats (id) VALUES (1); INSERT OR IGNORE INTO cache_stats (id) VALUES (1);
-- A random value that a login session carries from its login. pixa makes it
-- when the table has no row, and each logout replaces it; a login session
-- carrying another value is refused, so logging out ends every login session.
CREATE TABLE IF NOT EXISTS logout (
id INTEGER PRIMARY KEY CHECK (id = 1),
logout_value TEXT NOT NULL
);
+3 -78
View File
@@ -9,9 +9,7 @@ import (
"math" "math"
"net/http" "net/http"
"net/url" "net/url"
"slices"
"strconv" "strconv"
"strings"
"time" "time"
"sneak.berlin/go/pixa/internal/clientip" "sneak.berlin/go/pixa/internal/clientip"
@@ -91,18 +89,10 @@ func (s *Handlers) handleLoginPost(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "/", http.StatusSeeOther) http.Redirect(w, r, "/", http.StatusSeeOther)
} }
// HandleLogout logs out, which ends every login session (see // HandleLogout clears the session and redirects to login.
// session.Manager.Logout), and redirects to login.
func (s *Handlers) HandleLogout() http.HandlerFunc { func (s *Handlers) HandleLogout() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
err := s.sessMgr.Logout(w, r) s.sessMgr.ClearSession(w)
if err != nil {
s.log.Error("failed to log out", "error", err)
http.Error(w, "Failed to log out", http.StatusInternalServerError)
return
}
http.Redirect(w, r, "/", http.StatusSeeOther) http.Redirect(w, r, "/", http.StatusSeeOther)
} }
} }
@@ -127,7 +117,7 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
sourceURL := r.FormValue("url") sourceURL := r.FormValue("url")
// Validate source URL // Validate source URL
parsed, err := parseSourceURL(sourceURL) parsed, err := url.Parse(sourceURL)
if err != nil || parsed.Host == "" { if err != nil || parsed.Host == "" {
s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form, s.renderGeneratorWithForm(w, r, "Invalid source URL", r.Form,
http.StatusBadRequest) http.StatusBadRequest)
@@ -135,16 +125,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return return
} }
// /v1/image/ refuses a ".." segment in the source path, and so does
// the generator. The path is decoded, so %2e%2e counts too
if slices.Contains(strings.Split(parsed.Path, "/"), "..") {
s.renderGeneratorWithForm(w, r,
"Source URL path cannot hold a .. segment", r.Form,
http.StatusBadRequest)
return
}
payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form) payload, expiresAt, ttl, err := buildGeneratePayload(parsed, r.Form)
if err != nil { if err != nil {
s.renderGeneratorWithForm(w, r, err.Error(), r.Form, s.renderGeneratorWithForm(w, r, err.Error(), r.Form,
@@ -153,19 +133,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return return
} }
// The image URL must fetch the source URL as it was written
sourceRequest := payload.ToImageRequest()
sourceRequest.AllowHTTP = s.config.AllowHTTP
fetchURL := sourceRequest.SourceURL()
if !fetchedAsWritten(parsed, fetchURL) {
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
"written: pixa would fetch "+fetchURL,
r.Form, http.StatusBadRequest)
return
}
// Generate encrypted token // Generate encrypted token
token, err := s.encGen.Generate(payload) token, err := s.encGen.Generate(payload)
if err != nil { if err != nil {
@@ -198,48 +165,6 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
} }
} }
// parseSourceURL parses the source URL entered on the generator page with its
// host lower-cased and an empty path written as "/", as /v1/image/ reads a
// source URL, so that each spelling of one source URL makes one cache key and
// one upstream request.
func parseSourceURL(sourceURL string) (*url.URL, error) {
parsed, err := url.Parse(sourceURL)
if err != nil {
return nil, err
}
parsed.Host = strings.ToLower(parsed.Host)
if parsed.Path == "" {
parsed.Path = "/"
}
return parsed, nil
}
// fetchedAsWritten reports whether fetchURL, the URL pixa fetches for an
// image URL made from the source URL source, is source as it was written,
// apart from the scheme, which the token does not keep. It is not when source
// holds a user name and password, which the token does not keep either, or a
// "?" with no query after it. The token keeps the source path decoded and
// pixa fetches it percent-encoded again, so a path written another way, such
// as with %2F, is fetched as another path. url.URL keeps the path as written
// in RawPath when it is not the decoded path percent-encoded again.
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
// such as a space, so RawPath is compared itself.
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
writtenPath := source.RawPath
if writtenPath == "" {
writtenPath = source.EscapedPath()
}
fetched, err := url.Parse(fetchURL)
return err == nil && source.User == nil && !source.ForceQuery &&
fetched.EscapedPath() == writtenPath &&
fetched.RawQuery == source.RawQuery
}
// buildGeneratePayload parses the numeric form fields and assembles the // buildGeneratePayload parses the numeric form fields and assembles the
// encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A // encrypted URL payload. ttl=0 means never expires (ExpiresAt stays 0). A
// non-numeric or out-of-range width, height, quality or ttl, or an // non-numeric or out-of-range width, height, quality or ttl, or an
@@ -185,98 +185,6 @@ func TestHandleLogout_ClearsSessionCookie(t *testing.T) {
} }
} }
// TestHandleLogout_EndsCopiedSession verifies that after POST /logout a copy
// of the session cookie taken before it shows the login page, not the
// generator page.
func TestHandleLogout_EndsCopiedSession(t *testing.T) {
t.Parallel()
h, srv := newCSRFTestRouter(t)
copied := newSessionCookie(t, h)
req := httptest.NewRequestWithContext(
t.Context(), http.MethodPost, "/logout", nil)
req.AddCookie(copied)
rec := httptest.NewRecorder()
h.HandleLogout().ServeHTTP(rec, req)
if rec.Code != http.StatusSeeOther {
t.Fatalf("POST /logout status = %d, want %d",
rec.Code, http.StatusSeeOther)
}
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(copied)
rec = httptest.NewRecorder()
srv.ServeHTTP(rec, req)
body := rec.Body.String()
t.Logf("GET / with the copied cookie: status %d, login page %v",
rec.Code, strings.Contains(body, loginForm))
if !strings.Contains(body, loginForm) || strings.Contains(body, generatorForm) {
t.Error("a cookie copied before /logout still opens the generator page")
}
}
// TestHandleLogout_SaveFails_Answers500 verifies that when the logout cannot
// be saved in the database, POST /logout answers 500, leaves the session
// cookie in place and ends no session.
func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
t.Parallel()
h, srv := newCSRFTestRouter(t)
db := setupTestDB(t)
sessMgr, err := session.NewManagerWithDatabase(
t.Context(), testSigningKey, db)
if err != nil {
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
}
h.sessMgr = sessMgr
cookie := newSessionCookie(t, h)
// Saving the logout fails on a closed database.
err = db.Close()
if err != nil {
t.Fatalf("closing the database: %v", err)
}
req := httptest.NewRequestWithContext(
t.Context(), http.MethodPost, "/logout", nil)
req.AddCookie(cookie)
rec := httptest.NewRecorder()
h.HandleLogout().ServeHTTP(rec, req)
t.Logf("POST /logout: status %d, Set-Cookie %q",
rec.Code, rec.Header().Get("Set-Cookie"))
if rec.Code != http.StatusInternalServerError {
t.Errorf("POST /logout status = %d, want %d",
rec.Code, http.StatusInternalServerError)
}
if c := findSessionCookie(rec); c != nil {
t.Errorf("a logout that failed set the session cookie: %s", c)
}
req = httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(cookie)
rec = httptest.NewRecorder()
srv.ServeHTTP(rec, req)
if !strings.Contains(rec.Body.String(), generatorForm) {
t.Error("a logout that failed ended the session")
}
}
// TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate // TestGeneratePost_NoSession_RedirectsToLogin verifies that POST /generate
// with a valid CSRF token but no login session answers 303 to / and makes no // with a valid CSRF token but no login session answers 303 to / and makes no
// URL. // URL.
+2 -6
View File
@@ -124,8 +124,7 @@ func (s *Handlers) initImageService() error {
s.imgCache = cache s.imgCache = cache
// Background eviction: startup reconciliation, then periodic and // Background eviction: startup reconciliation, then periodic and
// write-pressure passes, and the removal of expired negative cache // write-pressure passes. No-op when the disk cache is disabled.
// rows, which is all it does when the disk cache is disabled.
cache.StartEviction(imgcache.DefaultEvictionInterval) cache.StartEviction(imgcache.DefaultEvictionInterval)
// Writes the counts requests could not write by their deadline // Writes the counts requests could not write by their deadline
@@ -164,10 +163,7 @@ func (s *Handlers) initImageService() error {
// Initialize session manager (signing key is validated at config load // Initialize session manager (signing key is validated at config load
// time). Session cookies are always Secure/HttpOnly/SameSite=Strict. // time). Session cookies are always Secure/HttpOnly/SameSite=Strict.
// The random value that each logout replaces is saved in the database, sessMgr, err := session.NewManager(s.config.SigningKey)
// so it outlasts a restart.
sessMgr, err := session.NewManagerWithDatabase(
context.Background(), s.config.SigningKey, s.db.DB())
if err != nil { if err != nil {
return err return err
} }
+9 -10
View File
@@ -9,6 +9,7 @@ import (
"strconv" "strconv"
"time" "time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
@@ -79,21 +80,20 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
} }
} }
// parseImageRequest parses the image URL's path and query parameters into // parseImageRequest parses the wildcard path and query parameters into
// an ImageRequest. On invalid input it writes an error response and // an ImageRequest. On invalid input it writes an error response and
// returns false. // returns false.
func (s *Handlers) parseImageRequest( func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) { ) (*imgcache.ImageRequest, bool) {
// The path is parsed decoded once, as r.URL.Path holds it, and not from // Get the wildcard path from chi
// chi's wildcard: chi routes on the path as the client wrote it whenever pathParam := chi.URLParam(r, "*")
// that is not Go's own encoding of it, so its wildcard would still hold
// the %3F before a source query when, say, a "/" in the query is written // Parse the URL path
// as %2F parsed, err := imgcache.ParseImagePath(pathParam)
parsed, err := imgcache.ParseImageURL(r.URL.Path)
if err != nil { if err != nil {
s.log.Warn("failed to parse image URL", s.log.Warn("failed to parse image URL",
"path", r.URL.Path, "path", pathParam,
"error", err, "error", err,
) )
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest) s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
@@ -240,8 +240,7 @@ func (s *Handlers) respondImageError(
} }
if errors.Is(err, httpfetcher.ErrTooManyConnections) || if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
errors.Is(err, imageprocessor.ErrTooManyImages) || errors.Is(err, imageprocessor.ErrTooManyImages) {
errors.Is(err, imgcache.ErrProcessingTimedOut) {
s.respondError(w, "server busy, try again later", s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable) http.StatusServiceUnavailable)
@@ -264,127 +264,3 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
}) })
} }
} }
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
// host can be, and checks that each is refused with 400 naming the host as
// decoded: allowlistedHost with a character after it that a URL host cannot
// hold, percent-encoded, an IPv6 address without brackets, and brackets
// around something that is not an IPv6 address. A "?" or "#" after
// allowlistedHost would otherwise end the host when the source URL is read,
// so the request would pass the allowlist and be fetched and cached under a
// host that is not the one fetched.
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost)
tests := []struct {
name string
writtenHost string
decodedHost string
}{
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
{"no address in brackets", "[abc]", "[abc]"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
checkErrorBody(t, rec, http.StatusBadRequest,
fmt.Sprintf("invalid image URL: invalid source host: %q",
tt.decodedHost))
})
}
}
// TestHandleImage_HostForms_Served checks that the image route serves, with
// a valid signature, the image at photoPath on a host name, an IPv4 address,
// an IPv6 address in brackets, and a host followed by ":" with or without a
// port.
func TestHandleImage_HostForms_Served(t *testing.T) {
t.Parallel()
hosts := []string{
allowlistedHost,
"img_1.example.com",
"203.0.113.5",
"[2001:db8::1]",
allowlistedHost + ":8443",
"[2001:db8::1]:8443",
allowlistedHost + ":",
}
photos := newPhotoFetcher(t, hosts...)
exp := time.Now().Add(time.Hour)
for _, host := range hosts {
t.Run(host, func(t *testing.T) {
t.Parallel()
sig := photoSignature(testSigningKey, host, exp)
rec := sendGet(t, newImageRoute(t, photos),
photoURLWithSig(host, sig, exp))
if rec.Code != http.StatusOK {
t.Errorf("status = %d, want %d: %s",
rec.Code, http.StatusOK, rec.Body)
}
})
}
}
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
// source path or source query holds a "#", written as %23, and checks that
// each is refused with 400, the one signed for that source path included. The
// source URL would otherwise end at the "#" when it is fetched, so each text
// after it would fetch the image at photoPath and cache it under keys of its
// own.
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
exp := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath + "#x",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: exp,
})
tests := []struct {
name string
target string
}{
{"# in the source path",
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
{"# in the source query",
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
{"# in the source path, signed",
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, sig, exp.Unix())},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), tt.target)
checkErrorBody(t, rec, http.StatusBadRequest,
`invalid image URL: invalid image path: `+
`the source path or query holds a "#"`)
})
}
}
@@ -169,166 +169,3 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
}) })
} }
} }
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
// by the service's signed-URL generator for source URLs with a query to the
// route, through a chi router: chi routes on the path as the client wrote it
// whenever that is not Go's own encoding of it. A 502 means the signature
// passed, so the route read the source path and query that were signed.
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
query string
}{
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
{"semicolon", "a=1;b=2"},
{"comma", "crop=0,0,50,50"},
{"equals and ampersand", "a=1&b=2"},
{"percent-escape", "name=my%20cat"},
{"question mark", "next=/a?b=1"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: tt.query,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
// and writes it into the image URL in ways the README allows: the "?" before
// the query and each "%" in it percent-encoded, and every other character as
// itself or percent-encoded. A 502 means the signature passed.
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
tests := []struct {
name string
writtenQuery string
}{
{"only ? and % encoded",
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
{"every reserved character encoded",
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
{"lower-case hex digits",
"%3furl=https://origin.example.com/my%2520cat.jpg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
// source query that each hold %25, the percent-escape of a literal "%", and
// sends each to the route written as the README says, with that "%" written as
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourcePath string
sourceQuery string
// writtenPathAndQuery is the source path and query as the README says
// to write them into the image URL
writtenPathAndQuery string
}{
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
}
for _, tt := range tests {
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
+1 -2
View File
@@ -158,8 +158,7 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
case errors.Is(err, httpfetcher.ErrUpstreamTimeout): case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout) s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
case errors.Is(err, httpfetcher.ErrTooManyConnections), case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages), errors.Is(err, imageprocessor.ErrTooManyImages):
errors.Is(err, imgcache.ErrProcessingTimedOut):
s.respondError(w, "server busy, try again later", s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable) http.StatusServiceUnavailable)
case errors.Is(err, httpfetcher.ErrInvalidContentType), case errors.Is(err, httpfetcher.ErrInvalidContentType),
+12 -13
View File
@@ -49,39 +49,38 @@ func TestServerBusyAnswers503(t *testing.T) {
} }
} }
// TestProcessingTimedOutAnswers503 checks that respondImageError and // TestProcessingTimedOutAnswers503 checks that both image routes answer 503
// handleImageError, which answer the errors of the two image routes, answer // with the server busy error when the image service reports that processing
// 503 with the server busy error when the image service reports that // stopped at the deadline of the request doing it, wrapped as the service
// processing stopped at the deadline of the request doing it, wrapped as the // wraps it. A request that waited for that processing gets this error while
// service wraps it. A request that waited for that processing gets this error // its own deadline may have time left.
// while its own deadline may have time left. The request's source only goes
// into a log line, so it is left empty.
func TestProcessingTimedOutAnswers503(t *testing.T) { func TestProcessingTimedOutAnswers503(t *testing.T) {
t.Parallel() t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)} h := &Handlers{log: slog.New(slog.DiscardHandler)}
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"}
err := fmt.Errorf("%w: failed to encode: %w", err := fmt.Errorf("%w: failed to encode: %w",
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded) imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
plain := httptest.NewRecorder() plain := httptest.NewRecorder()
h.respondImageError(plain, &imgcache.ImageRequest{}, err) h.respondImageError(plain, req, err)
encrypted := httptest.NewRecorder() encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err) h.handleImageError(encrypted, err)
for name, rec := range map[string]*httptest.ResponseRecorder{ for route, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted, "/v1/image/": plain, "/v1/e/": encrypted,
} { } {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String()) t.Logf("%s: status = %d, body = %q", route, rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable { if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s: status = %d, want %d", t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusServiceUnavailable) route, rec.Code, http.StatusServiceUnavailable)
} }
if !strings.Contains(rec.Body.String(), "server busy, try again later") { if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s: body = %q, want the server busy error", t.Errorf("%s: body = %q, want the server busy error",
name, rec.Body.String()) route, rec.Body.String())
} }
} }
} }
+12 -39
View File
@@ -59,9 +59,8 @@ const (
// builtinBlockedPrefixes are internal or special-use ranges that Go's // builtinBlockedPrefixes are internal or special-use ranges that Go's
// net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not // net.IP predicates (IsPrivate, IsLinkLocalUnicast, and the like) do not
// already cover. They are always blocked, in addition to any // already cover. They are always blocked, in addition to any
// operator-supplied networks. IPv4-mapped and IPv4-compatible IPv6 // operator-supplied networks. IPv4-mapped IPv6 addresses are unmapped
// addresses are read as the IPv4 address they hold before matching, so // before matching, so these IPv4 ranges are caught in both forms.
// these IPv4 ranges are caught in all three forms.
// //
//nolint:gochecknoglobals // immutable built-in blocklist //nolint:gochecknoglobals // immutable built-in blocklist
var builtinBlockedPrefixes = []netip.Prefix{ var builtinBlockedPrefixes = []netip.Prefix{
@@ -69,7 +68,6 @@ var builtinBlockedPrefixes = []netip.Prefix{
netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments netip.MustParsePrefix("192.0.0.0/24"), // RFC 6890 IETF protocol assignments
netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range netip.MustParsePrefix("198.18.0.0/15"), // RFC 2544 benchmarking range
netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4) netip.MustParsePrefix("64:ff9b::/96"), // RFC 6052 NAT64 (maps onto IPv4)
netip.MustParsePrefix("64:ff9b:1::/48"), // RFC 8215 local-use NAT64 (maps onto IPv4)
} }
// Fetcher errors. // Fetcher errors.
@@ -475,18 +473,18 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
return ErrUnsupportedScheme return ErrUnsupportedScheme
} }
parsedURL, err := neturl.Parse(rawURL) // Parse to extract host
if err != nil { host := extractHost(rawURL)
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
}
// Hostname drops any port, and the brackets of an IPv6 address whether
// or not a port follows them
host := parsedURL.Hostname()
if host == "" { if host == "" {
return ErrInvalidHost return ErrInvalidHost
} }
// Remove port if present
h, _, err := net.SplitHostPort(host)
if err == nil {
host = h
}
// Block obvious localhost patterns // Block obvious localhost patterns
if isLocalhost(host) { if isLocalhost(host) {
return ErrSSRFBlocked return ErrSSRFBlocked
@@ -545,9 +543,6 @@ func isPrivateIP(ip net.IP) bool {
return true return true
} }
// An IPv4-compatible IPv6 address is judged as the IPv4 address it holds
ip = ipv4CompatibleToIPv4(ip)
// Check for loopback // Check for loopback
if ip.IsLoopback() { if ip.IsLoopback() {
return true return true
@@ -599,12 +594,7 @@ func isPrivateIP(ip net.IP) bool {
} }
// isBlockedIP reports whether ip is refused, either by the built-in // isBlockedIP reports whether ip is refused, either by the built-in
// internal-range check or by one of the operator-supplied prefixes. An // internal-range check or by one of the operator-supplied prefixes.
// IPv4-compatible address (in ::/96) is refused when a prefix contains the
// address as written or the IPv4 address it holds. An IPv4-mapped address (in
// ::ffff:0:0/96) is matched only as the IPv4 address it holds: Go keeps a
// plain IPv4 address in that form too, and an IPv6 prefix such as ::/0 must
// not refuse every IPv4 address.
func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool { func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
if isPrivateIP(ip) { if isPrivateIP(ip) {
return true return true
@@ -617,28 +607,11 @@ func isBlockedIP(ip net.IP, blocked []netip.Prefix) bool {
addr = addr.Unmap() addr = addr.Unmap()
// ip read as IPv4 where it is or holds an IPv4 address, otherwise the
// zero netip.Addr, which no prefix contains. Matching asIPv4 as well as
// addr refuses more only for an IPv4-compatible address.
asIPv4, _ := netip.AddrFromSlice(ipv4CompatibleToIPv4(ip).To4())
return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool { return slices.ContainsFunc(blocked, func(prefix netip.Prefix) bool {
return prefix.Contains(addr) || prefix.Contains(asIPv4) return prefix.Contains(addr)
}) })
} }
// ipv4CompatibleToIPv4 returns the IPv4 address that an IPv4-compatible IPv6
// address (in ::/96, such as ::127.0.0.1) holds in its last four bytes, and
// any other address unchanged.
func ipv4CompatibleToIPv4(ip net.IP) net.IP {
addr, ok := netip.AddrFromSlice(ip)
if ok && netip.MustParsePrefix("::/96").Contains(addr) {
return ip[12:]
}
return ip
}
// ssrfSafeDialer validates IP addresses against the built-in blocked ranges // ssrfSafeDialer validates IP addresses against the built-in blocked ranges
// before connecting. New wraps dialSSRFSafe with the operator-supplied // before connecting. New wraps dialSSRFSafe with the operator-supplied
// blocklist; this entry point enforces the built-in ranges alone. // blocklist; this entry point enforces the built-in ranges alone.
@@ -186,44 +186,6 @@ func TestIsPrivateIP(t *testing.T) {
} }
} }
// TestIsPrivateIPJudgesEmbeddedIPv4Address checks IPv6 addresses that hold an
// IPv4 address in their last four bytes: the IPv4-compatible form (::/96) and
// the NAT64 form (64:ff9b::/96). One holding a loopback, private or
// link-local IPv4 address is refused as that address is, and one holding a
// public IPv4 address in the IPv4-compatible form passes.
func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
t.Parallel()
tests := []struct {
ip string
want bool
}{
{"::127.0.0.1", true}, // loopback
{"::10.0.0.1", true}, // private
{"::169.254.169.254", true}, // link-local
{"64:ff9b::7f00:1", true}, // NAT64 form of 127.0.0.1
{"::8.8.8.8", false}, // public
}
for _, tc := range tests {
t.Run(tc.ip, func(t *testing.T) {
t.Parallel()
ip := net.ParseIP(tc.ip)
if ip == nil {
t.Fatalf("failed to parse IP %q", tc.ip)
}
got := isPrivateIP(ip)
t.Logf("isPrivateIP(%s) = %v", ip, got)
if got != tc.want {
t.Errorf("isPrivateIP(%q) = %v, want %v", tc.ip, got, tc.want)
}
})
}
}
func TestValidateURL_RejectsNonHTTPS(t *testing.T) { func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
t.Parallel() t.Parallel()
@@ -50,69 +50,3 @@ func TestImageProcessor_RefusesComputedSideAboveMaximum(t *testing.T) {
}) })
} }
} }
// TestImageProcessor_ComputedSideIsAtLeastOne verifies that, for a size with
// one side 0, the side computed from the input's proportions is 1 where it
// would round down to 0, and that Process makes that size in every fit mode
// but FitOutside, which scales the input to cover the size and so can make
// it longer. Each case checks the size targetDimensions gives before it
// calls Process: with that side at 0, an 8192x1 input at 1300x0 makes
// libvips ask for about 39 GB with FitCover.
func TestImageProcessor_ComputedSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
wantWidth, wantHeight int
}{
{"8192x1 at 100x0", 8192, 1, Size{Width: 100}, 100, 1},
{"8192x1 at 1300x0", 8192, 1, Size{Width: 1300}, 1300, 1},
{"8192x2 at 100x0", 8192, 2, Size{Width: 100}, 100, 1},
{"1x8192 at 0x100", 1, 8192, Size{Height: 100}, 1, 100},
{"1x8192 at 0x1300", 1, 8192, Size{Height: 1300}, 1, 1300},
{"2x8192 at 0x100", 2, 8192, Size{Height: 100}, 1, 100},
}
fits := []FitMode{FitCover, FitContain, FitFill, FitInside, FitOutside}
for _, tt := range tests {
for _, fit := range fits {
t.Run(tt.name+" "+string(fit), func(t *testing.T) {
t.Parallel()
width, height, err := targetDimensions(tt.size,
tt.inputWidth, tt.inputHeight)
if err != nil || width != tt.wantWidth || height != tt.wantHeight {
t.Fatalf("targetDimensions() = %dx%d, %v, want %dx%d",
width, height, err, tt.wantWidth, tt.wantHeight)
}
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: fit})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
ok := result.Width == tt.wantWidth && result.Height == tt.wantHeight
if fit == FitOutside {
ok = result.Width >= tt.wantWidth && result.Height >= tt.wantHeight
}
if !ok {
t.Errorf("Process() made %dx%d, want %dx%d (at least, for %s)",
result.Width, result.Height, tt.wantWidth, tt.wantHeight,
FitOutside)
}
})
}
}
}
@@ -1,268 +0,0 @@
package imageprocessor
import (
"bytes"
"image"
"image/color"
"image/png"
"math"
"testing"
)
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
// a thin source at the requested size. An 8192x1 source scaled to cover
// 8192x8192 is 8192 times as long as the output on its long side. The centre
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
// wide. Each case is also turned on its side.
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
}{
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.size.Width || result.Height != tt.size.Height {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.size.Width, tt.size.Height)
}
})
}
}
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
// side by side when it is wider than tall and top to bottom otherwise.
func createBandsPNG(t *testing.T, width, height int) []byte {
t.Helper()
bands := []color.RGBA{
{R: 255, A: 255},
{G: 255, A: 255},
{B: 255, A: 255},
}
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
band := 3 * y / height
if width > height {
band = 3 * x / width
}
img.Set(x, y, bands[band])
}
}
return encodePNG(t, img)
}
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
// centre of the source: a 300x100 source of red, green and blue bands side
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
// its green band.
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
}{
{"bands side by side", 300, 100},
{"bands top to bottom", 100, 300},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{
Size: Size{Width: 50, Height: 50},
Format: FormatPNG,
FitMode: FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
if bounds.Dx() != 50 || bounds.Dy() != 50 {
t.Fatalf("Process() made %dx%d, want 50x50",
bounds.Dx(), bounds.Dy())
}
// Resampling may blend a little of the bands beside the
// green one into the output's edges.
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
for x := bounds.Min.X; x < bounds.Max.X; x++ {
r, g, b, _ := output.At(x, y).RGBA()
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
x, y, r>>8, g>>8, b>>8)
}
}
}
})
}
}
// The greys of createSquarePNG, and the grey halfway between them.
const (
backgroundGrey = 192
squareGrey = 64
halfwayGrey = (backgroundGrey + squareGrey) / 2
)
// createSquarePNG creates a grey PNG side pixels square with a darker grey
// square squareSide pixels wide in its centre.
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
t.Helper()
first := (side - squareSide) / 2
last := first + squareSide - 1
img := image.NewGray(image.Rect(0, 0, side, side))
for y := range side {
for x := range side {
grey := uint8(backgroundGrey)
if x >= first && x <= last && y >= first && y <= last {
grey = squareGrey
}
img.SetGray(x, y, color.Gray{Y: grey})
}
}
return encodePNG(t, img)
}
// squareLength returns how long the square is along greys, the greys of a
// row or column of the output that crosses it: the distance between the
// first and the last point where greys passes halfwayGrey, or 0 when there
// is no such point. Resampling blurs the square's edges over several
// pixels, and each point is placed between the two pixels on either side
// of it in proportion to their greys, so the length is measured to a
// fraction of a pixel.
func squareLength(greys []float64) float64 {
first, last := -1.0, -1.0
for i := 1; i < len(greys); i++ {
before, after := greys[i-1], greys[i]
if (before > halfwayGrey) == (after > halfwayGrey) {
continue
}
point := float64(i-1) + (before-halfwayGrey)/(before-after)
if first < 0 {
first = point
}
last = point
}
return last - first
}
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
// both sides of the source by the same amount: a square in the centre of
// the source comes out square. Its width and height are measured along the
// output's middle row and column.
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
t.Parallel()
tests := []struct {
name string
side, squareSide int
size Size
}{
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createSquarePNG(t, tt.side, tt.squareSide)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
t.Fatalf("Process() made %dx%d, want %dx%d",
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
}
row := make([]float64, bounds.Dx())
for x := range row {
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
row[x] = float64(r >> 8)
}
column := make([]float64, bounds.Dy())
for y := range column {
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
column[y] = float64(r >> 8)
}
width, height := squareLength(row), squareLength(column)
t.Logf("square is %.2f wide and %.2f high", width, height)
if width == 0 || math.Abs(width-height) > 0.5 {
t.Errorf("square is %.2f wide and %.2f high, want it square",
width, height)
}
})
}
}
@@ -60,45 +60,3 @@ func TestImageProcessor_RefusesFitOutsideAboveMaximum(t *testing.T) {
}) })
} }
} }
// TestImageProcessor_FitOutsideSideIsAtLeastOne verifies that FitOutside
// keeps each side at 1 or more when it works both sides out again from the
// larger scale. A side of 49 scaled to 1 is 49 times 1/49, which in floating
// point is just under 1 and so rounded down to 0, a width libvips refuses.
func TestImageProcessor_FitOutsideSideIsAtLeastOne(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
wantWidth, wantHeight int
}{
{"49x8192 at 0x100", 49, 8192, Size{Height: 100}, 1, 167},
{"49x49 at 1x1", 49, 49, Size{Width: 1, Height: 1}, 1, 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitOutside})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.wantWidth || result.Height != tt.wantHeight {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.wantWidth, tt.wantHeight)
}
})
}
}
+31 -72
View File
@@ -7,11 +7,11 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"runtime"
"sync" "sync"
"time" "time"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/config"
) )
// vipsOnce ensures vips is initialized exactly once. // vipsOnce ensures vips is initialized exactly once.
@@ -20,10 +20,11 @@ import (
var vipsOnce sync.Once var vipsOnce sync.Once
// initVips initializes libvips with quiet logging, one worker thread per // initVips initializes libvips with quiet logging, one worker thread per
// image and no operation cache. Each request decodes different source // image and no operation cache. Process already works on one image per CPU
// bytes, so the operation cache would rarely be hit and would hold memory // by default, so more threads per image would only compete for the CPUs.
// outside MaxConcurrentProcessing; repeated requests are served from // Each request decodes different source bytes, so the operation cache
// pixa's disk cache instead. // would rarely be hit and would hold memory outside MaxConcurrentProcessing;
// repeated requests are served from pixa's disk cache instead.
func initVips() { func initVips() {
vipsOnce.Do(func() { vipsOnce.Do(func() {
vips.LoggingSettings(nil, vips.LogLevelError) vips.LoggingSettings(nil, vips.LogLevelError)
@@ -178,7 +179,7 @@ type Params struct {
// If <= 0, DefaultMaxInputBytes is used. // If <= 0, DefaultMaxInputBytes is used.
MaxInputBytes int64 MaxInputBytes int64
// MaxConcurrentProcessing is the most images processed at once. // MaxConcurrentProcessing is the most images processed at once.
// If <= 0, config.DefaultMaxConcurrentProcessing() is used. // If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used.
MaxConcurrentProcessing int MaxConcurrentProcessing int
} }
@@ -194,7 +195,7 @@ func New(params Params) *ImageProcessor {
maxConcurrentProcessing := params.MaxConcurrentProcessing maxConcurrentProcessing := params.MaxConcurrentProcessing
if maxConcurrentProcessing <= 0 { if maxConcurrentProcessing <= 0 {
maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing() maxConcurrentProcessing = runtime.GOMAXPROCS(0)
} }
return &ImageProcessor{ return &ImageProcessor{
@@ -208,9 +209,8 @@ func New(params Params) *ImageProcessor {
// MaxConcurrentProcessing images are already being processed, it waits up // MaxConcurrentProcessing images are already being processed, it waits up
// to ProcessingWaitTimeout for one to finish, then fails with // to ProcessingWaitTimeout for one to finish, then fails with
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it // ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa // encodes, it returns ctx's error and gives the slot back. libvips cannot stop
// no way to stop libvips once an encode has begun, so an encode keeps its // an encode once it has begun, so an encode keeps its slot until it finishes.
// slot until it finishes.
func (p *ImageProcessor) Process( func (p *ImageProcessor) Process(
ctx context.Context, ctx context.Context,
input io.Reader, input io.Reader,
@@ -311,10 +311,8 @@ func (p *ImageProcessor) Process(
// targetDimensions calculates the output dimensions for a requested size, // targetDimensions calculates the output dimensions for a requested size,
// scaling proportionally when only one dimension is given and keeping the // scaling proportionally when only one dimension is given and keeping the
// original dimensions when both are zero. A side scaled proportionally is // original dimensions when both are zero. A side scaled proportionally can
// rounded down but kept at 1 or more, as for a thin image it would round // come out above MaxOutputDimension, which is ErrOutputTooLarge.
// down to 0, a side of no pixels. It can come out above MaxOutputDimension,
// which is ErrOutputTooLarge.
func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) { func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
width, height := size.Width, size.Height width, height := size.Width, size.Height
@@ -324,10 +322,10 @@ func targetDimensions(size Size, origWidth, origHeight int) (int, int, error) {
return origWidth, origHeight, nil return origWidth, origHeight, nil
case width == 0: case width == 0:
// Only height specified: calculate width proportionally // Only height specified: calculate width proportionally
width = max(1, origWidth*height/origHeight) width = origWidth * height / origHeight
case height == 0: case height == 0:
// Only width specified: calculate height proportionally // Only width specified: calculate height proportionally
height = max(1, origHeight*width/origWidth) height = origHeight * width / origWidth
} }
if width > MaxOutputDimension || height > MaxOutputDimension { if width > MaxOutputDimension || height > MaxOutputDimension {
@@ -477,20 +475,17 @@ func (p *ImageProcessor) resize(
) error { ) error {
switch fit { switch fit {
case FitCover, "": case FitCover, "":
// Crop the centre and resize to fill exact dimensions (default) // Resize and crop to fill exact dimensions (default)
return cropCentreAndResize(img, width, height) return img.Thumbnail(width, height, vips.InterestingCentre)
case FitContain: case FitContain:
// Resize to fit within dimensions, maintaining aspect ratio. Scaled // Resize to fit within dimensions, maintaining aspect ratio
// by the smaller scale, the short side of a thin image can round
// down to 0, which libvips refuses as a width, so each side is at
// least 1.
imgW, imgH := img.Width(), img.Height() imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW) scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH) scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH) scale := min(scaleW, scaleH)
newW := max(1, int(float64(imgW)*scale)) newW := int(float64(imgW) * scale)
newH := max(1, int(float64(imgH)*scale)) newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone) return img.Thumbnail(newW, newH, vips.InterestingNone)
@@ -508,22 +503,19 @@ func (p *ImageProcessor) resize(
scaleW := float64(width) / float64(imgW) scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH) scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH) scale := min(scaleW, scaleH)
newW := max(1, int(float64(imgW)*scale)) newW := int(float64(imgW) * scale)
newH := max(1, int(float64(imgH)*scale)) newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone) return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitOutside: case FitOutside:
// Resize so smallest dimension fits, may exceed target on other // Resize so smallest dimension fits, may exceed target on other dimension
// dimension. A side scaled to 1 can come out just under 1 in
// floating point, as 49 times 1/49 does, and round down to 0, which
// libvips refuses as a width, so each side is at least 1.
imgW, imgH := img.Width(), img.Height() imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW) scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH) scaleH := float64(height) / float64(imgH)
scale := max(scaleW, scaleH) scale := max(scaleW, scaleH)
newW := max(1, int(float64(imgW)*scale)) newW := int(float64(imgW) * scale)
newH := max(1, int(float64(imgH)*scale)) newH := int(float64(imgH) * scale)
// The dimension that exceeds the target is held to the maximum too // The dimension that exceeds the target is held to the maximum too
if newW > MaxOutputDimension || newH > MaxOutputDimension { if newW > MaxOutputDimension || newH > MaxOutputDimension {
@@ -538,43 +530,11 @@ func (p *ImageProcessor) resize(
} }
} }
// cropCentreAndResize scales img by the same amount on both sides, the
// least that makes it cover width x height, and cuts the centre width x
// height out of it, as libvips' thumbnail does with InterestingCentre.
// thumbnail alone scales the whole image and copies the result into memory
// before it cuts, and for a thin image that is far larger than the image or
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
// centre is cut from img first, widened to whole pixels of img, which adds
// less than one pixel of img on each side, and thumbnail scales and cuts
// only that.
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
imgW, imgH := img.Width(), img.Height()
// An image wider than the requested proportions keeps its height and
// loses width down to imgH*width/height, any other keeps its width and
// loses height down to imgW*height/width. Half of what is lost comes
// off each side, rounded down to whole pixels, so the part kept holds
// all of the centre.
left, top := 0, 0
if imgW*height > imgH*width {
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
} else {
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
}
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
if err != nil {
return err
}
return img.Thumbnail(width, height, vips.InterestingCentre)
}
const defaultQuality = 85 const defaultQuality = 85
// encode encodes an image to the specified format. govips gives pixa no way // encode encodes an image to the specified format. libvips cannot stop an
// to stop libvips once an encode has begun, so when ctx has ended encode // encode once it has begun, so when ctx has ended encode returns ctx's error
// returns ctx's error without beginning one. // without beginning one.
func (p *ImageProcessor) encode( func (p *ImageProcessor) encode(
ctx context.Context, img *vips.ImageRef, format Format, quality int, ctx context.Context, img *vips.ImageRef, format Format, quality int,
) ([]byte, error) { ) ([]byte, error) {
@@ -684,11 +644,10 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest // avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51 // govips can set. With one thread, as pixad runs libvips, 1 takes about 51
// seconds to save an 8192x8192 RGB image of random pixels, and about 110 // seconds to save an 8192x8192 image of random pixels, the worst case,
// with an alpha channel, the worst case, against the default // against the default downstream_timeout of 60 seconds. On an image of
// downstream_timeout of 60 seconds. On an image of milder noise, which 1 // milder noise, which 1 saves in about 12 seconds, 2 takes nearly a minute
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default, // and libvips' default, 4, takes minutes.
// 4, takes minutes.
const avifEffort = 1 const avifEffort = 1
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image. // avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
@@ -115,14 +115,14 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
} }
} }
func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) { func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
t.Parallel() t.Parallel()
for _, limit := range []int{0, -1} { for _, limit := range []int{0, -1} {
proc := New(Params{MaxConcurrentProcessing: limit}) proc := New(Params{MaxConcurrentProcessing: limit})
if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) { if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs", t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
limit, got, max(runtime.GOMAXPROCS(0)/2, 1)) limit, got, runtime.GOMAXPROCS(0))
} }
} }
@@ -132,30 +132,6 @@ func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
} }
} }
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
// rounded down, and one slot when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
got := cap(New(Params{}).processingSemaphore)
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
if got != tc.want {
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
}
}
}
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls // TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
// than MaxConcurrentProcessing allows and holds the first ones inside // than MaxConcurrentProcessing allows and holds the first ones inside
// Process until the test lets them go. No more than the limit may be // Process until the test lets them go. No more than the limit may be
+12 -53
View File
@@ -32,12 +32,6 @@ const httpStatusOK = 200
// variant not among them is served as before, reading its .meta file. // variant not among them is served as before, reading its .meta file.
const metaCacheSize = 10000 const metaCacheSize = 10000
// cleanExpiredBatchSize is the most expired negative cache rows one
// statement of CleanExpired deletes. Each batch is a statement of its
// own, so a request waits behind one batch at most, however many rows
// have expired.
const cleanExpiredBatchSize = 1000
// CacheConfig holds cache configuration. // CacheConfig holds cache configuration.
type CacheConfig struct { type CacheConfig struct {
StateDir string StateDir string
@@ -56,9 +50,8 @@ type CacheConfig struct {
// DisableDiskCache turns the disk cache off entirely: no cache // DisableDiskCache turns the disk cache off entirely: no cache
// directories are created, lookups always miss, stores are // directories are created, lookups always miss, stores are
// no-ops, and the eviction goroutine only removes the expired // no-ops, and no eviction machinery runs. The config layer sets
// negative cache rows. The config layer sets this when the // this when the operator configures cache_max_bytes: 0.
// operator configures cache_max_bytes: 0.
DisableDiskCache bool DisableDiskCache bool
// Logger receives accounting and eviction log output. A nil // Logger receives accounting and eviction log output. A nil
@@ -489,27 +482,17 @@ func (c *Cache) GetSourceContent(
return c.srcContent.LoadWithSize(contentHash) return c.srcContent.LoadWithSize(contentHash)
} }
// CleanExpired removes the negative cache rows past their expiry, one // CleanExpired removes expired entries from the cache.
// batch per statement, until none is left. Once ctx ends it starts no
// further batch and returns ctx's error. The eviction goroutine calls it
// at startup and on every periodic tick, so a row stays about
// NegativeTTL plus one eviction interval, whether or not its URL is
// requested again.
func (c *Cache) CleanExpired(ctx context.Context) error { func (c *Cache) CleanExpired(ctx context.Context) error {
for { // Clean expired negative cache entries
if ctx.Err() != nil { _, err := c.db.ExecContext(ctx, `
return ctx.Err() DELETE FROM negative_cache WHERE expires_at < CURRENT_TIMESTAMP
} `)
deleted, err := c.cleanExpiredBatch(ctx)
if err != nil { if err != nil {
return err return fmt.Errorf("failed to clean negative cache: %w", err)
} }
if deleted < cleanExpiredBatchSize {
return nil return nil
}
}
} }
// Stats returns cache statistics. // Stats returns cache statistics.
@@ -660,10 +643,10 @@ func (c *Cache) readHitAndMissCounts(ctx context.Context, stats *CacheStats) err
// hold: it reads the content type from the variant's .meta file and // hold: it reads the content type from the variant's .meta file and
// keeps it in metaCache, unless a StoreVariant has put one there // keeps it in metaCache, unless a StoreVariant has put one there
// meanwhile, as the store's is newer. A read that finds no .meta file, // meanwhile, as the store's is newer. A read that finds no .meta file,
// as one can when the store's .meta write failed or an eviction removed // as one can between a store's writing of the variant file and of its
// the file after the read opened the variant, serves // .meta file, serves application/octet-stream and keeps nothing, so
// application/octet-stream and keeps nothing, so metaCache only ever // metaCache only ever holds a type read from a .meta file or passed to
// holds a type read from a .meta file or passed to StoreVariant. // StoreVariant.
func (c *Cache) loadVariantWithMeta( func (c *Cache) loadVariantWithMeta(
cacheKey VariantKey, cacheKey VariantKey,
) (io.ReadCloser, int64, string, error) { ) (io.ReadCloser, int64, string, error) {
@@ -773,27 +756,3 @@ func (c *Cache) checkNegativeCache(
return true, nil return true, nil
} }
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
// rows past their expiry in one statement, and returns how many it
// deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery
// picks the rows.
func (c *Cache) cleanExpiredBatch(ctx context.Context) (int64, error) {
result, err := c.db.ExecContext(ctx, `
DELETE FROM negative_cache WHERE id IN (
SELECT id FROM negative_cache
WHERE expires_at < CURRENT_TIMESTAMP
LIMIT ?
)
`, cleanExpiredBatchSize)
if err != nil {
return 0, fmt.Errorf("failed to clean negative cache: %w", err)
}
deleted, err := result.RowsAffected()
if err != nil {
return 0, fmt.Errorf("failed to count cleaned negative cache rows: %w", err)
}
return deleted, nil
}
+4
View File
@@ -78,6 +78,10 @@ func (c *Cache) computeDefaultMaxBytes(
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
computed = min(computed, math.MaxInt64) computed = min(computed, math.MaxInt64)
// gosec cannot see that min() above bounds computed, so it reads
// this conversion as potentially overflowing. It cannot: computed is
// at most math.MaxInt64 on every path here.
//nolint:gosec // G115: clamped to MaxInt64 by min above
limit := int64(computed) limit := int64(computed)
limit = max(limit, DefaultCacheMaxBytesFloor) limit = max(limit, DefaultCacheMaxBytesFloor)
+53 -114
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"database/sql" "database/sql"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io/fs" "io/fs"
"os" "os"
@@ -438,16 +437,14 @@ func (c *Cache) notifyWritePressure() {
} }
// StartEviction launches the background eviction goroutine, which // StartEviction launches the background eviction goroutine, which
// reconciles the database accounting with the cache directories and // reconciles the database accounting with the cache directories at
// removes the expired negative cache rows at startup and again on every // startup and again on every periodic tick thereafter, and evicts to
// periodic tick thereafter, and evicts to the configured limit on the // the configured limit on the given periodic interval and on
// given periodic interval and on write-pressure notifications. On a // write-pressure notifications. It is a no-op on a disabled cache or
// disabled cache it only removes the expired negative cache rows, as // when already started. The goroutine outlives the caller, so it runs
// the negative cache is used either way. It is a no-op when already // with its own context, which StopEviction cancels.
// started. The goroutine outlives the caller, so it runs with its own
// context, which StopEviction cancels.
func (c *Cache) StartEviction(interval time.Duration) { func (c *Cache) StartEviction(interval time.Duration) {
if c.evictionCancel != nil { if c.disabled || c.evictionCancel != nil {
return return
} }
@@ -483,7 +480,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
defer close(c.evictionDone) defer close(c.evictionDone)
c.runReconciliationPass(ctx) c.runReconciliationPass(ctx)
c.runCleanExpired(ctx)
c.runEvictionPass(ctx) c.runEvictionPass(ctx)
ticker := time.NewTicker(interval) ticker := time.NewTicker(interval)
@@ -504,7 +500,6 @@ func (c *Cache) evictionLoop(ctx context.Context, interval time.Duration) {
// unaccounted for to one interval, on a process that is // unaccounted for to one interval, on a process that is
// already running this loop regardless. // already running this loop regardless.
c.runReconciliationPass(ctx) c.runReconciliationPass(ctx)
c.runCleanExpired(ctx)
case <-c.evictionPressure: case <-c.evictionPressure:
} }
@@ -540,20 +535,6 @@ func (c *Cache) runReconciliationPass(ctx context.Context) {
} }
} }
// runCleanExpired runs CleanExpired, logging a failure instead of
// propagating it (the loop must keep running). It does nothing once ctx
// is cancelled.
func (c *Cache) runCleanExpired(ctx context.Context) {
if ctx.Err() != nil {
return
}
err := c.CleanExpired(ctx)
if err != nil {
c.log.Warn("negative cache cleanup failed", "error", err)
}
}
// reconcileAccounting synchronizes the database size accounting with // reconcileAccounting synchronizes the database size accounting with
// the actual contents of the cache directories. It runs at startup and // the actual contents of the cache directories. It runs at startup and
// again on every periodic eviction tick thereafter, off the request // again on every periodic eviction tick thereafter, off the request
@@ -561,11 +542,10 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// (or whose accounting insert failed, e.g. StoreVariant's best-effort // (or whose accounting insert failed, e.g. StoreVariant's best-effort
// insert under transient DB contention), drops accounting rows whose // insert under transient DB contention), drops accounting rows whose
// files are missing, removes source blob files the database does not // files are missing, removes source blob files the database does not
// know (and rows whose files are gone), removes source and variant files // know (and rows whose files are gone), sweeps stale temp files left
// that are empty or not the size their rows record, sweeps stale temp // behind by crashed writes, and last checks the total cache usage
// files left behind by crashed writes, and last checks the total cache // against the tables. It reads the tables a page at a time. Running it
// usage against the tables. It reads the tables a page at a time. // periodically, not just once, bounds how long such drift can
// Running it periodically, not just once, bounds how long such drift can
// accumulate unaccounted for on a long-running process to one eviction // accumulate unaccounted for on a long-running process to one eviction
// interval. Once ctx is cancelled, it stops at the next file or row and // interval. Once ctx is cancelled, it stops at the next file or row and
// returns ctx's error. // returns ctx's error.
@@ -597,9 +577,8 @@ func (c *Cache) reconcileAccounting(ctx context.Context) error {
return c.reconcileUsageTotal(ctx) return c.reconcileUsageTotal(ctx)
} }
// reconcileVariantFiles walks the variant storage directory, checking // reconcileVariantFiles walks the variant storage directory, adopting
// each variant file against its accounting row (reconcileVariantFile) // files without accounting rows and sweeping stale temp files.
// and sweeping stale temp files.
func (c *Cache) reconcileVariantFiles(ctx context.Context) error { func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return filepath.WalkDir( return filepath.WalkDir(
c.variants.baseDir, c.variants.baseDir,
@@ -624,55 +603,38 @@ func (c *Cache) reconcileVariantFiles(ctx context.Context) error {
return nil return nil
} }
return c.reconcileVariantFile(ctx, path, entry, VariantKey(name)) return c.adoptVariantFile(ctx, path, entry, VariantKey(name))
}, },
) )
} }
// reconcileVariantFile checks one variant file against its accounting
// row. A file without a row is adopted into the accounting, unless it is
// empty. A file that is empty, or not the size its row records, as a
// crash or a restored snapshot can leave it, is removed with its row and
// a warning naming it, so the variant is converted again.
func (c *Cache) reconcileVariantFile(
ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat variant file: %w", err)
}
var recordedSize int64
err = c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM variant_content WHERE cache_key = ?`, string(cacheKey),
).Scan(&recordedSize)
switch {
case errors.Is(err, sql.ErrNoRows) && info.Size() > 0:
return c.adoptVariantFile(ctx, path, info, cacheKey)
case err != nil && !errors.Is(err, sql.ErrNoRows):
return fmt.Errorf("failed to check variant accounting row: %w", err)
case info.Size() > 0 && info.Size() == recordedSize:
return nil
}
// The file is empty, or not the size its row records
c.log.Warn("removing damaged cached variant file", "path", path,
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
return c.evictVariant(ctx, cacheKey)
}
// adoptVariantFile inserts an accounting row for a variant file that // adoptVariantFile inserts an accounting row for a variant file that
// has none, using the file's size and modification time. // has none, using the file's size and modification time.
func (c *Cache) adoptVariantFile( func (c *Cache) adoptVariantFile(
ctx context.Context, path string, info fs.FileInfo, cacheKey VariantKey, ctx context.Context, path string, entry fs.DirEntry, cacheKey VariantKey,
) error { ) error {
var rowExists int
err := c.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`, string(cacheKey),
).Scan(&rowExists)
if err != nil {
return fmt.Errorf("failed to check variant accounting row: %w", err)
}
if rowExists > 0 {
return nil
}
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat variant file: %w", err)
}
modTime := info.ModTime().UTC().Format(sqliteTimestampLayout) modTime := info.ModTime().UTC().Format(sqliteTimestampLayout)
contentType := c.variantContentTypeFromSidecar(path) contentType := c.variantContentTypeFromSidecar(path)
_, err := c.db.ExecContext(ctx, ` _, err = c.db.ExecContext(ctx, `
INSERT INTO variant_content INSERT INTO variant_content
(cache_key, size_bytes, content_type, created_at, last_accessed_at) (cache_key, size_bytes, content_type, created_at, last_accessed_at)
VALUES (?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?)
@@ -791,9 +753,10 @@ func queryStringColumn[T ~string](
return values, nil return values, nil
} }
// reconcileSourceFiles walks the source content directory, checking // reconcileSourceFiles walks the source content directory, removing
// each blob file against its source_content row (reconcileSourceFile) // blob files the database does not track (they are unreachable: source
// and sweeping stale temp files. // lookups always go through source_metadata) and sweeping stale temp
// files.
func (c *Cache) reconcileSourceFiles(ctx context.Context) error { func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return filepath.WalkDir( return filepath.WalkDir(
c.srcContent.baseDir, c.srcContent.baseDir,
@@ -814,55 +777,31 @@ func (c *Cache) reconcileSourceFiles(ctx context.Context) error {
return nil return nil
} }
return c.reconcileSourceFile(ctx, path, entry, ContentHash(name)) return c.removeUntrackedSourceFile(ctx, path, ContentHash(name))
}, },
) )
} }
// reconcileSourceFile checks one source blob file against its
// source_content row. A file without a row is removed: it is
// unreachable, as source lookups always go through source_metadata. A
// file that is empty, or not the size its row records, as a crash or a
// restored snapshot can leave it, is removed with its rows and a warning
// naming it, so the source is fetched again.
func (c *Cache) reconcileSourceFile(
ctx context.Context, path string, entry fs.DirEntry, contentHash ContentHash,
) error {
info, err := entry.Info()
if err != nil {
return fmt.Errorf("failed to stat source content file: %w", err)
}
var recordedSize int64
err = c.db.QueryRowContext(ctx,
`SELECT size_bytes FROM source_content WHERE content_hash = ?`,
string(contentHash),
).Scan(&recordedSize)
switch {
case errors.Is(err, sql.ErrNoRows):
return c.removeUntrackedSourceFile(ctx, path, contentHash)
case err != nil:
return fmt.Errorf("failed to check source content row: %w", err)
case info.Size() > 0 && info.Size() == recordedSize:
return nil
}
// The file is empty, or not the size its row records
c.log.Warn("removing damaged cached source file", "path", path,
"file_bytes", info.Size(), "recorded_bytes", recordedSize)
return c.evictSourceBlob(ctx, contentHash)
}
// removeUntrackedSourceFile deletes a source blob file that has no // removeUntrackedSourceFile deletes a source blob file that has no
// source_content row. Any source_metadata rows referencing the hash // source_content row. Any source_metadata rows referencing the hash
// are removed first so no row ever points at a deleted file. // are removed first so no row ever points at a deleted file.
func (c *Cache) removeUntrackedSourceFile( func (c *Cache) removeUntrackedSourceFile(
ctx context.Context, path string, contentHash ContentHash, ctx context.Context, path string, contentHash ContentHash,
) error { ) error {
_, err := c.db.ExecContext(ctx, var rowExists int
err := c.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM source_content WHERE content_hash = ?`, string(contentHash),
).Scan(&rowExists)
if err != nil {
return fmt.Errorf("failed to check source content row: %w", err)
}
if rowExists > 0 {
return nil
}
_, err = c.db.ExecContext(ctx,
`DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash)) `DELETE FROM source_metadata WHERE content_hash = ?`, string(contentHash))
if err != nil { if err != nil {
return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err) return fmt.Errorf("failed to delete metadata rows for untracked blob: %w", err)
+8 -16
View File
@@ -5,7 +5,6 @@ import (
"context" "context"
"errors" "errors"
"io" "io"
"net/url"
"time" "time"
) )
@@ -87,11 +86,9 @@ func ValidateImageRequest(req *ImageRequest) error {
type ImageRequest struct { type ImageRequest struct {
// SourceHost is the origin host (e.g., "cdn.example.com") // SourceHost is the origin host (e.g., "cdn.example.com")
SourceHost string SourceHost string
// SourcePath is the path on the origin with its percent-escapes decoded, // SourcePath is the path on the origin (e.g., "/photos/cat.jpg")
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
SourcePath string SourcePath string
// SourceQuery is the optional query string for the origin URL as the URL // SourceQuery is the optional query string for the origin URL
// writes it, percent-escapes included, as url.URL's RawQuery holds it
SourceQuery string SourceQuery string
// Size is the requested output dimensions // Size is the requested output dimensions
Size Size Size Size
@@ -110,25 +107,20 @@ type ImageRequest struct {
AllowHTTP bool AllowHTTP bool
} }
// SourceURL returns the full upstream URL to fetch: the source path // SourceURL returns the full upstream URL to fetch.
// percent-encoded again, as net/url writes a path, so each source path is one // Uses http:// scheme when AllowHTTP is true, otherwise https://.
// upstream path, and the source query as it is, with the characters
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
// true, otherwise https://.
func (r *ImageRequest) SourceURL() string { func (r *ImageRequest) SourceURL() string {
scheme := "https" scheme := "https"
if r.AllowHTTP { if r.AllowHTTP {
scheme = "http" scheme = "http"
} }
sourceURL := url.URL{ url := scheme + "://" + r.SourceHost + r.SourcePath
Scheme: scheme, if r.SourceQuery != "" {
Host: r.SourceHost, url += "?" + r.SourceQuery
Path: r.SourcePath,
RawQuery: escapeQuery(r.SourceQuery),
} }
return sourceURL.String() return url
} }
// ImageResponse represents a processed image ready to serve // ImageResponse represents a processed image ready to serve
@@ -3,7 +3,6 @@ package imgcache
import ( import (
"context" "context"
"errors" "errors"
"strconv"
"testing" "testing"
"time" "time"
) )
@@ -160,242 +159,6 @@ func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
} }
} }
// TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch
// that expires a second later and starts the eviction loop with a short
// interval: a later tick must remove its row although its URL is never
// requested again, and must keep a row that has not expired. It runs
// with the disk cache enabled and disabled, as the negative cache is
// used either way.
func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
t.Parallel()
tests := []struct {
name string
disableDiskCache bool
}{
{name: "disk cache enabled", disableDiskCache: false},
{name: "disk cache disabled", disableDiskCache: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{
StateDir: t.TempDir(),
NegativeTTL: time.Second,
DisableDiskCache: tt.disableDiskCache,
})
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
expired := &ImageRequest{
SourceHost: testHostExample, SourcePath: "/gone.jpg",
}
err = cache.StoreNegative(t.Context(), expired, 404, "not found")
if err != nil {
t.Fatalf("StoreNegative failed: %v", err)
}
const notExpiredPath = "/later.jpg"
_, err = cache.db.ExecContext(t.Context(), `
INSERT INTO negative_cache
(source_host, source_path, status_code, expires_at)
VALUES (?, ?, 404, datetime('now', '+1 hour'))
`, testHostExample, notExpiredPath)
if err != nil {
t.Fatalf("failed to insert a row that has not expired: %v", err)
}
started := time.Now()
cache.StartEviction(100 * time.Millisecond)
defer func() { _ = cache.StopEviction(t.Context()) }()
for negativeCacheRowCount(t, cache, expired.SourcePath) > 0 {
if time.Since(started) > 10*time.Second {
t.Fatal("the expired row is still in negative_cache")
}
time.Sleep(50 * time.Millisecond)
}
t.Logf("the expired row was removed %s after the loop started",
time.Since(started))
if negativeCacheRowCount(t, cache, notExpiredPath) != 1 {
t.Error("the row that has not expired was removed")
}
})
}
}
// negativeCacheRowCount returns how many negative_cache rows hold
// sourcePath.
func negativeCacheRowCount(tb testing.TB, cache *Cache, sourcePath string) int {
tb.Helper()
var count int
err := cache.db.QueryRowContext(tb.Context(),
`SELECT COUNT(*) FROM negative_cache WHERE source_path = ?`,
sourcePath,
).Scan(&count)
if err != nil {
tb.Fatalf("failed to count negative_cache rows: %v", err)
}
return count
}
// TestCleanExpiredDeletesInBatches stores one expired negative cache row
// more than one batch holds, and one row that has not expired. One
// statement must delete one batch and no more, so a request waits behind
// one batch at most; CleanExpired must then remove the expired row left
// over and keep the row that has not expired.
func TestCleanExpiredDeletesInBatches(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
ctx := t.Context()
insertExpiredNegativeCacheRows(t, cache, cleanExpiredBatchSize+1)
const notExpiredPath = "/later.jpg"
_, err = cache.db.ExecContext(ctx, `
INSERT INTO negative_cache
(source_host, source_path, status_code, expires_at)
VALUES (?, ?, 404, datetime('now', '+1 hour'))
`, testHostExample, notExpiredPath)
if err != nil {
t.Fatalf("failed to insert a row that has not expired: %v", err)
}
deleted, err := cache.cleanExpiredBatch(ctx)
if err != nil {
t.Fatalf("cleanExpiredBatch failed: %v", err)
}
t.Logf("one statement deleted %d of %d expired rows",
deleted, cleanExpiredBatchSize+1)
if deleted != cleanExpiredBatchSize {
t.Errorf("one statement deleted %d rows, want %d",
deleted, cleanExpiredBatchSize)
}
err = cache.CleanExpired(ctx)
if err != nil {
t.Fatalf("CleanExpired failed: %v", err)
}
if count := negativeCacheTableRowCount(t, cache); count != 1 {
t.Errorf("negative_cache holds %d rows after CleanExpired, want 1", count)
}
if negativeCacheRowCount(t, cache, notExpiredPath) != 1 {
t.Error("the row that has not expired was removed")
}
}
// TestCleanExpiredRemovesMoreThanOneBatch stores one expired negative
// cache row more than one batch holds and calls CleanExpired once: it
// must remove them all, so it fails if CleanExpired stops after its
// first batch.
func TestCleanExpiredRemovesMoreThanOneBatch(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
insertExpiredNegativeCacheRows(t, cache, cleanExpiredBatchSize+1)
before := negativeCacheTableRowCount(t, cache)
err = cache.CleanExpired(t.Context())
if err != nil {
t.Fatalf("CleanExpired failed: %v", err)
}
after := negativeCacheTableRowCount(t, cache)
t.Logf("CleanExpired removed %d of %d expired rows, %d per batch",
before-after, before, cleanExpiredBatchSize)
if after != 0 {
t.Errorf("negative_cache holds %d rows after CleanExpired, want 0", after)
}
}
// TestCleanExpiredStopsOnceContextEnds runs CleanExpired with a context
// that has already ended: it must return the context's error and delete
// no row.
func TestCleanExpiredStopsOnceContextEnds(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{StateDir: t.TempDir()})
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
insertExpiredNegativeCacheRows(t, cache, 1)
ctx, cancel := context.WithCancel(t.Context())
cancel()
err = cache.CleanExpired(ctx)
if !errors.Is(err, context.Canceled) {
t.Errorf("CleanExpired() error = %v, want context.Canceled", err)
}
if count := negativeCacheTableRowCount(t, cache); count != 1 {
t.Errorf("negative_cache holds %d rows, want the expired row kept", count)
}
}
// insertExpiredNegativeCacheRows inserts count negative_cache rows that
// expired an hour ago, each for a different source path.
func insertExpiredNegativeCacheRows(t *testing.T, cache *Cache, count int) {
t.Helper()
for i := range count {
_, err := cache.db.ExecContext(t.Context(), `
INSERT INTO negative_cache
(source_host, source_path, status_code, expires_at)
VALUES (?, ?, 404, datetime('now', '-1 hour'))
`, testHostExample, "/expired/"+strconv.Itoa(i)+".jpg")
if err != nil {
t.Fatalf("failed to insert an expired row: %v", err)
}
}
}
// negativeCacheTableRowCount returns how many rows negative_cache holds.
func negativeCacheTableRowCount(tb testing.TB, cache *Cache) int {
tb.Helper()
var count int
err := cache.db.QueryRowContext(tb.Context(),
`SELECT COUNT(*) FROM negative_cache`,
).Scan(&count)
if err != nil {
tb.Fatalf("failed to count negative_cache rows: %v", err)
}
return count
}
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) { func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1,359 +0,0 @@
package imgcache
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(t *testing.T, svc *Service, req *ImageRequest) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// truncateFile cuts the file at path down to size bytes.
func truncateFile(t *testing.T, path string, size int64) {
t.Helper()
t.Logf("cutting %s from %d to %d bytes",
filepath.Base(path), fileSize(t, path), size)
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("truncating %s: %v", path, err)
}
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(t *testing.T, svc *Service, req *ImageRequest) string {
t.Helper()
contentHash, _, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || contentHash == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", contentHash, err)
}
return svc.cache.srcContent.hashToPath(contentHash)
}
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
// when it starts, and returns what the pass logged.
func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
t.Helper()
var logBuf bytes.Buffer
cache.log = slog.New(slog.NewJSONHandler(&logBuf, nil))
err := cache.reconcileAccounting(t.Context())
if err != nil {
t.Fatalf("reconcileAccounting() error = %v", err)
}
t.Logf("reconciliation log: %s", logBuf.String())
return logBuf.String()
}
// assertWarningNames checks that logs hold a warning naming the file at
// path.
func assertWarningNames(t *testing.T, logs, path string) {
t.Helper()
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"WARN"`) &&
strings.Contains(line, `"path":"`+path+`"`) {
return
}
}
t.Errorf("reconciliation logged no warning naming %s", path)
}
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source
// file, as a crash or a restored snapshot could, and runs the reconciliation
// pass, which must remove it with a warning naming it. It then asks for two
// new sizes of the source: the first must fetch it again and cache it whole,
// so that the second is made from the cache without a third fetch.
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, 0)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
"was kept and fetched again on every new size", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
// file to half its size, as a crash or a restored snapshot could, and runs
// the reconciliation pass, which must remove it with a warning naming it.
// It then asks for two new sizes of the source: the first must fetch it
// again rather than fail to decode the partial file, and the second is
// made from the cache.
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, FitCover))
getImage(t, svc, photoVariant(fixtures, 30, FitCover))
fetches := readCacheStatsCounters(t, svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// assertConvertedAgain caches a variant of the photo, cuts its file to size
// bytes and runs the reconciliation pass, which must remove it with a
// warning naming it. It then checks that Get converts the image again
// instead of serving the file, and that the next Get serves the whole
// variant from the cache.
func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
got, status := getImage(t, svc, req)
t.Logf("after the cut: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
// variant file, as a crash or a restored snapshot could, and checks that it
// is converted again rather than served empty.
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 0)
}
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
// variant file to ten bytes, as a crash or a restored snapshot could, and
// checks that it is converted again rather than served cut short.
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 10)
}
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
// reconciliation pass removes an empty variant file that has no
// accounting row, with a warning naming it, instead of adding a row for
// it that would have it served empty.
func TestReconciliationRemovesEmptyVariantFileWithoutRow(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
_, err := cache.variants.Store(testVariantKeyOne, bytes.NewReader(nil),
testContentTypeWebP)
if err != nil {
t.Fatalf("failed to store an empty variant file: %v", err)
}
logs := reconcileAsAtStartup(t, cache)
assertWarningNames(t, logs, cache.variants.keyToPath(testVariantKeyOne))
if cache.variants.Exists(testVariantKeyOne) {
t.Error("the empty variant file is still on disk")
}
rows := countRows(t, cache,
`SELECT COUNT(*) FROM variant_content WHERE cache_key = ?`,
string(testVariantKeyOne))
if rows != 0 {
t.Errorf("the empty variant file got %d accounting rows, want 0", rows)
}
}
// TestVariantStorage_StoreReplacesMetaFile stores a variant over one
// already stored and checks that Store puts a new .meta file in place of
// the old one, as it does the image, rather than writing into the old
// file, which a crash could leave empty. The old .meta file is given a
// second name before the store, so that it can be told apart from the new
// one after it.
func TestVariantStorage_StoreReplacesMetaFile(t *testing.T) {
t.Parallel()
storage, key := storeTestVariant(t)
metaPath := storage.keyToPath(key) + variantMetaSuffix
oldMetaPath := metaPath + ".old"
err := os.Link(metaPath, oldMetaPath)
if err != nil {
t.Fatalf("giving the .meta file a second name: %v", err)
}
_, err = storage.Store(key, bytes.NewReader([]byte("new variant data")),
testContentTypeJPEG)
if err != nil {
t.Fatalf("Store() error = %v", err)
}
oldMeta, err := os.Stat(oldMetaPath)
if err != nil {
t.Fatalf("stat of the old .meta file: %v", err)
}
newMeta, err := os.Stat(metaPath)
if err != nil {
t.Fatalf("stat of the new .meta file: %v", err)
}
if os.SameFile(oldMeta, newMeta) {
t.Error("Store() wrote into the old .meta file instead of putting " +
"a new one in its place")
}
reader, _, contentType, err := storage.LoadWithMeta(key)
if err != nil {
t.Fatalf("LoadWithMeta() error = %v", err)
}
_ = reader.Close()
t.Logf("content type after the store: %s", contentType)
if contentType != testContentTypeJPEG {
t.Errorf("content type = %q, want %q", contentType, testContentTypeJPEG)
}
}
// TestContentStorage_StoreHashedReplacesFileOfAnotherSize checks that a
// store over a file of another size at the content's path, such as a
// partial file a crash left, replaces it with the content.
func TestContentStorage_StoreHashedReplacesFileOfAnotherSize(t *testing.T) {
t.Parallel()
storage, err := NewContentStorage(t.TempDir())
if err != nil {
t.Fatalf("NewContentStorage() error = %v", err)
}
content := []byte("the whole source image")
sum := sha256.Sum256(content)
hash := ContentHash(hex.EncodeToString(sum[:]))
path := storage.hashToPath(hash)
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm)
if err != nil {
t.Fatalf("creating the content's directory: %v", err)
}
err = os.WriteFile(path, content[:5], StorageFilePerm)
if err != nil {
t.Fatalf("writing a partial file: %v", err)
}
_, err = storage.StoreHashed(hash, content)
if err != nil {
t.Fatalf("StoreHashed() error = %v", err)
}
stored, err := storage.Load(hash)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
defer func() { _ = stored.Close() }()
got, err := io.ReadAll(stored)
if err != nil {
t.Fatalf("reading the stored content: %v", err)
}
if !bytes.Equal(got, content) {
t.Errorf("stored file = %q, want %q", got, content)
}
}
+1 -7
View File
@@ -53,7 +53,7 @@ type ServiceConfig struct {
// Allowlist is the list of hosts that don't require signatures // Allowlist is the list of hosts that don't require signatures
Allowlist []string Allowlist []string
// MaxConcurrentProcessing is the most images processed at once; zero // MaxConcurrentProcessing is the most images processed at once; zero
// uses the image processor's default, half the CPUs // uses the image processor's default, one per CPU
MaxConcurrentProcessing int MaxConcurrentProcessing int
// Logger for logging // Logger for logging
Logger *slog.Logger Logger *slog.Logger
@@ -555,13 +555,7 @@ func (s *Service) processAndStore(
FitMode: imageprocessor.FitMode(req.FitMode), FitMode: imageprocessor.FitMode(req.FitMode),
} }
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := s.processor.Process(ctx, source, processReq) processResult, err := s.processor.Process(ctx, source, processReq)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
if err != nil { if err != nil {
return nil, fmt.Errorf("image processing failed: %w", err) return nil, fmt.Errorf("image processing failed: %w", err)
} }
+60 -66
View File
@@ -80,8 +80,8 @@ func (s *ContentStorage) Store(r io.Reader) (ContentHash, int64, error) {
// from hash, without recomputing it. Callers that already know the // from hash, without recomputing it. Callers that already know the
// hash before writing (e.g. because they must hold a hash-keyed lock // hash before writing (e.g. because they must hold a hash-keyed lock
// across the whole store operation) use this instead of Store. Like // across the whole store operation) use this instead of Store. Like
// Store, it is idempotent: a file of the content's size already at that // Store, it is idempotent: content already on disk at that path is
// path is left untouched, and a file of another size is replaced. // left untouched.
func (s *ContentStorage) StoreHashed( func (s *ContentStorage) StoreHashed(
hash ContentHash, data []byte, hash ContentHash, data []byte,
) (int64, error) { ) (int64, error) {
@@ -152,35 +152,28 @@ func (s *ContentStorage) Exists(hash ContentHash) bool {
return err == nil return err == nil
} }
// writeIfAbsent writes data with writeFileSynced to the path derived // writeIfAbsent writes data to the path derived from hash, unless
// from hash, unless a file of the data's size is already there. // content already exists there, via a temp-file-plus-rename so
// concurrent readers never observe a partial file.
func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error { func (s *ContentStorage) writeIfAbsent(hash ContentHash, data []byte) error {
// Build path: <basedir>/<ab>/<cd>/<hash> // Build path: <basedir>/<ab>/<cd>/<hash>
path := s.hashToPath(hash) path := s.hashToPath(hash)
// A file of another size there is damaged, and is replaced // Check if already exists
info, statErr := os.Stat(path) _, statErr := os.Stat(path)
if statErr == nil && info.Size() == int64(len(data)) { if statErr == nil {
return nil return nil
} }
err := os.MkdirAll(filepath.Dir(path), StorageDirPerm) // Create directory structure
dir := filepath.Dir(path)
err := os.MkdirAll(dir, StorageDirPerm)
if err != nil { if err != nil {
return fmt.Errorf("failed to create directory: %w", err) return fmt.Errorf("failed to create directory: %w", err)
} }
return writeFileSynced(path, data) // Write to temp file first, then rename for atomicity
}
// writeFileSynced writes data to path through a temp file in the same
// directory: the temp file is synced to disk, renamed to path, and the
// directory synced after, so that neither a concurrent reader nor a
// crash finds a partial file under path. When the directory sync fails,
// the file is removed from path, so that a failed write leaves no file
// there.
func writeFileSynced(path string, data []byte) error {
dir := filepath.Dir(path)
tmpFile, err := os.CreateTemp(dir, ".tmp-*") tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil { if err != nil {
return fmt.Errorf("failed to create temp file: %w", err) return fmt.Errorf("failed to create temp file: %w", err)
@@ -188,20 +181,18 @@ func writeFileSynced(path string, data []byte) error {
tmpPath := tmpFile.Name() tmpPath := tmpFile.Name()
// Each failure path below unlinks the temp file explicitly. This
// replaces a deferred cleanup that read a named result, which the
// canonical config does not permit; the set of paths that remove
// tmpPath, and the order relative to Close, is unchanged. This
// mirrors how MetadataStorage.Store and VariantStorage.Store below
// already express the same cleanup.
_, err = tmpFile.Write(data) _, err = tmpFile.Write(data)
if err != nil { if err != nil {
_ = tmpFile.Close() _ = tmpFile.Close()
_ = os.Remove(tmpPath) _ = os.Remove(tmpPath)
return fmt.Errorf("failed to write temp file: %w", err) return fmt.Errorf("failed to write content: %w", err)
}
err = tmpFile.Sync()
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return fmt.Errorf("failed to sync temp file: %w", err)
} }
err = tmpFile.Close() err = tmpFile.Close()
@@ -211,6 +202,7 @@ func writeFileSynced(path string, data []byte) error {
return fmt.Errorf("failed to close temp file: %w", err) return fmt.Errorf("failed to close temp file: %w", err)
} }
// Atomic rename
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path)) err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil { if err != nil {
_ = os.Remove(tmpPath) _ = os.Remove(tmpPath)
@@ -218,31 +210,6 @@ func writeFileSynced(path string, data []byte) error {
return fmt.Errorf("failed to rename temp file: %w", err) return fmt.Errorf("failed to rename temp file: %w", err)
} }
err = syncDirectory(dir)
if err != nil {
_ = os.Remove(path)
return err
}
return nil
}
// syncDirectory syncs the directory dir to disk, so that a file just
// renamed into it is still there after a crash.
func syncDirectory(dir string) error {
d, err := os.Open(filepath.Clean(dir))
if err != nil {
return fmt.Errorf("failed to open directory: %w", err)
}
err = d.Sync()
_ = d.Close()
if err != nil {
return fmt.Errorf("failed to sync directory: %w", err)
}
return nil return nil
} }
@@ -460,9 +427,7 @@ func NewVariantStorage(baseDir string, log *slog.Logger) (*VariantStorage, error
return &VariantStorage{baseDir: baseDir, log: log}, nil return &VariantStorage{baseDir: baseDir, log: log}, nil
} }
// Store writes content and metadata to storage at the given key, each // Store writes content and metadata to storage at the given key.
// with writeFileSynced. The .meta file is written first, so that a crash
// cannot leave the content in place without it.
func (s *VariantStorage) Store( func (s *VariantStorage) Store(
key VariantKey, r io.Reader, contentType string, key VariantKey, r io.Reader, contentType string,
) (int64, error) { ) (int64, error) {
@@ -475,11 +440,46 @@ func (s *VariantStorage) Store(
path := s.keyToPath(key) path := s.keyToPath(key)
metaPath := path + ".meta" metaPath := path + ".meta"
err = os.MkdirAll(filepath.Dir(path), StorageDirPerm) // Create directory structure
dir := filepath.Dir(path)
err = os.MkdirAll(dir, StorageDirPerm)
if err != nil { if err != nil {
return 0, fmt.Errorf("failed to create directory: %w", err) return 0, fmt.Errorf("failed to create directory: %w", err)
} }
// Write content to temp file first, then rename for atomicity
tmpFile, err := os.CreateTemp(dir, ".tmp-*")
if err != nil {
return 0, fmt.Errorf("failed to create temp file: %w", err)
}
tmpPath := tmpFile.Name()
_, err = tmpFile.Write(data)
if err != nil {
_ = tmpFile.Close()
_ = os.Remove(tmpPath)
return 0, fmt.Errorf("failed to write content: %w", err)
}
err = tmpFile.Close()
if err != nil {
_ = os.Remove(tmpPath)
return 0, fmt.Errorf("failed to close temp file: %w", err)
}
// Atomic rename content
err = os.Rename(filepath.Clean(tmpPath), filepath.Clean(path))
if err != nil {
_ = os.Remove(tmpPath)
return 0, fmt.Errorf("failed to rename temp file: %w", err)
}
// Write metadata
meta := VariantMeta{ meta := VariantMeta{
ContentType: contentType, ContentType: contentType,
Size: size, Size: size,
@@ -491,19 +491,13 @@ func (s *VariantStorage) Store(
return 0, fmt.Errorf("failed to marshal metadata: %w", err) return 0, fmt.Errorf("failed to marshal metadata: %w", err)
} }
// Metadata write failure is non-fatal: the content is then served as // Metadata write failure is non-fatal; content is already stored.
// application/octet-stream. err = os.WriteFile(metaPath, metaData, StorageFilePerm)
err = writeFileSynced(metaPath, metaData)
if err != nil { if err != nil {
s.log.Warn("failed to write variant metadata sidecar", s.log.Warn("failed to write variant metadata sidecar",
"path", metaPath, "error", err) "path", metaPath, "error", err)
} }
err = writeFileSynced(path, data)
if err != nil {
return 0, err
}
return size, nil return size, nil
} }
+25 -89
View File
@@ -3,7 +3,6 @@ package imgcache
import ( import (
"errors" "errors"
"fmt" "fmt"
"net/netip"
"net/url" "net/url"
"regexp" "regexp"
"strconv" "strconv"
@@ -29,9 +28,7 @@ var (
const MaxDimension = imageprocessor.MaxOutputDimension const MaxDimension = imageprocessor.MaxOutputDimension
// ValidateDimension checks one requested width or height; name ("width" or // ValidateDimension checks one requested width or height; name ("width" or
// "height") appears in the error. 0 is valid: that side is then worked out from // "height") appears in the error. 0 means "original size" and is valid.
// the source's proportions and the other requested side, rounded down but at
// least 1, or is the source's own side when the other side is 0 too.
func ValidateDimension(name string, value int) error { func ValidateDimension(name string, value int) error {
if value < 0 { if value < 0 {
return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name) return fmt.Errorf("%w: %s is negative", ErrInvalidSize, name)
@@ -50,24 +47,6 @@ func ValidateDimension(name string, value int) error {
var sizeFormatRegex = regexp.MustCompile( var sizeFormatRegex = regexp.MustCompile(
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`) `^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
// percentEscapeRegex matches one percent-escape, such as "%2e"
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
// queryEscapeRegex matches a percent-escape, or one character that a browser
// percent-encodes in the query of an http or https URL: one outside the
// printable ASCII characters "!" to "~", such as a space, a control character
// or a character outside ASCII, or one of `"#'<>`. A "%" that does not begin a
// percent-escape matches too.
var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#'<>%]`)
// hostRegex matches the forms the host of an image URL may take: a host name
// or an IPv4 address (letters, digits, "-", "_" and "."), or an IPv6 address
// in brackets (hex digits, ":" and "."), either one optionally followed by ":"
// and a port, which may be empty. Its one group holds the text in brackets,
// which parseImageComponents also checks is an IPv6 address.
var hostRegex = regexp.MustCompile(
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
// ParsedURL contains the parsed components of an image proxy URL. // ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct { type ParsedURL struct {
// Host is the source origin host (e.g., "cdn.example.com") // Host is the source origin host (e.g., "cdn.example.com")
@@ -82,8 +61,9 @@ type ParsedURL struct {
Format ImageFormat Format ImageFormat
} }
// ParseImagePath parses an image URL's path after /v1/image/, decoded once: // ParseImagePath parses the path captured by chi's wildcard:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL. // <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
// This is the primary entry point when using chi routing.
// Examples: // Examples:
// - cdn.example.com/photos/cat.jpg/800x600.webp // - cdn.example.com/photos/cat.jpg/800x600.webp
// - cdn.example.com/photos/cat.jpg/0x0.jpeg // - cdn.example.com/photos/cat.jpg/0x0.jpeg
@@ -99,10 +79,9 @@ func ParseImagePath(path string) (*ParsedURL, error) {
return parseImageComponents(path) return parseImageComponents(path)
} }
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds // ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size> // or /v1/image/<host>/<path>/<size> for JPEG XL
// for JPEG XL. A source query follows the source path after a "?", which the // Use ParseImagePath instead when working with chi's wildcard capture.
// URL writes as %3F.
func ParseImageURL(urlPath string) (*ParsedURL, error) { func ParseImageURL(urlPath string) (*ParsedURL, error) {
// Remove the /v1/image/ prefix // Remove the /v1/image/ prefix
const prefix = "/v1/image/" const prefix = "/v1/image/"
@@ -121,6 +100,12 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
// parseImageComponents parses <host>/<path>/<size>.<format>, or // parseImageComponents parses <host>/<path>/<size>.<format>, or
// <host>/<path>/<size> for JPEG XL. // <host>/<path>/<size> for JPEG XL.
func parseImageComponents(remainder string) (*ParsedURL, error) { func parseImageComponents(remainder string) (*ParsedURL, error) {
// Check for path traversal before any other processing
err := checkPathTraversal(remainder)
if err != nil {
return nil, err
}
// Find the last path segment, which holds "size" or "size.format" // Find the last path segment, which holds "size" or "size.format"
lastSlash := strings.LastIndex(remainder, "/") lastSlash := strings.LastIndex(remainder, "/")
if lastSlash == -1 { if lastSlash == -1 {
@@ -159,20 +144,10 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
return nil, ErrMissingHost return nil, ErrMissingHost
} }
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in // A URL reads what comes before "@" as a user name and password, and the
// the host would make Go's URL parser, which the allowlist check and the // rest as the host
// fetcher use, read a different host from the one allowlisted and cached if strings.Contains(host, "@") {
hostMatch := hostRegex.FindStringSubmatch(host) return nil, fmt.Errorf("%w: %q holds @", ErrInvalidHost, host)
if hostMatch == nil {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
// Go's URL parser refuses brackets around anything but an IPv6 address
if inBrackets := hostMatch[1]; inBrackets != "" {
addr, err := netip.ParseAddr(inBrackets)
if err != nil || !addr.Is6() {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
} }
// Host names do not depend on letter case, so the allowlist, the upstream // Host names do not depend on letter case, so the allowlist, the upstream
@@ -180,30 +155,12 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
// host // host
host = strings.ToLower(host) host = strings.ToLower(host)
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a // Extract query string if present in path
// fragment and drops the rest, so each text after it would be another
// spelling, cached under its own keys, of one upstream request
if strings.Contains(path, "#") {
return nil, fmt.Errorf(
`%w: the source path or query holds a "#"`, ErrInvalidPath)
}
// Extract query string if present in path. It is the source URL's query
// as that URL writes it. A character a URL's query cannot hold as it is,
// such as a space written %20 in the image URL, is percent-encoded here,
// so the query is signed and cached as it is sent upstream
if qIndex := strings.Index(path, "?"); qIndex != -1 { if qIndex := strings.Index(path, "?"); qIndex != -1 {
query = escapeQuery(path[qIndex+1:]) query = path[qIndex+1:]
path = path[:qIndex] path = path[:qIndex]
} }
// The source query names no file on the upstream, so only the host and
// the source path are checked for path traversal
err = checkPathTraversal(host + path)
if err != nil {
return nil, err
}
// Ensure path starts with / // Ensure path starts with /
if !strings.HasPrefix(path, "/") { if !strings.HasPrefix(path, "/") {
path = "/" + path path = "/" + path
@@ -229,7 +186,12 @@ func checkPathTraversal(path string) error {
// Decode multiple times to catch double-encoding // Decode multiple times to catch double-encoding
decoded := path decoded := path
for range 3 { for range 3 {
newDecoded := decodePercentEscapes(decoded) newDecoded, err := url.PathUnescape(decoded)
if err != nil {
// Malformed encoding is suspicious
return ErrPathTraversal
}
if newDecoded == decoded { if newDecoded == decoded {
break break
} }
@@ -270,32 +232,6 @@ func checkPathTraversal(path string) error {
return nil return nil
} }
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
// keeps a "%" that does not start one as it is, where url.PathUnescape would
// refuse the whole string. A source path holding %25, a literal "%", has such
// a "%" after one more decoding.
func decodePercentEscapes(s string) string {
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
decoded, _ := url.PathUnescape(escape)
return decoded
})
}
// escapeQuery returns a source query as a URL's query holds it: each
// character that queryEscapeRegex names, such as a space, percent-encoded,
// and the rest, the query's percent-escapes included, kept as they are. The
// result is unchanged by a second escapeQuery.
func escapeQuery(query string) string {
return queryEscapeRegex.ReplaceAllStringFunc(query, func(match string) string {
if percentEscapeRegex.MatchString(match) {
return match
}
return url.PathEscape(match)
})
}
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600" // parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
func parseSizeFormat(s string) (Size, ImageFormat, error) { func parseSizeFormat(s string) (Size, ImageFormat, error) {
matches := sizeFormatRegex.FindStringSubmatch(s) matches := sizeFormatRegex.FindStringSubmatch(s)
@@ -407,71 +407,6 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
} }
} }
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
// a source query holding %25, the percent-escape of a literal "%", are
// accepted as they are, although decoding them once more leaves a "%" that
// does not start a percent-escape.
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
want *ParsedURL
}{
{
name: "path",
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: "/sale/50%25off.jpg",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
{
name: "query",
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImageURL(tt.input)
if err != nil {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
})
}
}
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
// double-encoded ".." is still refused when the path also holds a "%" that
// does not start a percent-escape, either as written or after one more
// decoding.
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
t.Parallel()
for _, input := range []string{
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
} {
_, err := ParseImageURL(input)
t.Logf("ParseImageURL(%q) error = %v", input, err)
if !errors.Is(err, ErrPathTraversal) {
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
input, err)
}
}
}
// errorIs checks if err matches target (handles wrapped errors). // errorIs checks if err matches target (handles wrapped errors).
func errorIs(err, target error) bool { func errorIs(err, target error) bool {
if errors.Is(err, target) { if errors.Is(err, target) {
+3 -87
View File
@@ -8,15 +8,12 @@ import (
"net/http" "net/http"
"net/netip" "net/netip"
"regexp" "regexp"
"strings"
"time" "time"
basicauth "github.com/99designs/basicauth-go" basicauth "github.com/99designs/basicauth-go"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors" "github.com/go-chi/cors"
"github.com/go-chi/httprate" "github.com/go-chi/httprate"
"github.com/prometheus/client_golang/prometheus"
metrics "github.com/slok/go-http-metrics/metrics/prometheus" metrics "github.com/slok/go-http-metrics/metrics/prometheus"
ghmm "github.com/slok/go-http-metrics/middleware" ghmm "github.com/slok/go-http-metrics/middleware"
"github.com/slok/go-http-metrics/middleware/std" "github.com/slok/go-http-metrics/middleware/std"
@@ -66,10 +63,6 @@ type Middleware struct {
log *slog.Logger log *slog.Logger
config *config.Config config *config.Config
clientIP *clientip.Resolver clientIP *clientip.Resolver
// metricsRegistry is the registry Metrics registers its metrics with.
// nil is Prometheus' default registry, the one /metrics serves.
metricsRegistry prometheus.Registerer
} }
// New creates a new Middleware instance. // New creates a new Middleware instance.
@@ -216,91 +209,14 @@ func (s *Middleware) CORS() func(http.Handler) http.Handler {
}) })
} }
// Metrics returns a Prometheus metrics middleware. Its handler label is the // Metrics returns a Prometheus metrics middleware.
// pattern of the chi route a request matches, such as /v1/image/*, or
// "unmatched" for a request that matches none, so the routes fix how many
// values the label has. It is never the path, with which any client could
// add series to the registry without limit. Its method label is the
// request's method when that is a standard one, such as GET, and "other"
// for any other: Go's HTTP server accepts any method name, so a label per
// method would let a client add series the same way.
func (s *Middleware) Metrics() func(http.Handler) http.Handler { func (s *Middleware) Metrics() func(http.Handler) http.Handler {
mdlw := ghmm.New(ghmm.Config{ mdlw := ghmm.New(ghmm.Config{
Recorder: metrics.NewRecorder(metrics.Config{ Recorder: metrics.NewRecorder(metrics.Config{}),
Registry: s.metricsRegistry,
}),
}) })
return func(next http.Handler) http.Handler { return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return std.Handler("", mdlw, next)
// The metrics library takes the method label from the request
// it is handed, so it is handed a shallow copy of r whose
// method is the label. next is served r itself, through the
// response writer the library wraps to see the status and size.
labelled := r.WithContext(r.Context())
labelled.Method = methodLabel(r.Method)
serveNext := http.HandlerFunc(
func(wrapped http.ResponseWriter, _ *http.Request) {
next.ServeHTTP(wrapped, r)
})
std.Handler(routePattern(r), mdlw, serveNext).ServeHTTP(w, labelled)
})
}
}
// routePattern returns the pattern of the chi route r matches, or
// "unmatched" when it matches none or is not served by a chi router. The
// metrics library takes the label before the request is routed, so the
// router is asked which route matches, with the path it routes on:
// r.URL.RawPath when set, else r.URL.Path, else /.
func routePattern(r *http.Request) string {
pattern := ""
rctx := chi.RouteContext(r.Context())
if rctx != nil {
path := r.URL.RawPath
if path == "" {
path = r.URL.Path
}
if path == "" {
path = "/"
}
pattern = rctx.Routes.Find(chi.NewRouteContext(), r.Method, path)
// For the path a subrouter is mounted at, such as /v1 or /v1/, Find
// answers that path without asking the subrouter, which chi hands
// the request as /.
for _, route := range rctx.Routes.Routes() {
if route.SubRoutes != nil &&
route.Pattern == strings.TrimSuffix(pattern, "/")+"/*" &&
!route.SubRoutes.Match(chi.NewRouteContext(), r.Method, "/") {
pattern = ""
}
}
}
if pattern == "" {
return "unmatched"
}
return pattern
}
// methodLabel returns method when it is one of the standard methods net/http
// names: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS, CONNECT and TRACE.
// For any other method it returns "other".
func methodLabel(method string) string {
switch method {
case http.MethodGet, http.MethodHead, http.MethodPost, http.MethodPut,
http.MethodPatch, http.MethodDelete, http.MethodOptions,
http.MethodConnect, http.MethodTrace:
return method
default:
return "other"
} }
} }
+1 -148
View File
@@ -3,16 +3,12 @@ package middleware
import ( import (
"bytes" "bytes"
"log/slog" "log/slog"
"maps"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
"slices"
"strings" "strings"
"testing" "testing"
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp" "github.com/prometheus/client_golang/prometheus/promhttp"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
@@ -202,7 +198,7 @@ func TestMetricsRecordsServedRequest(t *testing.T) {
// The line /metrics shows once one GET /test has been served. // The line /metrics shows once one GET /test has been served.
const want = `http_request_duration_seconds_count{` + const want = `http_request_duration_seconds_count{` +
`code="200",handler="unmatched",method="GET",service=""} 1` `code="200",handler="/test",method="GET",service=""} 1`
mw := &Middleware{log: slog.Default(), config: &config.Config{}} mw := &Middleware{log: slog.Default(), config: &config.Config{}}
@@ -224,149 +220,6 @@ func TestMetricsRecordsServedRequest(t *testing.T) {
} }
} }
// TestMetricsLabelsRequestsByRoute checks that the metrics middleware labels
// each request with the pattern of the route that answers it, never with its
// path: two paths one route answers share that route's pattern, and requests
// no route answers, a 404 or a 405, share "unmatched", among them requests for
// the path the subrouter is mounted at, /v1 and /v1/. A request with an empty
// path is answered by the / route, as chi routes it as /, and is labelled /.
// With a label per path, any client could add series to the registry without
// limit. The middleware registers with a registry of its own here, as another
// test in this package registers with Prometheus' default registry, which can
// be done only once.
func TestMetricsLabelsRequestsByRoute(t *testing.T) {
t.Parallel()
registry := prometheus.NewRegistry()
mw := &Middleware{
log: slog.Default(),
config: &config.Config{},
metricsRegistry: registry,
}
// The image route is in a subrouter, as in pixa's router.
router := chi.NewRouter()
router.Use(mw.Metrics())
router.Get("/", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
router.Route("/v1", func(r chi.Router) {
r.Get("/image/*", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
})
for _, req := range []struct{ method, path string }{
{http.MethodGet, "/v1/image/a.example.com/one.jpg/1x1.png"},
{http.MethodGet, "/v1/image/b.example.com/two.jpg/1x1.png"},
{http.MethodGet, "/no-such-route"},
{http.MethodGet, "/v1/no-such-route"},
{http.MethodGet, "/v1"},
{http.MethodGet, "/v1/"},
{http.MethodPost, "/v1/image/a.example.com/one.jpg/1x1.png"},
// The request line GET http://pixa.example HTTP/1.1 leaves both
// r.URL.Path and r.URL.RawPath empty.
{http.MethodGet, "http://pixa.example"},
} {
r := httptest.NewRequestWithContext(
t.Context(), req.method, req.path, nil)
t.Logf("%s %s: r.URL.Path %q, r.URL.RawPath %q",
req.method, req.path, r.URL.Path, r.URL.RawPath)
router.ServeHTTP(httptest.NewRecorder(), r)
}
got := requestCountsByHandler(t, registry)
t.Logf("requests counted by handler label: %v", got)
want := map[string]uint64{"/": 1, "/v1/image/*": 2, "unmatched": 5}
if !maps.Equal(got, want) {
t.Errorf("requests counted by handler label = %v, want %v", got, want)
}
}
// requestCountsByHandler returns how many requests the request duration
// histogram in registry counted under each value of its handler label.
func requestCountsByHandler(
t *testing.T, registry *prometheus.Registry,
) map[string]uint64 {
t.Helper()
families, err := registry.Gather()
if err != nil {
t.Fatalf("registry.Gather() error = %v", err)
}
counts := map[string]uint64{}
for _, family := range families {
if family.GetName() != "http_request_duration_seconds" {
continue
}
for _, metric := range family.GetMetric() {
for _, label := range metric.GetLabel() {
if label.GetName() == "handler" {
counts[label.GetValue()] += metric.GetHistogram().GetSampleCount()
}
}
}
}
return counts
}
// TestMetricsGivesMadeUpMethodsOneLabel checks that the metrics middleware
// gives requests with methods that are not standard ones a single method
// label, "other", and still serves each request with the method it was sent
// with. With a label per method, any client could add series to the
// registry without limit by making up a new method for each request.
func TestMetricsGivesMadeUpMethodsOneLabel(t *testing.T) {
t.Parallel()
// The line /metrics shows once both made-up methods have been served.
const want = `http_request_duration_seconds_count{` +
`code="200",handler="unmatched",method="other",service=""} 2`
registry := prometheus.NewRegistry()
mw := &Middleware{
log: slog.Default(),
config: &config.Config{},
metricsRegistry: registry,
}
var served []string
handler := mw.Metrics()(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
served = append(served, r.Method)
w.WriteHeader(http.StatusOK)
}))
methods := []string{"FROBNICATE", "WIBBLE"}
for _, method := range methods {
handler.ServeHTTP(httptest.NewRecorder(), httptest.NewRequestWithContext(
t.Context(), method, "/", nil))
}
t.Logf("methods the handler was served: %v", served)
if !slices.Equal(served, methods) {
t.Errorf("methods the handler was served = %v, want %v",
served, methods)
}
rec := httptest.NewRecorder()
promhttp.HandlerFor(registry, promhttp.HandlerOpts{}).ServeHTTP(rec,
httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/metrics", nil))
if !strings.Contains(rec.Body.String(), want) {
t.Errorf("/metrics does not count both made-up methods as one; "+
"want the line %q in:\n%s", want, rec.Body.String())
}
}
// TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST / // TestLoggingLeavesOutSubmittedSigningKey checks that a login, a POST /
// whose form carries the signing key, leaves no trace of the key in the // whose form carries the signing key, leaves no trace of the key in the
// request's log line. // request's log line.
@@ -14,12 +14,10 @@ import (
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/netip"
"os" "os"
"path/filepath" "path/filepath"
"sync/atomic" "sync/atomic"
"testing" "testing"
"time"
"go.uber.org/fx" "go.uber.org/fx"
"go.uber.org/fx/fxtest" "go.uber.org/fx/fxtest"
@@ -185,57 +183,6 @@ func startImageProxy(
return s, db.DB(), stateDir return s, db.DB(), stateDir
} }
// startImageProxyWithBlockedNetworks starts the components pixad's fx app
// builds, from a config with a fresh state directory and blockedNetworks as
// blocked_networks. Unlike startImageProxy it provides no upstream fetcher, so
// the handlers build the one pixad uses, which checks every address it
// connects to against the built-in blocked ranges and blocked_networks. It
// returns the server with its routes.
func startImageProxyWithBlockedNetworks(
t *testing.T, blockedNetworks []netip.Prefix,
) *Server {
t.Helper()
stateDir := t.TempDir()
cfg := &config.Config{
SigningKey: testSigningKey,
StateDir: stateDir,
DBURL: "file:" + filepath.Join(stateDir, "state.sqlite3"),
BlockedNetworks: blockedNetworks,
// A fetch that gets past the checks and finds nothing at the address
// fails within this time at the latest.
UpstreamFetchTimeout: 2 * time.Second,
// With no connection slots, the fetch would fail before connecting.
UpstreamConnections: config.DefaultUpstreamConnections,
DownstreamTimeout: config.DefaultDownstreamTimeout,
}
var (
h *handlers.Handlers
mw *middleware.Middleware
)
app := fxtest.New(t,
fx.Supply(cfg),
fx.Provide(
globals.New,
logger.New,
database.New,
healthcheck.New,
handlers.New,
middleware.New,
),
fx.Populate(&h, &mw),
)
app.RequireStart()
t.Cleanup(app.RequireStop)
s := &Server{config: cfg, mw: mw, h: h}
s.SetupRoutes()
return s
}
// getImage sends a GET for target to s and fails unless it answers 200. // getImage sends a GET for target to s and fails unless it answers 200.
func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder { func getImage(t *testing.T, s *Server, target string) *httptest.ResponseRecorder {
t.Helper() t.Helper()
-269
View File
@@ -1,269 +0,0 @@
package server
import (
"net/http"
"net/http/httptest"
"net/url"
"path/filepath"
"regexp"
"strings"
"testing"
"go.uber.org/fx"
"go.uber.org/fx/fxtest"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/globals"
"sneak.berlin/go/pixa/internal/handlers"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/middleware"
"sneak.berlin/go/pixa/internal/session"
)
// logoutFormPattern extracts the CSRF token rendered into the generator
// page's logout form.
var logoutFormPattern = regexp.MustCompile(
`(?s)action="/logout".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
// startPixa starts the components pixad's fx app builds, from cfg, and
// returns the server with its routes, as newTestServer does.
func startPixa(t *testing.T, cfg *config.Config) *Server {
t.Helper()
var (
h *handlers.Handlers
mw *middleware.Middleware
)
app := fxtest.New(t,
fx.Supply(cfg),
fx.Provide(
globals.New,
logger.New,
database.New,
healthcheck.New,
handlers.New,
middleware.New,
),
fx.Populate(&h, &mw),
)
app.RequireStart()
t.Cleanup(app.RequireStop)
s := &Server{config: cfg, mw: mw, h: h}
s.SetupRoutes()
return s
}
// logIn logs in to s with the signing key, as postLogin does, and returns
// the session cookie it sets.
func logIn(t *testing.T, s *Server) *http.Cookie {
t.Helper()
rec := postLogin(t, s, firstClient, "", testSigningKey)
for _, c := range rec.Result().Cookies() {
if c.Name == session.CookieName {
return c
}
}
t.Fatal("logging in set no session cookie")
return nil
}
// opensGeneratorPage reports whether GET / with cookie shows s's generator
// page, which only a valid login session opens.
func opensGeneratorPage(t *testing.T, s *Server, cookie *http.Cookie) bool {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(cookie)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
return rec.Code == http.StatusOK &&
strings.Contains(rec.Body.String(), `action="/generate"`)
}
// postLogout loads the generator page with GET / and sessionCookie and
// submits its logout form with POST /logout, as a browser does.
func postLogout(
t *testing.T, s *Server, sessionCookie *http.Cookie,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(sessionCookie)
page := httptest.NewRecorder()
s.ServeHTTP(page, req)
match := logoutFormPattern.FindStringSubmatch(page.Body.String())
if match == nil {
t.Fatal("the generator page has no logout form with a CSRF token")
}
form := url.Values{"gorilla.csrf.Token": {match[1]}}
req = httptest.NewRequestWithContext(t.Context(), http.MethodPost,
"/logout", strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.AddCookie(sessionCookie)
for _, c := range page.Result().Cookies() {
req.AddCookie(c)
}
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
return rec
}
// TestLogoutWithoutCSRFTokenEndsNoSession verifies that a request to /logout
// carrying the session cookie but not the logout form's CSRF token ends no
// session: GET /logout, as an image on another page of the same site makes a
// browser send, answers 405, and POST /logout without the token answers 403.
func TestLogoutWithoutCSRFTokenEndsNoSession(t *testing.T) {
t.Parallel()
tests := []struct {
method string
want int
}{
{http.MethodGet, http.StatusMethodNotAllowed},
{http.MethodPost, http.StatusForbidden},
}
for _, tc := range tests {
t.Run(tc.method, func(t *testing.T) {
t.Parallel()
s := newTestServer(t)
cookie := logIn(t, s)
req := httptest.NewRequestWithContext(
t.Context(), tc.method, "/logout", nil)
req.AddCookie(cookie)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
t.Logf("%s /logout without the CSRF token: status %d",
tc.method, rec.Code)
if rec.Code != tc.want {
t.Errorf("%s /logout without the CSRF token status = %d, "+
"want %d", tc.method, rec.Code, tc.want)
}
if !opensGeneratorPage(t, s, cookie) {
t.Errorf("%s /logout without the CSRF token ended the "+
"login session", tc.method)
}
})
}
}
// TestLoginAfterLogoutOpensGeneratorPage verifies that after a logout from
// the generator page a copy of the session cookie taken before it is
// refused, and that a login after it opens the generator page.
func TestLoginAfterLogoutOpensGeneratorPage(t *testing.T) {
t.Parallel()
s := newTestServer(t)
copied := logIn(t, s)
if !opensGeneratorPage(t, s, copied) {
t.Fatal("logging in did not open the generator page")
}
logout := postLogout(t, s, copied)
if logout.Code != http.StatusSeeOther {
t.Fatalf("POST /logout status = %d, want %d",
logout.Code, http.StatusSeeOther)
}
if opensGeneratorPage(t, s, copied) {
t.Error("a cookie copied before the logout still opens the " +
"generator page")
}
opens := opensGeneratorPage(t, s, logIn(t, s))
t.Logf("a login after the logout opens the generator page: %v", opens)
if !opens {
t.Error("a login after the logout does not open the generator page")
}
}
// TestLogoutOutlastsRestart verifies that a copy of the session cookie taken
// before a logout is still refused by a pixa started again on the same state
// directory, as pixad is after a restart.
func TestLogoutOutlastsRestart(t *testing.T) {
t.Parallel()
s := newTestServer(t)
copied := logIn(t, s)
if !opensGeneratorPage(t, s, copied) {
t.Fatal("logging in did not open the generator page")
}
logout := postLogout(t, s, copied)
if logout.Code != http.StatusSeeOther {
t.Fatalf("POST /logout status = %d, want %d",
logout.Code, http.StatusSeeOther)
}
restarted := startPixa(t, s.config)
opens := opensGeneratorPage(t, restarted, copied)
t.Logf("after the restart, the copied cookie opens the generator page: %v",
opens)
if opens {
t.Error("after a restart, a cookie copied before the logout " +
"still opens the generator page")
}
}
// TestLostStateDirectoryEndsEveryLoginSession verifies that a pixa started on
// an empty state directory, as pixad is after a restart that lost its state
// directory, refuses a session cookie from before, and that a login on it
// opens the generator page.
func TestLostStateDirectoryEndsEveryLoginSession(t *testing.T) {
t.Parallel()
s := newTestServer(t)
before := logIn(t, s)
if !opensGeneratorPage(t, s, before) {
t.Fatal("logging in did not open the generator page")
}
emptyStateDir := *s.config
emptyStateDir.StateDir = t.TempDir()
emptyStateDir.DBURL = "file:" +
filepath.Join(emptyStateDir.StateDir, "state.sqlite3")
restarted := startPixa(t, &emptyStateDir)
opens := opensGeneratorPage(t, restarted, before)
t.Logf("on an empty state directory, the cookie from before opens the "+
"generator page: %v", opens)
if opens {
t.Error("on an empty state directory, a cookie from before still " +
"opens the generator page")
}
if !opensGeneratorPage(t, restarted, logIn(t, restarted)) {
t.Error("a login on an empty state directory does not open the " +
"generator page")
}
}
+2 -1
View File
@@ -69,9 +69,10 @@ func (s *Server) SetupRoutes() {
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)). r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
Post("/", s.h.HandleRoot()) Post("/", s.h.HandleRoot())
r.Post("/generate", s.h.HandleGenerateURL()) r.Post("/generate", s.h.HandleGenerateURL())
r.Post("/logout", s.h.HandleLogout())
}) })
s.router.Get("/logout", s.h.HandleLogout())
// Image routes, the only ones that send CORS headers, as pages on other // Image routes, the only ones that send CORS headers, as pages on other
// sites read them. They are a subrouter rather than a group: a group's // sites read them. They are a subrouter rather than a group: a group's
// middleware runs only for a request that matches one of its routes, // middleware runs only for a request that matches one of its routes,
-665
View File
@@ -1,665 +0,0 @@
package server
import (
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"regexp"
"slices"
"strconv"
"strings"
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
)
// percentSignPath is a source path holding a "%", which a URL writes %25.
const percentSignPath = "/100%25.png"
// generateFormPattern extracts the CSRF token rendered into the generator
// page's form.
var generateFormPattern = regexp.MustCompile(
`(?s)action="/generate".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
// generatedURLPattern extracts the path of the encrypted URL the generator
// page shows.
var generatedURLPattern = regexp.MustCompile(`value="[^"]*(/v1/e/[^"]+)"`)
// recordingUpstream is a test upstream that answers every request with a PNG
// and records the request target, the path and query as sent, of each request
// it receives.
type recordingUpstream struct {
*httptest.Server
mu sync.Mutex
received []string
}
// startRecordingUpstream starts a recordingUpstream, which is closed when the
// test ends.
func startRecordingUpstream(t *testing.T) *recordingUpstream {
t.Helper()
source := encodeTestPNG(t, 64, 48)
upstream := &recordingUpstream{}
upstream.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
upstream.mu.Lock()
upstream.received = append(upstream.received, r.RequestURI)
upstream.mu.Unlock()
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(source)
}))
t.Cleanup(upstream.Close)
return upstream
}
// requestTargets returns the request targets the upstream has received, in
// the order it received them.
func (u *recordingUpstream) requestTargets() []string {
u.mu.Lock()
defer u.mu.Unlock()
return slices.Clone(u.received)
}
// pageOrigin is the origin of the login and generator pages below, opened
// over HTTPS; httptest gives every request the host example.com.
const pageOrigin = "https://example.com"
// openRoot sends GET / to s over HTTPS, with cookie when it is not nil, and
// returns the page: the login page, or the generator page for a login session.
func openRoot(
t *testing.T, s *Server, cookie *http.Cookie,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
pageOrigin+"/", nil)
if cookie != nil {
req.AddCookie(cookie)
}
page := httptest.NewRecorder()
s.ServeHTTP(page, req)
return page
}
// submitForm posts form to path on s as a browser submits it from page over
// HTTPS: with the CSRF token tokenPattern finds in page, the cookies page set,
// cookie when it is not nil, and a Referer naming the page, which pixa needs
// while debug is off, as it is in startImageProxy's config.
func submitForm(
t *testing.T, s *Server, page *httptest.ResponseRecorder,
tokenPattern *regexp.Regexp, path string, form url.Values,
cookie *http.Cookie,
) *httptest.ResponseRecorder {
t.Helper()
match := tokenPattern.FindStringSubmatch(page.Body.String())
if match == nil {
t.Fatalf("the page has no form for %s with a CSRF token", path)
}
form.Set("gorilla.csrf.Token", match[1])
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost,
pageOrigin+path, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Referer", pageOrigin+"/")
for _, c := range page.Result().Cookies() {
req.AddCookie(c)
}
if cookie != nil {
req.AddCookie(cookie)
}
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
return rec
}
// postGenerate logs in to s with the signing key, submits the generator page's
// form with sourceURL, and returns the response.
func postGenerate(
t *testing.T, s *Server, sourceURL string,
) *httptest.ResponseRecorder {
t.Helper()
login := submitForm(t, s, openRoot(t, s, nil), csrfFieldPattern, "/",
url.Values{"key": {testSigningKey}}, nil)
var sessionCookie *http.Cookie
for _, c := range login.Result().Cookies() {
if c.Name == session.CookieName {
sessionCookie = c
}
}
if sessionCookie == nil {
t.Fatalf("logging in answered %d and set no session cookie", login.Code)
}
rec := submitForm(t, s, openRoot(t, s, sessionCookie), generateFormPattern,
"/generate", url.Values{"url": {sourceURL}}, sessionCookie)
t.Logf("POST /generate with url %s: %d", sourceURL, rec.Code)
return rec
}
// TestImageURLFetchesItsSourcePathAndQuery requests image URLs whose source
// query holds a space, written percent-encoded, or a "'", and whose source
// path holds a "%", written percent-encoded, and checks the one request the
// upstream receives: the source path percent-encoded again, and the source
// query with the space written %20 and the "'" written %27, as a browser
// writes them in a query.
func TestImageURLFetchesItsSourcePathAndQuery(t *testing.T) {
t.Parallel()
tests := []struct {
name string
// pathAndQuery is the source path and query as the image URL writes
// them
pathAndQuery string
// wantRequest is the path and query the upstream must receive
wantRequest string
}{
{"space in the query", "/photo.png%3Fname=my%20cat",
"/photo.png?name=my%20cat"},
{"apostrophe in the query", "/photo.png%3Fname=it's",
"/photo.png?name=it%27s"},
{"percent sign in the path", percentSignPath, percentSignPath},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
getImage(t, s, "/v1/image/"+upstreamHost+tt.pathAndQuery+"/orig.png")
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.wantRequest}) {
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
}
})
}
}
// TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart requests twice each
// of the source paths /my cat.png and /my%20cat.png, which the image URL
// writes my%20cat.png and my%2520cat.png. Each is fetched once, as a request
// of its own, and its second request is served from the cache.
func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
imageURLs := []string{
"/v1/image/" + upstreamHost + "/my%20cat.png/orig.png",
"/v1/image/" + upstreamHost + "/my%2520cat.png/orig.png",
}
for _, imageURL := range imageURLs {
getImage(t, s, imageURL)
}
for _, imageURL := range imageURLs {
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) {
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
imageURL, cacheStatus)
}
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
want := []string{"/my%20cat.png", "/my%2520cat.png"}
if !slices.Equal(got, want) {
t.Errorf("upstream received %q, want %q", got, want)
}
}
// TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest requests the source query
// name=my%20cat written as the README says, name=my%2520cat, and with the
// space written %20, name=my%20cat. Both are one source query: the upstream
// receives one request, and the second image URL is served from the cache.
func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
getImage(t, s, "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%2520cat/orig.png")
second := getImage(t, s,
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
want := []string{"/photo.png?name=my%20cat"}
if !slices.Equal(got, want) {
t.Errorf("upstream received %q, want %q", got, want)
}
}
// TestGeneratedURLFetchesTheSourceURLAsEntered makes an encrypted URL on the
// generator page for source URLs holding %20, %25 and %23 in the path or the
// query, requests it, and checks that the upstream receives the path and
// query exactly as entered.
func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
t.Parallel()
// The names hold no "%": each test's state directory is named after it
// and goes into the database URL, which decodes percent-escapes
tests := []struct {
name string
pathAndQuery string
}{
{"space in the path", "/my%20cat.png"},
{"percent sign in the path", percentSignPath},
{"number sign in the path", "/a%23b.png"},
{"all three in the query", "/photo.png?name=my%20cat%25%23"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
rec := postGenerate(t, s, "https://"+upstreamHost+tt.pathAndQuery)
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
if rec.Code != http.StatusOK || match == nil {
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
rec.Code, rec.Body.String())
}
getImage(t, s, match[1])
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.pathAndQuery}) {
t.Errorf("upstream received %q, want [%q]", got, tt.pathAndQuery)
}
})
}
}
// TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL enters two
// spellings of one source URL on the generator page, requests the encrypted
// URL made for each, and checks that the upstream receives one request and
// that the second encrypted URL is served from the cache. One pair writes the
// host in upper and in lower case, which /v1/image/ lower-cases. The host is
// an IPv6 address, whose hex digits have letter case, because the fetcher
// would look a host name up in DNS. It has a port, as the fetcher refuses an
// IPv6 address written without one. The other pair is a source URL with no
// path and with the path "/", both of which fetch "/".
func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
t.Parallel()
tests := []struct {
name string
// sourceURLs are the two spellings, after "https://"
sourceURLs []string
// wantRequest is the path the upstream must receive
wantRequest string
}{
{"host in upper and in lower case",
[]string{"[2001:DB8::A]:8443/a.png", "[2001:db8::a]:8443/a.png"},
"/a.png"},
{"no path and the path written as a slash",
[]string{upstreamHost, upstreamHost + "/"}, "/"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
var imageURLs []string
for _, sourceURL := range tt.sourceURLs {
rec := postGenerate(t, s, "https://"+sourceURL)
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
if rec.Code != http.StatusOK || match == nil {
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
rec.Code, rec.Body.String())
}
imageURLs = append(imageURLs, match[1])
}
getImage(t, s, imageURLs[0])
second := getImage(t, s, imageURLs[1])
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.wantRequest}) {
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
}
})
}
}
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
// generator page must make no encrypted URL for, and checks that it answers
// 400 and shows why. pixa keeps the source path decoded and fetches it
// percent-encoded again, so a path holding %2F would be fetched with a "/"
// there, whatever else the path holds. pixa sends no user name or password,
// and no "?" with no query after it. For these the page shows the URL pixa
// would fetch. A source path holding a ".." segment is refused, as
// /v1/image/ refuses it.
func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
t.Parallel()
// The test config sets allow_http, so pixa would fetch over plain HTTP
const fetchedFrom = "http://" + upstreamHost
tests := []struct {
name string
// sourceURL is the source URL after "https://"
sourceURL string
// wantShown is what the page must show: the URL pixa would fetch, or
// why it makes no encrypted URL
wantShown string
}{
{"encoded slash", upstreamHost + "/a%2Fb.png",
fetchedFrom + "/a/b.png"},
{"encoded slash and a space", upstreamHost + "/a%2Fb c.png",
fetchedFrom + "/a/b%20c.png"},
{"encoded slash and a bar", upstreamHost + "/a%2Fb|c.png",
fetchedFrom + "/a/b%7Cc.png"},
{"encoded slash and a letter outside ASCII", upstreamHost + "/café%2Fx.png",
fetchedFrom + "/caf%C3%A9/x.png"},
{"user name and password", "user:pass@" + upstreamHost + "/a.png",
fetchedFrom + "/a.png"},
{"question mark with no query", upstreamHost + "/a.png?",
fetchedFrom + "/a.png"},
{"two-dot segment", upstreamHost + "/a/../b.png",
"cannot hold a .. segment"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startImageProxy(t, startRecordingUpstream(t).Server)
rec := postGenerate(t, s, "https://"+tt.sourceURL)
body := rec.Body.String()
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
if strings.Contains(body, "/v1/e/") {
t.Error("the page shows an encrypted URL")
}
if !strings.Contains(body, tt.wantShown) {
t.Errorf("the page does not show %s; body %s", tt.wantShown, body)
}
})
}
}
// ipv6SourcePath is the source path of the image URLs signedImageURL and
// encryptedImageURL make.
const ipv6SourcePath = "/ipv6.png"
// TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort requests ipv6SourcePath
// on an IPv6 source host, written with and without a port, through a
// /v1/image/ URL with a signature and through an encrypted URL. The public
// address 2001:db8::a, a documentation address (RFC 3849) the fetcher accepts
// as public, answers 200, and the upstream receives the request for
// ipv6SourcePath. The private address fd00::1 answers 403, and the upstream
// receives no request: the test's fetcher connects every address to the test
// upstream, so only the fetcher's check of the source URL can refuse it.
func TestImageRoutesFetchIPv6SourceHostWithOrWithoutPort(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourceHost string
wantStatus int
// wantRequests are the request targets the upstream must receive
wantRequests []string
}{
{"public address without a port", "[2001:db8::a]",
http.StatusOK, []string{ipv6SourcePath}},
{"public address with a port", "[2001:db8::a]:8443",
http.StatusOK, []string{ipv6SourcePath}},
{"private address without a port", "[fd00::1]",
http.StatusForbidden, nil},
{"private address with a port", "[fd00::1]:8443",
http.StatusForbidden, nil},
}
for _, tt := range tests {
imageURLs := map[string]string{
"image URL with a signature": signedImageURL(t, tt.sourceHost),
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
}
for route, imageURL := range imageURLs {
t.Run(tt.name+" "+route, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s: %d", imageURL, rec.Code)
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d; body %s",
rec.Code, tt.wantStatus, rec.Body.String())
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, tt.wantRequests) {
t.Errorf("upstream received %q, want %q", got, tt.wantRequests)
}
})
}
}
}
// signedImageURL returns the /v1/image/ URL of ipv6SourcePath on sourceHost at
// its original size as PNG, with a signature made with the test signing key,
// which a host not on the allowlist needs. The URL has no q or fit, so it is
// signed with the quality 85 and the fit cover.
func signedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
path, sig, exp := signature.New(testSigningKey).GenerateSignedURL(
&signature.Request{
SourceHost: sourceHost,
SourcePath: ipv6SourcePath,
Format: string(imgcache.FormatPNG),
Quality: 85,
FitMode: string(imgcache.FitCover),
}, time.Hour)
return path + "?sig=" + sig + "&exp=" + strconv.FormatInt(exp, 10)
}
// encryptedImageURL returns the encrypted URL of ipv6SourcePath on sourceHost
// at its original size as PNG, made with the test signing key.
func encryptedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
generator, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
}
token, err := generator.Generate(&encurl.Payload{
SourceHost: sourceHost, SourcePath: ipv6SourcePath,
Format: imgcache.FormatPNG,
})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
return "/v1/e/" + token + "/img.png"
}
// TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address requests
// ipv6SourcePath on IPv6 source hosts that hold a loopback or private IPv4
// address in their last four bytes, in the IPv4-compatible form (::/96) or the
// NAT64 form (64:ff9b::/96), through a /v1/image/ URL with a signature and
// through an encrypted URL. Each answers 403, and the upstream receives no
// request: the test's fetcher connects every address to the test upstream, so
// only the fetcher's check of the source URL can refuse it.
func TestImageRoutesRefuseIPv6SourceHostHoldingInternalIPv4Address(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
sourceHosts := []string{
"[::127.0.0.1]", // IPv4-compatible form of a loopback address
"[::10.0.0.1]", // IPv4-compatible form of a private address
"[64:ff9b::7f00:1]", // NAT64 form of 127.0.0.1
}
for _, sourceHost := range sourceHosts {
imageURLs := []string{
signedImageURL(t, sourceHost),
encryptedImageURL(t, sourceHost),
}
for _, imageURL := range imageURLs {
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s (source host %s): %d", imageURL, sourceHost, rec.Code)
if rec.Code != http.StatusForbidden {
t.Errorf("source host %s: status = %d, want %d; body %s",
sourceHost, rec.Code, http.StatusForbidden, rec.Body.String())
}
}
}
if got := upstream.requestTargets(); len(got) != 0 {
t.Errorf("upstream received %q, want no request", got)
}
}
// TestImageRoutesCheckIPv4AddressInIPv6SourceHost requests ipv6SourcePath on
// source hosts that hold an IPv4 address, through a /v1/image/ URL with a
// signature and through an encrypted URL, with the fetcher pixad uses.
// [64:ff9b:1::a00:1], in the local-use NAT64 range (64:ff9b:1::/48, RFC 8215),
// answers 403. [::203.0.113.5] holds the public address 203.0.113.5 in the
// IPv4-compatible form (::/96) and answers 403 when blocked_networks lists
// 203.0.113.0/24 or ::/96. [::ffff:203.0.113.5] holds it in the IPv4-mapped
// form (::ffff:0:0/96) and answers 403 when blocked_networks lists
// 203.0.113.0/24. Go keeps the plain address 203.0.113.5 in the mapped form
// too, and an IPv6 range must not refuse it: with ::/0 listed it is not
// refused. Neither is [::203.0.113.5] with blocked_networks empty. Where a
// host is not refused, the fetcher tries to connect to it, finds nothing
// there, and the request fails with another status.
func TestImageRoutesCheckIPv4AddressInIPv6SourceHost(t *testing.T) {
t.Parallel()
const (
ipv4CompatibleHost = "[::203.0.113.5]"
ipv4MappedHost = "[::ffff:203.0.113.5]"
)
tests := []struct {
name string
sourceHost string
blockedNetworks []netip.Prefix
wantForbidden bool
}{
{"local-use NAT64 form of 10.0.0.1", "[64:ff9b:1::a00:1]", nil, true},
{"IPv4-compatible form of an address in blocked_networks",
ipv4CompatibleHost,
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
{"IPv4-compatible form with ::/96 in blocked_networks",
ipv4CompatibleHost,
[]netip.Prefix{netip.MustParsePrefix("::/96")}, true},
{"IPv4-mapped form of an address in blocked_networks",
ipv4MappedHost,
[]netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, true},
{"plain IPv4 address with ::/0 in blocked_networks",
"203.0.113.5",
[]netip.Prefix{netip.MustParsePrefix("::/0")}, false},
{"IPv4-compatible form of an address with blocked_networks empty",
ipv4CompatibleHost, nil, false},
}
for _, tt := range tests {
imageURLs := map[string]string{
"image URL with a signature": signedImageURL(t, tt.sourceHost),
"encrypted URL": encryptedImageURL(t, tt.sourceHost),
}
for route, imageURL := range imageURLs {
t.Run(tt.name+" "+route, func(t *testing.T) {
t.Parallel()
s := startImageProxyWithBlockedNetworks(t, tt.blockedNetworks)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, httptest.NewRequestWithContext(
t.Context(), http.MethodGet, imageURL, nil))
t.Logf("GET %s: %d", imageURL, rec.Code)
forbidden := rec.Code == http.StatusForbidden
if tt.wantForbidden && !forbidden {
t.Errorf("status = %d, want %d; body %s",
rec.Code, http.StatusForbidden, rec.Body.String())
}
if !tt.wantForbidden && forbidden {
t.Errorf("status = %d, want any other status; body %s",
rec.Code, rec.Body.String())
}
})
}
}
}
+1 -115
View File
@@ -1,16 +1,9 @@
// Package session provides encrypted session cookie management, and keeps in // Package session provides encrypted session cookie management.
// the database a random value that each logout replaces, so that a logout
// ends every session made before it.
package session package session
import ( import (
"context"
"crypto/rand"
"database/sql"
"errors" "errors"
"fmt"
"net/http" "net/http"
"sync"
"time" "time"
"github.com/gorilla/securecookie" "github.com/gorilla/securecookie"
@@ -39,36 +32,14 @@ type Data struct {
Authenticated bool `json:"auth"` Authenticated bool `json:"auth"`
CreatedAt time.Time `json:"created"` CreatedAt time.Time `json:"created"`
ExpiresAt time.Time `json:"expires"` ExpiresAt time.Time `json:"expires"`
// LogoutValue is the Manager's random value, which each logout
// replaces, when the session was made.
LogoutValue string `json:"logoutValue"`
} }
// Manager handles session creation and validation using encrypted cookies. // Manager handles session creation and validation using encrypted cookies.
//
// Every login is made with the one signing key, so a logout ends every
// session made before it, in every browser. The Manager keeps a random
// value, which each logout replaces: a session carries the value current at
// its login, and a session carrying another value is refused. The check for
// a logout compares no time, so a login after a logout works whatever the
// host's clock does; the session's expiry is still checked against the
// clock. The Manager keeps the value in memory, so checking a session reads
// no database; a Manager from NewManagerWithDatabase also saves it in the
// database, so that it outlasts a restart.
type Manager struct { type Manager struct {
sc *securecookie.SecureCookie sc *securecookie.SecureCookie
db *sql.DB
// mu guards logoutValue, and is held while a logout is saved, so that
// the database and logoutValue agree.
mu sync.Mutex
logoutValue string
} }
// NewManager creates a session manager with keys derived from the signing key. // NewManager creates a session manager with keys derived from the signing key.
// It makes a new random value and keeps it in memory only, so a restart ends
// every session; NewManagerWithDatabase keeps the value in the database.
// //
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict // Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
// attributes; this cannot be configured. Browsers treat http://localhost as a // attributes; this cannot be configured. Browsers treat http://localhost as a
@@ -93,43 +64,9 @@ func NewManager(signingKey string) (*Manager, error) {
return &Manager{ return &Manager{
sc: sc, sc: sc,
logoutValue: rand.Text(),
}, nil }, nil
} }
// NewManagerWithDatabase creates a session manager like NewManager that keeps
// its random value in db, which must have pixa's schema: it saves the value
// NewManager made when db has none, and otherwise starts from the one saved
// there.
func NewManagerWithDatabase(
ctx context.Context, signingKey string, db *sql.DB,
) (*Manager, error) {
m, err := NewManager(signingKey)
if err != nil {
return nil, err
}
_, err = db.ExecContext(ctx,
"INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)",
m.logoutValue)
if err != nil {
return nil, fmt.Errorf(
"failed to save the random value in the logout table: %w", err)
}
err = db.QueryRowContext(ctx,
"SELECT logout_value FROM logout WHERE id = 1",
).Scan(&m.logoutValue)
if err != nil {
return nil, fmt.Errorf(
"failed to read the random value in the logout table: %w", err)
}
m.db = db
return m, nil
}
// CreateSession creates a new authenticated session and sets the cookie. // CreateSession creates a new authenticated session and sets the cookie.
func (m *Manager) CreateSession(w http.ResponseWriter) error { func (m *Manager) CreateSession(w http.ResponseWriter) error {
now := time.Now() now := time.Now()
@@ -137,7 +74,6 @@ func (m *Manager) CreateSession(w http.ResponseWriter) error {
Authenticated: true, Authenticated: true,
CreatedAt: now, CreatedAt: now,
ExpiresAt: now.Add(SessionTTL), ExpiresAt: now.Add(SessionTTL),
LogoutValue: m.currentLogoutValue(),
} }
encoded, err := m.sc.Encode(CookieName, data) encoded, err := m.sc.Encode(CookieName, data)
@@ -186,30 +122,9 @@ func (m *Manager) ValidateSession(r *http.Request) (*Data, error) {
return nil, ErrInvalidSession return nil, ErrInvalidSession
} }
// A logout since the session was made has ended it.
if data.LogoutValue != m.currentLogoutValue() {
return nil, ErrInvalidSession
}
return &data, nil return &data, nil
} }
// Logout removes the session cookie and, when r has a valid session, ends
// every session made until now. A request without one ends no session, so
// someone who has not logged in cannot end the sessions of those who have.
func (m *Manager) Logout(w http.ResponseWriter, r *http.Request) error {
if m.IsAuthenticated(r) {
err := m.saveLogout(r.Context())
if err != nil {
return err
}
}
m.ClearSession(w)
return nil
}
// ClearSession removes the session cookie. // ClearSession removes the session cookie.
func (m *Manager) ClearSession(w http.ResponseWriter) { func (m *Manager) ClearSession(w http.ResponseWriter) {
http.SetCookie(w, &http.Cookie{ http.SetCookie(w, &http.Cookie{
@@ -230,32 +145,3 @@ func (m *Manager) IsAuthenticated(r *http.Request) bool {
return err == nil && data != nil && data.Authenticated return err == nil && data != nil && data.Authenticated
} }
// saveLogout replaces the random value with a new one, saving it in the
// database first when the Manager has one.
func (m *Manager) saveLogout(ctx context.Context) error {
value := rand.Text()
m.mu.Lock()
defer m.mu.Unlock()
if m.db != nil {
_, err := m.db.ExecContext(ctx,
"UPDATE logout SET logout_value = ? WHERE id = 1", value)
if err != nil {
return fmt.Errorf("failed to save the logout: %w", err)
}
}
m.logoutValue = value
return nil
}
// currentLogoutValue returns the Manager's random value.
func (m *Manager) currentLogoutValue() string {
m.mu.Lock()
defer m.mu.Unlock()
return m.logoutValue
}
-148
View File
@@ -1,148 +0,0 @@
package session_test
import (
"database/sql"
"errors"
"net/http"
"net/http/httptest"
"testing"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/session"
)
const logoutTestSigningKey = "test-signing-key-12345"
// newTestDatabase opens an empty in-memory database with pixa's schema. It
// has one connection, as pixad's has: each connection to ":memory:" would
// open a database of its own.
func newTestDatabase(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("sql.Open() error = %v", err)
}
db.SetMaxOpenConns(1)
t.Cleanup(func() { _ = db.Close() })
err = database.ApplyMigrations(t.Context(), db, nil)
if err != nil {
t.Fatalf("ApplyMigrations() error = %v", err)
}
return db
}
// createSessionCookie logs in with mgr and returns the session cookie it
// sets.
func createSessionCookie(t *testing.T, mgr *session.Manager) *http.Cookie {
t.Helper()
rec := httptest.NewRecorder()
err := mgr.CreateSession(rec)
if err != nil {
t.Fatalf("CreateSession() error = %v", err)
}
for _, c := range rec.Result().Cookies() {
if c.Name == session.CookieName {
return c
}
}
t.Fatalf("CreateSession() did not set a %q cookie", session.CookieName)
return nil
}
// requestWithCookie returns a GET request carrying cookie, or no cookie when
// cookie is nil.
func requestWithCookie(t *testing.T, cookie *http.Cookie) *http.Request {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
if cookie != nil {
req.AddCookie(cookie)
}
return req
}
// TestLogout_EndsEverySessionMadeBeforeIt verifies that after a logout a
// copy of the session cookie taken before it is refused and a login after it
// works, and that both still hold for a new Manager on the same database, as
// after a restart.
func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
t.Parallel()
db := newTestDatabase(t)
mgr, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db)
if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err)
}
copied := createSessionCookie(t, mgr)
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, copied))
if err != nil {
t.Fatalf("Logout() error = %v", err)
}
_, err = mgr.ValidateSession(requestWithCookie(t, copied))
t.Logf("cookie copied before the logout: ValidateSession() error = %v", err)
if !errors.Is(err, session.ErrInvalidSession) {
t.Errorf("cookie copied before the logout: error = %v, want %v",
err, session.ErrInvalidSession)
}
loginAfter := createSessionCookie(t, mgr)
_, err = mgr.ValidateSession(requestWithCookie(t, loginAfter))
if err != nil {
t.Errorf("login after the logout: ValidateSession() error = %v", err)
}
restarted, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db)
if err != nil {
t.Fatalf("NewManagerWithDatabase() after restart error = %v", err)
}
if restarted.IsAuthenticated(requestWithCookie(t, copied)) {
t.Error("after a restart, a cookie copied before the logout is accepted")
}
if !restarted.IsAuthenticated(requestWithCookie(t, loginAfter)) {
t.Error("after a restart, a login made after the logout is refused")
}
}
// TestLogout_WithoutSession_EndsNoSession verifies that a logout from a
// request without a session cookie ends no session, so someone who has not
// logged in cannot end the sessions of those who have.
func TestLogout_WithoutSession_EndsNoSession(t *testing.T) {
t.Parallel()
mgr, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, newTestDatabase(t))
if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err)
}
cookie := createSessionCookie(t, mgr)
err = mgr.Logout(httptest.NewRecorder(), requestWithCookie(t, nil))
if err != nil {
t.Fatalf("Logout() error = %v", err)
}
if !mgr.IsAuthenticated(requestWithCookie(t, cookie)) {
t.Error("a logout without a session cookie ended a session")
}
}
+22 -11
View File
@@ -126,20 +126,31 @@ func (s *Signer) GenerateSignedURL(
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height) sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
} }
// Build the path. A source query follows the source path after a "?". // Build the path.
// The image route reads the path decoded once, so the whole path is // When a source query is present, it is embedded as a path segment
// written in Go's own percent-encoding of a URL path, which writes "?" // (e.g. /host/path?query/size.fmt) so that the URL parser can extract
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they // it from the last-slash split. The "?" inside a path segment is
// are. // percent-encoded by clients but chi delivers it decoded, which is
sourcePathAndQuery := req.SourcePath // exactly what the URL parser expects.
var path string
if req.SourceQuery != "" { if req.SourceQuery != "" {
sourcePathAndQuery += "?" + req.SourceQuery path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
req.SourceHost,
req.SourcePath,
url.PathEscape(req.SourceQuery),
sizeStr,
req.Format,
)
} else {
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost,
req.SourcePath,
sizeStr,
req.Format,
)
} }
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s", return path, sig, exp
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
} }
// buildSignatureData creates the string to be signed. // buildSignatureData creates the string to be signed.
-38
View File
@@ -475,44 +475,6 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
} }
} }
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
// source URL with a query, the image at
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
// signature with the README's key, and the path GenerateSignedURL writes for
// it, with the "?" written as %3F and the "%" as %25.
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
t.Parallel()
signer := signature.New("example-signing-key-for-documentation")
req := signature.Request{
SourceHost: testHost,
SourcePath: "/img",
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Unix(1704067200, 0),
}
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
gotSig := signer.Sign(&req)
if gotSig != wantSig {
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
}
const wantPath = "/v1/image/cdn.example.com" +
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
if gotPath != wantPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
}
}
func TestParseParams(t *testing.T) { func TestParseParams(t *testing.T) {
t.Parallel() t.Parallel()
+2 -10
View File
@@ -122,20 +122,12 @@ header {
margin-bottom: 2rem; margin-bottom: 2rem;
} }
/* The logout button, drawn as a link. */ header a {
header button {
width: auto;
margin: 0;
padding: 0;
background: none;
font-size: 0.875rem; font-size: 0.875rem;
color: #4b5563; color: #4b5563;
text-decoration: underline;
} }
header button:hover { header a:hover {
background: none;
color: #1f2937; color: #1f2937;
} }
+2 -5
View File
@@ -10,12 +10,9 @@
<div class="page"> <div class="page">
<header> <header>
<h1>Pixa URL Generator</h1> <h1>Pixa URL Generator</h1>
<form method="POST" action="/logout"> <a href="/logout">
{{ .CSRFField }}
<button type="submit">
Logout Logout
</button> </a>
</form>
</header> </header>
{{if .GeneratedURL}} {{if .GeneratedURL}}
+12 -33
View File
@@ -3,13 +3,11 @@
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git, # or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). The installed node is used only when its major version # make, or go). Node is used directly if installed; otherwise it is
# is the pinned one; otherwise (node missing, or another major) the # installed at a pinned version via nvm (installing nvm itself first,
# pinned version is installed via nvm (installing nvm itself first, # from a hash-verified release archive, never curl | sh). The linter is
# from a hash-verified release archive, never curl | sh), and yarn and # never installed on the host: golangci-lint runs only in the lint phase
# the packages are installed under it. The linter is never installed on # of the Dockerfile (see script/lint).
# the host: golangci-lint runs only in the lint phase of the Dockerfile
# (see script/lint).
# #
# script/bootstrap git, make, Go, and Node, Yarn and the # script/bootstrap git, make, Go, and Node, Yarn and the
# prettier in yarn.lock for script/fmt and # prettier in yarn.lock for script/fmt and
@@ -112,50 +110,31 @@ ensure_nvm() {
rm -rf "$tmp" rm -rf "$tmp"
} }
# True when the node on PATH has the pinned major version. Only the
# major is compared, as REPO_POLICIES.md asks. Another major is not
# used: the pinned yarn 1 prints a deprecation warning under node 24.
# script/fmt and script/fmt-check carry this function unchanged, to run
# the yarn installed here.
node_is_pinned_major() {
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
ensure_node() { ensure_node() {
if node_is_pinned_major; then if ! missing node; then return 0; fi
echo "bootstrap: using node $(node --version)"
return 0
fi
ensure_nvm ensure_nvm
nvm_sh "nvm install $NODE_VERSION" nvm_sh "nvm install $NODE_VERSION"
nvm_sh "nvm use $NODE_VERSION >/dev/null && \
echo \"bootstrap: using node \$(node --version)\""
} }
ensure_yarn() { ensure_yarn() {
if ! node_is_pinned_major; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
return 0
fi
if ! missing yarn; then return 0; fi if ! missing yarn; then return 0; fi
if ! missing corepack; then if ! missing corepack; then
corepack enable corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else else
npm install -g "yarn@$YARN_VERSION" npm install -g "yarn@$YARN_VERSION"
fi fi
} }
install_js_deps() { install_js_deps() {
if node_is_pinned_major; then if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
yarn install --frozen-lockfile
else
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \ nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile" yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi fi
} }
+10 -13
View File
@@ -2,11 +2,9 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner # script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs # checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do. # the formatter on the host, which a pristine checkout cannot do.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase # --no-cache for the same reason as script/docker: the gate phases the
# is a check that did not run; unlike --no-cache, it keeps Go's build cache. # final stage depends on are RUN steps, and a cached one is a check that
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST # did not run.
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -16,17 +14,16 @@ main() {
cd "$ROOT" cd "$ROOT"
"$SCRIPT_DIR/bootstrap" "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check" "$SCRIPT_DIR/check"
# The version and the tag each get their own line: a failing # Own line: a failing command substitution inside an argument does
# command substitution inside an argument does not trip `set -e`, # not trip `set -e`, so the inline form degrades silently to an
# so the inline form degrades silently to an empty constant. The # empty constant. VERSION is computed here because .dockerignore
# VERSION build argument takes precedence over the version a build # excludes .git, so `git describe` in a build stage yields an empty
# stage derives from the .git in the context. # version without failing.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")" docker build --no-cache \
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+9 -13
View File
@@ -1,11 +1,8 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase # --no-cache because the gate phases the final stage depends on are RUN
# is a check that did not run; unlike --no-cache, it keeps Go's build cache. # steps, and a cached one is a check that did not run.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -13,17 +10,16 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
# The version and the tag each get their own line: a failing # Own line: a failing command substitution inside an argument does
# command substitution inside an argument does not trip `set -e`, # not trip `set -e`, so the inline form degrades silently to an
# so the inline form degrades silently to an empty constant. The # empty constant. VERSION is computed here because .dockerignore
# VERSION build argument takes precedence over the version a build # excludes .git, so `git describe` in a build stage yields an empty
# stage derives from the .git in the context. # version without failing.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")" docker build --no-cache \
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+6 -16
View File
@@ -7,25 +7,15 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap. # Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
# True when the node on PATH has the pinned major version: the same # script/bootstrap installs node and yarn under nvm and leaves neither
# function as in script/bootstrap, which uses it to decide where it # on the PATH of the shell that called it, so resolve the pinned
# installs yarn. # toolchain here the way bootstrap's own install step does. nvm is a
node_is_pinned_major() { # bash script, hence the subshell.
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
# Run the yarn script/bootstrap installed: the yarn on PATH when the
# node on PATH has the pinned major version, and otherwise yarn under
# the pinned node in nvm, which bootstrap leaves off the PATH of the
# shell that called it. nvm is a bash script, hence the subshell.
run_yarn() { run_yarn() {
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then if command -v yarn >/dev/null 2>&1; then
exec yarn "$@" exec yarn "$@"
fi fi
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2 echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1 exit 1
fi fi
+6 -16
View File
@@ -8,25 +8,15 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap. # Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0" NODE_VERSION="22.17.0"
# True when the node on PATH has the pinned major version: the same # script/bootstrap installs node and yarn under nvm and leaves neither
# function as in script/bootstrap, which uses it to decide where it # on the PATH of the shell that called it, so resolve the pinned
# installs yarn. # toolchain here the way bootstrap's own install step does. nvm is a
node_is_pinned_major() { # bash script, hence the subshell.
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
# Run the yarn script/bootstrap installed: the yarn on PATH when the
# node on PATH has the pinned major version, and otherwise yarn under
# the pinned node in nvm, which bootstrap leaves off the PATH of the
# shell that called it. nvm is a bash script, hence the subshell.
run_yarn() { run_yarn() {
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then if command -v yarn >/dev/null 2>&1; then
exec yarn "$@" exec yarn "$@"
fi fi
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2 echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1 exit 1
fi fi
+5 -8
View File
@@ -5,12 +5,9 @@
# make its answer untrustworthy. # make its answer untrustworthy.
# #
# The phase is not the last stage in the file, so it is built only when # The phase is not the last stage in the file, so it is built only when
# --target names it. A new CACHEBUST value reruns every RUN step, since # --target names it. --no-cache because a cached lint layer is a lint
# a cached lint layer is a lint that did not run; unlike --no-cache, it # that did not run. The tag makes each build replace the previous image
# keeps Go's build cache. This relies on every stage of the Dockerfile # instead of leaving a dangling one behind.
# declaring ARG CACHEBUST first; a stage without that line is served
# from cache on an unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -18,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \ docker build --no-cache \
--target lint \ --target lint \
--output type=cacheonly . -t "$("$SCRIPT_DIR/projectname")-lint" .
} }
main "$@" main "$@"
+4 -8
View File
@@ -2,12 +2,8 @@
# script/test: run the test suite. Testing is a phase of the Dockerfile # script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint: # and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when # --target because a phase that is not the last stage is built only when
# named, and a new CACHEBUST value because a cached test layer is a test # named, --no-cache because a cached test layer is a test that did not
# that did not run; unlike --no-cache, it keeps Go's build cache. This # run, and a tag so each build replaces the previous image.
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
# a stage without that line is served from cache on an unchanged tree,
# with no warning. --output type=cacheonly writes no image, since
# nothing uses one.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -15,9 +11,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \ docker build --no-cache \
--target test \ --target test \
--output type=cacheonly . -t "$("$SCRIPT_DIR/projectname")-test" .
} }
main "$@" main "$@"