Author SHA1 Message Date
sneak 47a2c2360a Keep WebP, AVIF and JPEG XL encodes at the limits under 40 seconds (closes #272)
check / check (push) Waiting to run
WebP is saved at effort 2 and JPEG XL at effort 4, and an AVIF output of
more than 16,777,216 pixels is refused with 400 before anything is
decoded, as AVIF is already at the lowest effort govips can set. With
one libvips thread, at any quality, 8192x8192 random pixels with an
alpha channel take up to about 23 seconds as WebP and 27 as JPEG XL, and
4096x4096, 8192x2048 or 2048x8192 up to about 34 as AVIF. Resizing with
fit=cover already decodes the input, so the output's dimensions are now
worked out before resizing, in outputDimensions. Reading the input moves
into readInput to keep Process within the linter's limits.

Model: opus-5-5
2026-10-15 22:47:20 +00:00
sneak 84a86274cd Test lower WebP and JPEG XL effort and the AVIF pixel limit
The WebP test now expects effort 2 instead of libvips' default of 4, a
new test expects JPEG XL at effort 4 instead of 7, and new tests expect an
AVIF output of more than 4096x4096 pixels to be refused before anything
is decoded and answered 400 on both image routes. They fail until the
settings and the refusal are in.

Model: opus-5-5
2026-10-15 22:41:24 +00:00
clawbot 22bfc432b6 Document how a source query goes into an image URL, and accept it (closes #255)
check / check (push) Canceled after 0s
The README gave no way to put a source URL's query into a signed image
URL, and GenerateSignedURL built one that failed: the route parsed
chi's wildcard, which keeps the path as the client wrote it, so one
%2F, %3B or %2C left the %3F undecoded and the signature failed. The
route now parses the decoded path, GenerateSignedURL writes Go's own
encoding of it, and the README says the ? before the query, any ? in it
and every % must be percent-encoded, with a signed example. The path
traversal check no longer reads the query and accepts a literal %, while
a double-encoded .. stays refused. A percent-escape in a source path is
now decoded once.

Model: opus-5-5
2026-10-16 00:17:33 +02:00
clawbot f743930f71 Default max_concurrent_processing to half the CPUs pixa sees (closes #242)
check / check (push) Canceled after 0s
The default max_concurrent_processing was one image per CPU pixa sees,
so in a container with no CPU limit on a 48-CPU host pixa processed 48
images at once. It is now half the CPUs, rounded down and at least 1,
defined once in internal/config; PIXA_MAX_CONCURRENT_PROCESSING stays
the override, and a value that is not a whole number of at least 1
stops startup. "Running under upaas" gives the worst-case memory, 4.2
GiB per image (the largest peak measured over every source format and
kind at 8192x8192) plus the upstream buffers, and the temporary file
libvips keeps under /tmp. The two existing tests expect the new
default, as the owner allowed.

Model: opus-5-5
2026-10-15 23:07:22 +02:00
clawbot 4e30dc7467 Re-vendor the canonical files from sneak/prompts at e3f79c4 (closes #221)
check / check (push) Canceled after 0s
The dotfiles, REPO_POLICIES.md and the standard scripts are fetched
whole from sneak/prompts e3f79c4, with pixa's own entries kept after
them. Check builds pass a new CACHEBUST build argument instead of
--no-cache, and every Dockerfile stage declares it first, so every RUN
step still runs while Go's build cache is kept. Lint runs golangci-lint
v2.14.0, and the test phase runs on the Debian Go image. Of the two new
lint findings, one is fixed and one (gosec G703 on the fixed config file
locations) is suppressed. The agent guidance moves unchanged from
CLAUDE.md to AGENTS.md.

Deviation: the workflow keeps its docker-smoke step while issue 203
waits on the owner.

Model: opus-5-5
2026-10-15 22:40:38 +02:00
35 changed files with 1348 additions and 390 deletions
+18 -9
View File
@@ -13,12 +13,21 @@
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# Unlike the standard file, which leaves out all of .git, pixa sends
# .git without its config. Without a VERSION build argument the stage
# that compiles runs `git describe --tags --always` on .git, which does
# not need .git/config; that file can hold a credential, such as a
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
.git/config
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
@@ -42,7 +51,9 @@
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# Dependencies: restored inside the image, never copied in.
**/node_modules
@@ -60,10 +71,8 @@
**/.vscode
**/*.sublime-*
# pixa's own entries. Nothing in the build reads .gitignore. On the
# host, `make build` writes bin/pixad, and the example config keeps its
# state directory in data/.
.gitignore
# pixa's own entries. On the host, `make build` writes bin/pixad, and the
# example config keeps its state directory in data/.
/bin
/data
+3
View File
@@ -13,3 +13,6 @@ indent_style = tab
[*.go]
indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
+9 -3
View File
@@ -1,15 +1,21 @@
name: check
on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
check:
runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# The default clone is shallow and has no tags, so the
# version the build takes from `git describe` would be a
# bare commit; this fetches the whole history with its tags.
# script/cibuild needs no token, so none is left in .git/config.
with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0
- run: script/cibuild
- run: script/docker-smoke
+37 -6
View File
@@ -17,15 +17,46 @@ Thumbs.db
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Environment / secrets
.env
.env.*
*.pem
*.key
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
# Go
*.log
*.out
*.test
# Dependencies
vendor/
node_modules/
# Build output
/bin/
+3
View File
@@ -17,6 +17,7 @@ linters:
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- exhaustruct_v5 # Requires all struct fields (successor to exhaustruct)
- godot # Requires comments to end with periods
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
@@ -24,6 +25,8 @@ linters:
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings:
lll:
line-length: 88
-5
View File
@@ -1,7 +1,2 @@
node_modules/
yarn.lock
# A byte-for-byte copy of the one in sneak/prompts.
REPO_POLICIES.md
vendor/
View File
+48 -30
View File
@@ -1,7 +1,12 @@
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# Lint phase. script/lint builds it alone. The linter is run directly:
# `make lint` and script/lint are themselves a docker build.
# golangci/golangci-lint:v2.12.2, 2026-10-04
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
# golangci/golangci-lint:v2.14.0, 2026-10-15
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
ARG CACHEBUST
# The linter compiles every package, and govips needs the libvips
# headers for that. REPO_POLICIES.md has the lint phase install them
@@ -14,17 +19,20 @@ WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
RUN --mount=type=cache,target=/root/.cache/go-build \
golangci-lint run --config .golangci.yml ./...
# Test phase. script/test builds it alone.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS test
# Test phase. script/test builds it alone. -race needs cgo and so a C
# compiler, which the Debian Go image ships and the alpine one does not.
# golang:1.25.4-trixie, 2026-10-15
FROM golang@sha256:a02d35efc036053fdf0da8c15919276bf777a80cbfda6a35c5e9f087e652adfc AS test
ARG CACHEBUST
WORKDIR /src
# script/bootstrap --cgo installs the build dependencies (a C compiler,
# the libvips and libheif headers, and libvips' JPEG XL support, which
# the tests need) and downloads the Go modules.
# script/bootstrap --cgo installs the build dependencies (the libvips
# and libheif headers, and libvips' JPEG XL support, which the tests
# need) and downloads the Go modules.
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
@@ -32,11 +40,14 @@ RUN script/bootstrap --cgo
COPY . .
# Without -v first; on a failure, again with -v for the details, and
# the step fails even if the second run passes. -parallel 4: by default
# a package runs as many of its tests at once as the host has CPUs, and
# on a busy host they then wait so long to be scheduled that a test
# that times a request can see it return a second late.
RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
# the step fails even if the second run passes. -count=1 because the
# mounted cache also holds test results, which go test would otherwise
# replay. -parallel 4: by default a package runs as many of its tests at
# once as the host has CPUs, and on a busy host they then wait so long
# to be scheduled that a test that times a request can see it return a
# second late.
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -parallel 4 -v ./...; exit 1; }
@@ -45,38 +56,44 @@ RUN go test -count=1 -timeout 90s -race -parallel 4 -cover ./... || \
# test passed.
# golang:1.25.4-alpine3.22, 2026-02-25; the runtime stage uses Alpine 3.22 too
FROM golang:1.25.4-alpine3.22@sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb AS builder
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
WORKDIR /src
# Build dependencies and Go modules, as in the test phase
# Build dependencies, git among them, and Go modules, as in the test
# phase
COPY script/ ./script/
COPY go.mod go.sum ./
RUN script/bootstrap --cgo
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
# Copy source code
COPY . .
# VERSION is declared here, not earlier: a new value reruns only the
# build, not script/bootstrap. Given none, the version is
# `git describe --tags --always` of the .git in the build context (git
# comes from script/bootstrap): the tag on a tagged commit, tag-N-gHASH
# after one, the short commit when no tag is reachable. A context that
# carries .git and still yields no version fails the build; one without
# .git, as from a source tarball, stamps an empty version. CGO stays
# enabled for govips; -trimpath keeps build paths out of the binary, and
# -s -w leave out the symbol table and debug information.
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context: the
# tag on a tagged commit, tag-N-gHASH after one, the short commit when
# no tag is reachable. With .git present, a version that is still empty,
# dev or unknown fails the build: git is missing or could not read the
# checkout. A context without .git, as from a source tarball, stamps an
# empty version. CGO stays enabled for govips; -trimpath keeps build
# paths out of the binary, and -s -w leave out the symbol table and
# debug information.
ARG VERSION
RUN version="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
[ "$version" = unknown ]; }; then \
echo "the build context carries .git but yields no version" >&2; \
exit 1; \
RUN --mount=type=cache,target=/root/.cache/go-build \
VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=1 GOTOOLCHAIN=auto go build -trimpath \
-ldflags "-s -w -X main.Version=${version}" \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /pixad ./cmd/pixad
# Runtime stage, and the last one: a plain `docker build .` builds this
@@ -85,6 +102,7 @@ RUN version="${VERSION:-$(git describe --tags --always)}"; \
# the libvips and musl it was built with.
# alpine:3.22, 2026-10-08
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
ARG CACHEBUST
# Install runtime dependencies only. vips-jxl is libvips' JPEG XL
# support, without which pixad does not start.
+92 -21
View File
@@ -131,6 +131,34 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait
for processing. 4.2 GiB is the most resident memory was measured to grow while
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
largest it accepts, saving it at full size to each output format in turn, one
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use.
- **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
written, so a listing of `/tmp` does not show it, but it keeps its space, on
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
well as written to that file, and the 4.2 GiB above includes that memory. For
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
does not include these files; where `/tmp` is a tmpfs, they are memory on top
of it.
- **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
after a deploy and marks the deploy failed unless it is `healthy`. The probe
@@ -288,12 +316,32 @@ once, is refused with 400.
both optional (values under Signature Specification). Both are part of what is
cached, so each value of either is a separate cached image.
A source URL's query follows its path after `%3F`, the encoded `?`:
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
percent-escapes in the image URL's path once, then reads the host up to the
first `/`, the size and format after the last `/`, and between them the source
path up to the first `?` and the source query after it. So the `?` before the
query, any `?` in the query, and every `%` in the source path or query must be
percent-encoded, as `%3F` and `%25`. Any other character may be written as
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
example,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 WebP is
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
signed as shown under Signature Specification.
A size with one side `0` has that side scaled to keep the source's proportions,
and is refused with 400 when that side would come out above 8192, the most
either side may be; so is a size with `fit=outside`, which scales the image to
cover the size and so past it on one side, when that side would come out
above 8192.
An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
have, is refused with 400 before anything is decoded, as its encode could take
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
not `image/jxl` is refused such a size.
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
`Content-Type` must name its format, and the image's first bytes must match it.
An upstream answering with any other `Content-Type`, SVG included, is refused
@@ -343,14 +391,18 @@ transcode that stops this way is answered 503 with the error
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
encode keeps its slot until it finishes, which can be after
`downstream_timeout`. The limit of 8192 on each side of the source and of the
output bounds how long that takes. With one libvips thread, as pixad runs it,
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or
JPEG XL encode can therefore hold its slot past the default `downstream_timeout`
of 60 seconds.
output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
encoders run below libvips' default effort, and AVIF at the lowest effort govips
can set, so that no encode takes more than 40 seconds of the default
`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
8192x8192 random pixels with an alpha channel take up to about 23 seconds as
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
up to about 11 seconds.
The login form (`POST /`) is limited to 5 attempts per minute per client
address, counting an IPv6 client by its /64; an attempt over the limit is
@@ -402,7 +454,8 @@ nor change what it asks for.
sent with an empty format, or none, also makes a JPEG XL URL. Width and
height both empty or `0` keep the original size; if only one of them is empty
or `0`, that side is scaled to keep the image's proportions. A URL whose
scaled side would come out above 8192 answers 400.
scaled side would come out above 8192 answers 400, and so does an AVIF URL of
more than 16,777,216 pixels (see Routes).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on, and the URL starts
with `http` instead while `debug` is on. The name after the token is ignored
@@ -507,6 +560,17 @@ For the same image at quality 40 with fit `contain`, the input ends in
and the URL is
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
For a source URL with a query,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
800x600 WebP with the same key and expiration and the default quality and fit:
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
`%25` (see Routes):
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
**Allowlist patterns:**
- **Exact match**: `cdn.example.com` — matches only that host
@@ -566,7 +630,7 @@ tell. With no file, pixa uses the environment and the defaults.
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs |
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
@@ -652,10 +716,12 @@ Key settings in more detail:
does, and `downstream_timeout` has not ended first, the request is answered
503 with the error `server busy, try again later`
- `max_concurrent_processing` — the most images decoded and encoded at once;
default the number of CPUs pixa can use (`GOMAXPROCS`), which follows a
container's CPU limit. A request that finds all of them in use waits up to 10
seconds for one to free up; if none does, and `downstream_timeout` has not
ended first, it is answered 503 the same way
default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's
CPU limit), rounded down, and at least 1. A value that is not a whole number
of at least 1 aborts startup. A request that finds all of them in use waits up
to 10 seconds for one to free up; if none does, and `downstream_timeout` has
not ended first, it is answered 503 the same way. Each one can take a lot of
memory: see "Running under upaas"
- `image_requests_per_minute` — how many requests one client may send to the
image routes per minute (see the limit on them under Routes); default `600`.
`0` turns the limit off, and a negative or non-integer value aborts startup.
@@ -706,9 +772,9 @@ provide:
install-precommit)
- `script/projectname` — output the project name ("pixa")
- `script/test` — run the test suite: build the `test` phase of the
`Dockerfile`, tagged `pixa-test`
`Dockerfile`, writing no image
- `script/lint` — run golangci-lint: build the `lint` phase of the `Dockerfile`,
tagged `pixa-lint`; the linter never runs on the host
writing no image; the linter never runs on the host
- `script/fmt` — format the Go code with gofmt and the markdown with prettier
(writes)
- `script/fmt-check` — check the same formatting (read-only), on the host
@@ -726,15 +792,20 @@ provide:
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`
Every `docker build` in these scripts passes `--no-cache`, so the lint and test
phases run on every build instead of coming from the build cache.
Every `docker build` in these scripts passes a new value of the `CACHEBUST`
build argument, which every stage of the `Dockerfile` declares first, so every
`RUN` step, the lint and test phases included, runs on every build instead of
coming from the build cache. Go's build cache is kept from build to build: the
steps that compile mount it, so a package whose inputs are unchanged is not
compiled again, and both `go test` runs pass `-count=1` so that no stored test
result is replayed. The `test` phase runs on the Debian Go image, with Debian's
libvips and libheif; the image pixa ships is built and runs on Alpine.
`script/check`, `script/cibuild`, `script/docker`, `script/lint`, `script/test`,
`script/setup` and `script/install-precommit` are the standard copies from
`sneak/prompts`, kept identical to them. `script/fmt` and `script/fmt-check` are
the standard copies with pixa's `gofmt` step kept before prettier. prettier
formats the markdown only: not the HTML templates, as it cannot parse a Go
template action inside a tag, and not `REPO_POLICIES.md` (see
`.prettierignore`), a copy of the one in `sneak/prompts`.
formats the markdown only, not the HTML templates, as it cannot parse a Go
template action inside a tag.
## Load Test
+258 -105
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-09-08
last_modified: 2026-10-15
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -54,34 +54,39 @@ style conventions are in separate documents:
`cibuild`. `script/bootstrap` installs all dependencies idempotently and
assumes nothing is present: base tools come from nix, apt, brew, or apk
(detected in that order; apt runs noninteractive). For node it uses the
installed node if present; otherwise it installs a PINNED node version via
nvm, first installing nvm itself if missing — from a hash-verified GitHub
release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
installed node only when its major version is the pinned one; otherwise (node
missing, or another major, such as the node 24 the CI runner image ships) it
installs the PINNED node version via nvm and installs yarn and the packages
under it. nvm itself is installed first if missing, from a hash-verified
GitHub release archive (never `curl | sh`), with bash installed as an explicit
prerequisite since nvm requires bash. Only the major version is compared
because the `Dockerfile` stages start from a node image whose exact version is
not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
via `corepack prepare yarn@<version> --activate`. Never install "latest" or
"lts"; always exact versions. `script/cibuild` runs the CI build: it changes
to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
image with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
containerised gates have passed. **The bootstrap alone is not enough**: when
`script/bootstrap` installs node and yarn under nvm it leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
under another node. The host entrypoints that need yarn — `script/fmt` and
`script/fmt-check` — therefore use the same test as `script/bootstrap`: when
the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
pinned node. A runner carrying nothing but docker and git then gets through
`script/check`. Four further scripts are our own extensions to the standard:
`script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
`script/precommit` is what the git pre-commit hook runs, and it calls
`script/check`; `script/install-precommit` installs the git pre-commit hook
(the `make hooks` target shims to it); and `script/projectname` (literally
that filename) simply outputs the project's name. Scripts that need the name
call `script/projectname` — e.g. `script/docker` assembles its image tag from
it — so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -104,18 +109,24 @@ style conventions are in separate documents:
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
Dockerfiles install development prerequisites by running `script/bootstrap`
rather than duplicating installs inline; COPY `script/` and the dependency
manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
running it.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target lint --output type=cacheonly .
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target test --output type=cacheonly .
```
**A stage that is not the last one in the file is built only when the final
@@ -125,10 +136,15 @@ style conventions are in separate documents:
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
**The gate builds write no image.** With `--output type=cacheonly` the phase
runs and a failing step fails the build, but the result is not exported.
Nothing uses those images, and writing one out is slow: a Go test phase's
image holds the toolchain. A build given neither `--output` nor `-t` writes
an untagged image and leaves it dangling, on every developer host and every
CI runner. `script/cibuild` and `script/docker` build the image that ships
and tag it, so each build replaces the previous image; each assigns the tag
on its own line before the build, so `set -e` stops it where
`script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -143,66 +159,107 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
org, in both directions. A container has its own golangci-lint cache, its
own `TMPDIR` and a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
invalidates a `COPY` layer only when the copied content changes, so on an
unchanged tree the check `RUN` is served from cache, nothing executes, and the
build still exits 0. Every `docker build` in `script/` therefore passes
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
id, so no two builds share a value, not even the three in one
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. Every
stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
`FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
are compared by content and need nothing. A stage without the `ARG` line is
served from cache on an unchanged tree, so a repository re-vendors the scripts
and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
it reruns every step too, but it also empties every cache mount, and Go's
build cache, which the canonical Go `Dockerfile` below mounts so that compiled
packages survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every
other build on the host. When a check is added or changed, prove it works by
planting a defect it must catch and watching the run fail on it, then revert
the defect. A green run alone shows neither that the check ran nor that it
covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Go image. The canonical Go repo
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Every stage declares ARG CACHEBUST first: script/ passes a new value
# on every build, which reruns every RUN step after it. The steps that
# compile mount Go's build cache, which survives from build to build.
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
ARG CACHEBUST
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN golangci-lint run --config .golangci.yml ./...
RUN --mount=type=cache,target=/root/.cache/go-build \
golangci-lint run --config .golangci.yml ./...
# Test phase
# golang:1.x-alpine, YYYY-MM-DD
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not. -count=1 because the mounted
# cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
ARG CACHEBUST
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN go test -timeout 90s -race -cover ./... || \
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN --mount=type=cache,target=/root/.cache/go-build \
VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:...
ARG CACHEBUST
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
@@ -217,23 +274,88 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
it for root), and only on the steps that compile: `golangci-lint run`,
both `go test` runs and `go build`. It keeps compiled packages between
check builds, and Go reuses one only when all its inputs are unchanged, so
it saves the compile without skipping a check. Nothing else is kept:
golangci-lint's own cache would replay stored findings in place of a lint
run, and Go builds from whatever files a kept module cache holds without
checking them again (only `go mod verify` does), so a file changed there
would go unnoticed. Go's build cache also holds test results, which is why
both `go test` runs carry `-count=1`.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint phase with `apk add`.
- `ARG VERSION=dev` is declared in the stage that compiles and supplied by
`script/docker` and `script/cibuild`; no stage may call `git describe`.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
A checkout whose `.git` is a file (a linked worktree, or a repository
checked out as a submodule) is the exception: that file points to a git
directory outside the build context, so the build cannot read the version
and a plain `docker build .` fails; pass the version with
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
runs `script/cibuild` on push, and checks out the repo as its only other step,
with `persist-credentials: false`: `script/cibuild` needs no token, and
without it the checkout leaves the job's token in `.git/config` for every
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
tags `git describe` needs: by default it clones shallow with no tags, and a
tagged repository's CI build would stamp a bare short commit id. The
workflow's `concurrency` block groups runs by workflow and branch
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
so a new push cancels the older run on the same branch, queued or running, and
no other: runs for replaced commits do not hold up the shared runner.
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry.
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
second time. That is the price of the rule above, and it is worth paying: the
image that ships is built from a run of its own gates rather than from a cache
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
shared runner after 20 minutes. That allows for the three Docker builds
described above (the test phase, the lint phase, then the image), each held to
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
limited to `main` by a `branches` list under `on: push` cannot be checked by
review: to try a change to it, add the feature branch to that list and push,
then remove the branch from the list again before merging. Keep any job in it
that publishes behind `if: github.ref_name == 'main'`, so the run from the
feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -286,17 +408,18 @@ style conventions are in separate documents:
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so the target cannot report a pass it did not earn, and the rerun
reproduces a failure instead of replaying it. It leaves the build cache
alone, so it costs the runtime of the suite and no recompilation.
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
Note that this is a second, independent cache, stacked below the Docker
layer cache that [issue #26](https://git.eeqj.de/sneak/prompts/issues/26)
addresses. `CHECK_EPOCH` guarantees the `RUN make test` _step_ re-executes;
it does not guarantee `go test` inside that step does any work, because the
`GOCACHE` baked into earlier image layers survives into the re-executed
step. They are two separate defects requiring two separate fixes, and a fix
for one must not be recorded as covering the other.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. The directory lasts from one run to the next on a
developer's machine, where this target runs, and in the `test` phase of the
`Dockerfile`, which mounts it to keep compiled packages between check
builds, so the phase's `go test` runs carry `-count=1` too. A new
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
Python example:
@@ -324,9 +447,12 @@ style conventions are in separate documents:
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
`node_modules/`, and the repo's own build outputs. Fetch the standard
`.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
@@ -340,7 +466,7 @@ style conventions are in separate documents:
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.git`, and the repo's own host-built
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
@@ -365,21 +491,24 @@ style conventions are in separate documents:
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **Excluding `.git` means `git describe` cannot run inside any build stage, and
it fails quietly there.** In a build stage there is no repository, so
`git describe` writes nothing to stdout, `-X main.Version=` comes out empty,
the binary reports no version at all, and the build still exits 0. Compute the
version on the host and thread it in as a build arg. `script/docker` and
`script/cibuild` do this, byte-identically across repos:
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
# The version and the tag each get their own line: a failing command
# substitution inside an argument does not trip `set -e`, so the inline
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
tag="$(script/projectname)"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
-t "$tag" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
@@ -387,12 +516,12 @@ style conventions are in separate documents:
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION=dev` in the stage that compiles, declared
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
checkout action clones shallow and fetches no tags, so the canonical
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
@@ -426,12 +555,18 @@ style conventions are in separate documents:
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.12.2 (released 2026-05-06), pinned as the digest of the lint phase's base
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240`,
which reports `2.12.2 built with go1.26.2 from c0d3ddc9`). That digest is the
only pin, since no repo installs golangci-lint on the host: bumping the
version means changing it and nothing else.
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
@@ -453,8 +588,18 @@ style conventions are in separate documents:
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Node is the exception to the whole-token comparison: the canonical
`script/bootstrap` compares only its major version, because the node 22
alpine image the `Dockerfile` stages start from is not the exact pin, and
nvm cannot install a prebuilt node on alpine.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -564,13 +709,17 @@ style conventions are in separate documents:
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
settings: the standard file from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
@@ -601,3 +750,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
+59 -1
View File
@@ -18,7 +18,7 @@
pre-1.0. No git tags exist. The `1.0.0` milestone is in progress; work lands on
`next`, and `main` receives only the milestone PR that the owner merges. `next`
is at the canonical `golangci-lint` v2.12.2 config and is green. Recent work
is at the canonical `golangci-lint` v2.14.0 config and is green. Recent work
extracted the internal/magic, internal/allowlist, internal/httpfetcher, and
internal/signature packages. The gosec findings from the 2026-07-06 survey are
resolved. The disk cache is now size-bounded with LRU eviction
@@ -30,6 +30,64 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-15 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
an alpha channel took about 61 seconds to encode as WebP, 96 as AVIF and 81 as
JPEG XL, so with the default `downstream_timeout` of 60 seconds such an image
was never delivered, yet its encode held a processing slot to the end. WebP is
now saved at effort 2 and JPEG XL at effort 4, which take up to about 23 and
27 seconds. AVIF was already at the lowest effort govips can set, so an AVIF
output of more than 16,777,216 pixels is refused with 400 before anything is
decoded; 4096x4096, 8192x2048 or 2048x8192 takes up to about 34 seconds. Each
time is the slowest at any quality. Resizing with `fit=cover` already decodes
the input, so the output's dimensions are worked out before resizing.
`README.md` gives the limit and the new times.
- 2026-10-15 the README says how a source URL with a query is written into an
image URL, and the image route accepts it (closes #255): the query follows the
source path after `%3F`. pixa now reads the image URL's path decoded once, so
the `?` before the query, any `?` in it and every `%` in the source path or
query must be percent-encoded, and any other character may be. The route used
to parse chi's wildcard, which holds the path as the client wrote it whenever
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
writes Go's own encoding of the path. The README has an example with a query
and its signature. The path traversal check no longer reads the source query,
and keeps a `%` that does not start a percent-escape as it is instead of
refusing the URL, so a source path or query holding `%25` is accepted, while a
double-encoded `..` next to such a `%` is still refused.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
`.prettierignore`, `.prettierrc`, `REPO_POLICIES.md` and the standard scripts
are its copies. pixa's own entries follow the standard part: `/bin`, `/data`
and the local config files in `.dockerignore`; Go's `*.log`, `*.out` and
`*.test`, `vendor/`, the binaries, `/data/`, `*.sqlite3` and the local config
files in `.gitignore`. The workflow keeps its `script/docker-smoke` step while
https://git.eeqj.de/sneak/pixa/issues/203 is open. The scripts pass a new
`CACHEBUST` build argument in place of `--no-cache`, and every `Dockerfile`
stage declares it first, so every `RUN` step still runs on every build while
Go's build cache, mounted on the steps that compile, is kept; both `go test`
runs keep `-count=1`. The lint and test builds write no image. The lint phase
runs golangci-lint v2.14.0, the test phase runs on the Debian Go image, and
the build stage marks `/src` safe for git. `script/bootstrap`, `script/fmt`
and `script/fmt-check` use the node on the `PATH` only when it has the pinned
major version. Of golangci-lint v2.14.0's two findings, an unused `nolint`
directive is removed, and gosec's new path check is marked as not applying to
the fixed config file locations. The agent guidance moves unchanged from
`CLAUDE.md` to a root `AGENTS.md`, where `REPO_POLICIES.md` keeps it.
- 2026-10-15 the request metrics record any method that is not a standard one as
`other` (closes #274): their `method` label was each request's method as sent,
and Go's HTTP server accepts any method name, so a client sending made-up
+5 -4
View File
@@ -108,10 +108,11 @@ upstream_connections_per_host: 20
# downstream_timeout has ended first.
upstream_connections: 64
# Maximum number of images decoded and encoded at once (default: the
# number of CPUs pixa can use, which follows a container's CPU limit). A
# request that finds none free waits up to 10 seconds for one, and if none
# frees up it is answered 503, unless downstream_timeout has ended first.
# Maximum number of images decoded and encoded at once (default: half the
# number of CPUs pixa can use, at least 1; the number of CPUs follows a
# container's CPU limit). A request that finds none free waits up to 10
# seconds for one, and if none frees up it is answered 503, unless
# downstream_timeout has ended first.
# max_concurrent_processing: 4
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
@@ -1,6 +1,7 @@
package config
import (
"fmt"
"runtime"
"testing"
)
@@ -12,8 +13,8 @@ const (
)
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
// max_concurrent_processing is the number of CPUs Go uses and an omitted
// upstream_connections is 64.
// max_concurrent_processing is half the CPUs Go uses, and at least 1, and
// an omitted upstream_connections is 64.
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Parallel()
@@ -22,9 +23,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU",
c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0))
if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) {
t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs",
c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1))
}
if c.UpstreamConnections != 64 {
@@ -32,6 +33,36 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
}
}
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
// that an omitted max_concurrent_processing is half the CPUs Go uses,
// rounded down, and 1 when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
tc.cpus, c.MaxConcurrentProcessing)
if c.MaxConcurrentProcessing != tc.want {
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
tc.cpus, c.MaxConcurrentProcessing, tc.want)
}
}
}
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
// two limits are used as given.
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
@@ -160,3 +191,19 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
})
}
}
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
// least 1, an empty one included, aborts startup naming the variable and
// the key instead of taking the default.
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
t.Setenv(testMaxConcurrentProcessingVar, value)
_, err := configFromYAML(t, signingKeyLine)
wantStartupError(t, err,
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
})
}
}
+10 -3
View File
@@ -37,6 +37,14 @@ const (
DefaultImageRequestsPerMinute = 600
)
// DefaultMaxConcurrentProcessing returns the most images processed at once
// when max_concurrent_processing is omitted: half the CPUs Go uses
// (runtime.GOMAXPROCS(0), which follows a container's CPU limit), rounded
// down, and at least 1.
func DefaultMaxConcurrentProcessing() int {
return max(runtime.GOMAXPROCS(0)/2, 1) //nolint:mnd // half the CPUs
}
// Configuration key names.
const (
keyDebug = "debug"
@@ -297,10 +305,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
UpstreamConnections: loader.intVal(
keyUpstreamConnections, DefaultUpstreamConnections),
// Decoding and encoding are CPU-bound, so the default is one image
// per CPU Go uses, which follows a container's CPU limit.
MaxConcurrentProcessing: loader.intVal(
keyMaxConcurrentProcessing, runtime.GOMAXPROCS(0)),
keyMaxConcurrentProcessing, DefaultMaxConcurrentProcessing()),
UpstreamFetchTimeout: loader.durationVal(
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
UpstreamMaxResponseSize: loader.int64Val(
@@ -823,6 +829,7 @@ func loadConfigFile(log *slog.Logger, appName string) (*smartconfig.Config, erro
// one whose path runs through a file, such as under a HOME of
// /dev/null. One that cannot be read or does not parse is a
// fatal startup error.
//nolint:gosec // G703: a fixed config location; HOME is the operator's
_, statErr := os.Stat(cleanPath)
if errors.Is(statErr, fs.ErrNotExist) || errors.Is(statErr, syscall.ENOTDIR) {
continue
@@ -0,0 +1,49 @@
package handlers
import (
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
// output above it, wrapped as the service wraps it. The request's source only
// goes into a log line, so it is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("image processing failed: %w: 4097x4096",
imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("%s: body = %q, want it to say %q",
name, rec.Body.String(), wantBody)
}
}
}
+16 -8
View File
@@ -9,7 +9,6 @@ import (
"strconv"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
@@ -80,20 +79,21 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
}
}
// parseImageRequest parses the wildcard path and query parameters into
// parseImageRequest parses the image URL's path and query parameters into
// an ImageRequest. On invalid input it writes an error response and
// returns false.
func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) {
// Get the wildcard path from chi
pathParam := chi.URLParam(r, "*")
// Parse the URL path
parsed, err := imgcache.ParseImagePath(pathParam)
// The path is parsed decoded once, as r.URL.Path holds it, and not from
// chi's wildcard: chi routes on the path as the client wrote it whenever
// that is not Go's own encoding of it, so its wildcard would still hold
// the %3F before a source query when, say, a "/" in the query is written
// as %2F
parsed, err := imgcache.ParseImageURL(r.URL.Path)
if err != nil {
s.log.Warn("failed to parse image URL",
"path", pathParam,
"path", r.URL.Path,
"error", err,
)
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
@@ -264,6 +264,14 @@ func (s *Handlers) respondImageError(
return
}
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
}
@@ -169,3 +169,166 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
})
}
}
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
// by the service's signed-URL generator for source URLs with a query to the
// route, through a chi router: chi routes on the path as the client wrote it
// whenever that is not Go's own encoding of it. A 502 means the signature
// passed, so the route read the source path and query that were signed.
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
query string
}{
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
{"semicolon", "a=1;b=2"},
{"comma", "crop=0,0,50,50"},
{"equals and ampersand", "a=1&b=2"},
{"percent-escape", "name=my%20cat"},
{"question mark", "next=/a?b=1"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: tt.query,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
// and writes it into the image URL in ways the README allows: the "?" before
// the query and each "%" in it percent-encoded, and every other character as
// itself or percent-encoded. A 502 means the signature passed.
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
tests := []struct {
name string
writtenQuery string
}{
{"only ? and % encoded",
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
{"every reserved character encoded",
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
{"lower-case hex digits",
"%3furl=https://origin.example.com/my%2520cat.jpg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
// source query that each hold %25, the percent-escape of a literal "%", and
// sends each to the route written as the README says, with that "%" written as
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourcePath string
sourceQuery string
// writtenPathAndQuery is the source path and query as the README says
// to write them into the image URL
writtenPathAndQuery string
}{
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
}
for _, tt := range tests {
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
+4
View File
@@ -170,6 +170,10 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
default:
s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError)
@@ -0,0 +1,64 @@
package imageprocessor
import (
"bytes"
"errors"
"testing"
)
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
// before anything is decoded, whatever size and fit give it, and that an AVIF
// output of at most that many pixels, or one in another format, is not. The
// input is a 64x32 PNG cut off in its pixel data: libvips reads its size from
// its header but fails when it decodes it, so a request that gets that failure
// was not refused, and a refusal came before any decoding.
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
t.Parallel()
whole := createTestPNG(t, 64, 32)
cutOff := whole[:len(whole)/2]
tests := []struct {
name string
format Format
width, height int
fit FitMode
refused bool
}{
{"AVIF at the limit", FormatAVIF, 4096, 4096, FitCover, false},
{"AVIF above the limit", FormatAVIF, 4096, 4097, FitCover, true},
{"WebP above the AVIF limit", FormatWebP, 4096, 4097, FitCover, false},
// FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", FormatAVIF, 8192, 8192, FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", FormatAVIF, 8192, 1, FitOutside, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(cutOff),
&Request{
Size: Size{Width: tt.width, Height: tt.height},
Format: tt.format,
FitMode: tt.fit,
})
t.Logf("Process() error = %v", err)
if tt.refused {
if !errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
}
return
}
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want the failure to decode "+
"the cut-off input", err)
}
})
}
}
@@ -77,10 +77,10 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
}
}
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at
// libvips' default effort, 4: the output is the size of the same image saved
// at that effort.
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
// TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
// not libvips' default of 4: the output is the size of the same image saved
// at effort 2.
func TestImageProcessor_WebPAtEffort2(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 200, 150)
@@ -90,14 +90,14 @@ func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
StripMetadata: true,
Quality: 85,
ReductionEffort: 4,
ReductionEffort: 2,
})
if err != nil {
t.Fatalf("ExportWebp() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4",
t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
size, len(want))
}
}
@@ -143,3 +143,29 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
"and 8 bits", size, len(want))
}
}
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
// effort 4, not libvips' default of 7: the output is the size of the same
// image saved at effort 4, with the distance exportJXL gives quality 85.
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
params := vips.NewJxlExportParams()
params.Quality = 85
params.Distance = jxlDistance(85)
params.Effort = 4
want, _, err := decode(t, input).ExportJxl(params)
if err != nil {
t.Fatalf("ExportJxl() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
size, len(want))
}
}
+147 -83
View File
@@ -7,11 +7,11 @@ import (
"errors"
"fmt"
"io"
"runtime"
"sync"
"time"
"github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/config"
)
// vipsOnce ensures vips is initialized exactly once.
@@ -20,11 +20,10 @@ import (
var vipsOnce sync.Once
// initVips initializes libvips with quiet logging, one worker thread per
// image and no operation cache. Process already works on one image per CPU
// by default, so more threads per image would only compete for the CPUs.
// Each request decodes different source bytes, so the operation cache
// would rarely be hit and would hold memory outside MaxConcurrentProcessing;
// repeated requests are served from pixa's disk cache instead.
// image and no operation cache. Each request decodes different source
// bytes, so the operation cache would rarely be hit and would hold memory
// outside MaxConcurrentProcessing; repeated requests are served from
// pixa's disk cache instead.
func initVips() {
vipsOnce.Do(func() {
vips.LoggingSettings(nil, vips.LogLevelError)
@@ -143,6 +142,19 @@ const MaxOutputDimension = 8192
// requested size.
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
// encode takes at most 40 seconds of the default downstream_timeout of 60
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
// 2048x8192 random pixels with an alpha channel takes up to about 34
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
// 5120x4096 at 8 bits already takes about 36.
const MaxAVIFOutputPixels = 4096 * 4096
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
// output would have more than MaxAVIFOutputPixels pixels.
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
// ErrUnsupportedOutputFormat is returned when the requested output format is
// not supported.
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
@@ -179,7 +191,7 @@ type Params struct {
// If <= 0, DefaultMaxInputBytes is used.
MaxInputBytes int64
// MaxConcurrentProcessing is the most images processed at once.
// If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used.
// If <= 0, config.DefaultMaxConcurrentProcessing() is used.
MaxConcurrentProcessing int
}
@@ -195,7 +207,7 @@ func New(params Params) *ImageProcessor {
maxConcurrentProcessing := params.MaxConcurrentProcessing
if maxConcurrentProcessing <= 0 {
maxConcurrentProcessing = runtime.GOMAXPROCS(0)
maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing()
}
return &ImageProcessor{
@@ -211,7 +223,8 @@ func New(params Params) *ImageProcessor {
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
// no way to stop libvips once an encode has begun, so an encode keeps its
// slot until it finishes.
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
// refused with ErrAVIFOutputTooLarge before anything is decoded.
func (p *ImageProcessor) Process(
ctx context.Context,
input io.Reader,
@@ -224,18 +237,9 @@ func (p *ImageProcessor) Process(
defer release()
// Read input with a size limit to prevent unbounded memory consumption.
// We read at most maxInputBytes+1 so we can detect if the input exceeds
// the limit without consuming additional memory.
limited := io.LimitReader(input, p.maxInputBytes+1)
data, err := io.ReadAll(limited)
data, err := p.readInput(input)
if err != nil {
return nil, fmt.Errorf("failed to read input: %w", err)
}
if int64(len(data)) > p.maxInputBytes {
return nil, ErrInputDataTooLarge
return nil, err
}
// govips picks the libvips loader from the input's first bytes with
@@ -278,20 +282,28 @@ func (p *ImageProcessor) Process(
return nil, err
}
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
// orig is the source's own format; encode refuses an empty format
outputFormat := req.Format
if outputFormat == FormatOriginal {
outputFormat = sourceFormat
}
// Determine output dimensions; too large ones are refused before
// resizing, which with FitCover already decodes the input
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
targetWidth, targetHeight, req.FitMode, outputFormat)
if err != nil {
return nil, err
}
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
// Encode to target format
output, err := p.encode(ctx, img, outputFormat, req.Quality)
if err != nil {
@@ -444,6 +456,23 @@ func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
return release, nil
}
// readInput reads input with a size limit to prevent unbounded memory
// consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
// reads at most maxInputBytes+1 so it can detect if the input exceeds the
// limit without consuming additional memory.
func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
if err != nil {
return nil, fmt.Errorf("failed to read input: %w", err)
}
if int64(len(data)) > p.maxInputBytes {
return nil, ErrInputDataTooLarge
}
return data, nil
}
// detectFormat returns the format string for a govips image type.
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
switch imageType {
@@ -468,9 +497,61 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
}
}
// resize resizes the image according to the fit mode. With FitOutside one
// side can come out larger than requested; when it would be above
// MaxOutputDimension, resize returns ErrOutputTooLarge without resizing.
// outputDimensions returns the dimensions resize gives an image of imgWidth x
// imgHeight for the target dimensions and fit mode, without resizing it, and
// refuses an output too large for format. With FitOutside one side can come
// out larger than the target; when it would be above MaxOutputDimension,
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
func outputDimensions(
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
) (int, int, error) {
scaleW := float64(width) / float64(imgWidth)
scaleH := float64(height) / float64(imgHeight)
outputWidth, outputHeight := width, height
switch fit {
case FitCover, FitFill, "":
// Cover crops, and fill stretches, to exactly the target dimensions
case FitContain:
// Fit within the target dimensions, maintaining aspect ratio
scale := min(scaleW, scaleH)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
case FitInside:
// Same as contain, but only shrink
scale := min(scaleW, scaleH, 1)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
case FitOutside:
// The smallest dimension fits, the other may exceed the target
scale := max(scaleW, scaleH)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
default:
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
}
// The side FitOutside scales past the target is held to the maximum too
if max(outputWidth, outputHeight) > MaxOutputDimension {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
outputWidth, outputHeight, MaxOutputDimension)
}
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
}
return outputWidth, outputHeight, nil
}
// resize resizes the image to width x height, the dimensions
// outputDimensions gives for the fit mode.
func (p *ImageProcessor) resize(
img *vips.ImageRef, width, height int, fit FitMode,
) error {
@@ -479,52 +560,20 @@ func (p *ImageProcessor) resize(
// Resize and crop to fill exact dimensions (default)
return img.Thumbnail(width, height, vips.InterestingCentre)
case FitContain:
// Resize to fit within dimensions, maintaining aspect ratio
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitFill:
// Resize to exact dimensions (may distort)
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
case FitInside:
// Same as contain, but only shrink
if img.Width() <= width && img.Height() <= height {
return nil // Already fits
// outputDimensions keeps the dimensions of an image that already fits
if img.Width() == width && img.Height() == height {
return nil
}
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(width, height, vips.InterestingNone)
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitOutside:
// Resize so smallest dimension fits, may exceed target on other dimension
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := max(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
// The dimension that exceeds the target is held to the maximum too
if newW > MaxOutputDimension || newH > MaxOutputDimension {
return fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
newW, newH, MaxOutputDimension)
}
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitContain, FitOutside:
return img.Thumbnail(width, height, vips.InterestingNone)
default:
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
@@ -628,11 +677,16 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
return output, err
}
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6.
const webpEffort = 4
// webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
// default of 4 so that an encode takes at most 40 seconds of the default
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
// saving an 8192x8192 image of random pixels with an alpha channel takes up
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
const webpEffort = 2
// exportWebP encodes img as lossy WebP at quality and libvips' default
// effort, without metadata.
// exportWebP encodes img as lossy WebP at quality and webpEffort, without
// metadata.
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportWebp(&vips.WebpExportParams{
StripMetadata: true,
@@ -644,12 +698,13 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
}
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
// seconds to save an 8192x8192 RGB image of random pixels, and about 110
// with an alpha channel, the worst case, against the default
// downstream_timeout of 60 seconds. On an image of milder noise, which 1
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default,
// 4, takes minutes.
// govips can set. With one thread, as pixad runs libvips, 1 takes about 52
// seconds to save an 8192x8192 RGB image of random pixels at quality 85,
// and about 96 with an alpha channel, against the default
// downstream_timeout of 60 seconds, so an AVIF output is held to
// MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
// 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
// minutes.
const avifEffort = 1
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
@@ -676,10 +731,18 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
// JPEG that names none.
const jxlResolution = 72 / 25.4
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
// and without metadata. govips sends libvips a distance, the JPEG XL
// encoder's own measure of quality, along with the quality, and libvips then
// uses the distance alone, so the quality is also given as a distance.
// jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
// libvips' default of 7 so that an encode takes at most 40 seconds of the
// default downstream_timeout of 60 seconds. With one thread, as pixad runs
// libvips, saving an 8192x8192 image of random pixels with an alpha channel
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
const jxlEffort = 4
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
// measure of quality, along with the quality, and libvips then uses the
// distance alone, so the quality is also given as a distance.
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
// PNG, but cannot save one as JPEG XL. encode has already converted
@@ -718,6 +781,7 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
params := vips.NewJxlExportParams()
params.Quality = quality
params.Distance = jxlDistance(quality)
params.Effort = jxlEffort
output, _, err := toSave.ExportJxl(params)
if err != nil {
@@ -115,14 +115,14 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
}
}
func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
t.Parallel()
for _, limit := range []int{0, -1} {
proc := New(Params{MaxConcurrentProcessing: limit})
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
limit, got, runtime.GOMAXPROCS(0))
if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs",
limit, got, max(runtime.GOMAXPROCS(0)/2, 1))
}
}
@@ -132,6 +132,30 @@ func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
}
}
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
// rounded down, and one slot when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
got := cap(New(Params{}).processingSemaphore)
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
if got != tc.want {
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
}
}
}
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
// than MaxConcurrentProcessing allows and holds the first ones inside
// Process until the test lets them go. No more than the limit may be
-4
View File
@@ -78,10 +78,6 @@ func (c *Cache) computeDefaultMaxBytes(
computed := spaceBytes / freeSpaceFractionDenominator * freeSpaceFractionNumerator
computed = min(computed, math.MaxInt64)
// gosec cannot see that min() above bounds computed, so it reads
// this conversion as potentially overflowing. It cannot: computed is
// at most math.MaxInt64 on every path here.
//nolint:gosec // G115: clamped to MaxInt64 by min above
limit := int64(computed)
limit = max(limit, DefaultCacheMaxBytesFloor)
+1 -1
View File
@@ -53,7 +53,7 @@ type ServiceConfig struct {
// Allowlist is the list of hosts that don't require signatures
Allowlist []string
// MaxConcurrentProcessing is the most images processed at once; zero
// uses the image processor's default, one per CPU
// uses the image processor's default, half the CPUs
MaxConcurrentProcessing int
// Logger for logging
Logger *slog.Logger
+29 -18
View File
@@ -47,6 +47,9 @@ func ValidateDimension(name string, value int) error {
var sizeFormatRegex = regexp.MustCompile(
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
// percentEscapeRegex matches one percent-escape, such as "%2e"
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
// ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct {
// Host is the source origin host (e.g., "cdn.example.com")
@@ -61,9 +64,8 @@ type ParsedURL struct {
Format ImageFormat
}
// ParseImagePath parses the path captured by chi's wildcard:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
// This is the primary entry point when using chi routing.
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
// Examples:
// - cdn.example.com/photos/cat.jpg/800x600.webp
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
@@ -79,9 +81,10 @@ func ParseImagePath(path string) (*ParsedURL, error) {
return parseImageComponents(path)
}
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
// or /v1/image/<host>/<path>/<size> for JPEG XL
// Use ParseImagePath instead when working with chi's wildcard capture.
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
// for JPEG XL. A source query follows the source path after a "?", which the
// URL writes as %3F.
func ParseImageURL(urlPath string) (*ParsedURL, error) {
// Remove the /v1/image/ prefix
const prefix = "/v1/image/"
@@ -100,12 +103,6 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
// parseImageComponents parses <host>/<path>/<size>.<format>, or
// <host>/<path>/<size> for JPEG XL.
func parseImageComponents(remainder string) (*ParsedURL, error) {
// Check for path traversal before any other processing
err := checkPathTraversal(remainder)
if err != nil {
return nil, err
}
// Find the last path segment, which holds "size" or "size.format"
lastSlash := strings.LastIndex(remainder, "/")
if lastSlash == -1 {
@@ -161,6 +158,13 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
path = path[:qIndex]
}
// The source query names no file on the upstream, so only the host and
// the source path are checked for path traversal
err = checkPathTraversal(host + path)
if err != nil {
return nil, err
}
// Ensure path starts with /
if !strings.HasPrefix(path, "/") {
path = "/" + path
@@ -186,12 +190,7 @@ func checkPathTraversal(path string) error {
// Decode multiple times to catch double-encoding
decoded := path
for range 3 {
newDecoded, err := url.PathUnescape(decoded)
if err != nil {
// Malformed encoding is suspicious
return ErrPathTraversal
}
newDecoded := decodePercentEscapes(decoded)
if newDecoded == decoded {
break
}
@@ -232,6 +231,18 @@ func checkPathTraversal(path string) error {
return nil
}
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
// keeps a "%" that does not start one as it is, where url.PathUnescape would
// refuse the whole string. A source path holding %25, a literal "%", has such
// a "%" after one more decoding.
func decodePercentEscapes(s string) string {
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
decoded, _ := url.PathUnescape(escape)
return decoded
})
}
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
func parseSizeFormat(s string) (Size, ImageFormat, error) {
matches := sizeFormatRegex.FindStringSubmatch(s)
@@ -407,6 +407,71 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
}
}
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
// a source query holding %25, the percent-escape of a literal "%", are
// accepted as they are, although decoding them once more leaves a "%" that
// does not start a percent-escape.
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
want *ParsedURL
}{
{
name: "path",
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: "/sale/50%25off.jpg",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
{
name: "query",
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImageURL(tt.input)
if err != nil {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
})
}
}
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
// double-encoded ".." is still refused when the path also holds a "%" that
// does not start a percent-escape, either as written or after one more
// decoding.
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
t.Parallel()
for _, input := range []string{
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
} {
_, err := ParseImageURL(input)
t.Logf("ParseImageURL(%q) error = %v", input, err)
if !errors.Is(err, ErrPathTraversal) {
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
input, err)
}
}
}
// errorIs checks if err matches target (handles wrapped errors).
func errorIs(err, target error) bool {
if errors.Is(err, target) {
+11 -22
View File
@@ -126,31 +126,20 @@ func (s *Signer) GenerateSignedURL(
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
}
// Build the path.
// When a source query is present, it is embedded as a path segment
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract
// it from the last-slash split. The "?" inside a path segment is
// percent-encoded by clients but chi delivers it decoded, which is
// exactly what the URL parser expects.
var path string
// Build the path. A source query follows the source path after a "?".
// The image route reads the path decoded once, so the whole path is
// written in Go's own percent-encoding of a URL path, which writes "?"
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
// are.
sourcePathAndQuery := req.SourcePath
if req.SourceQuery != "" {
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
req.SourceHost,
req.SourcePath,
url.PathEscape(req.SourceQuery),
sizeStr,
req.Format,
)
} else {
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost,
req.SourcePath,
sizeStr,
req.Format,
)
sourcePathAndQuery += "?" + req.SourceQuery
}
return path, sig, exp
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
}
// buildSignatureData creates the string to be signed.
+38
View File
@@ -475,6 +475,44 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
}
}
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
// source URL with a query, the image at
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
// signature with the README's key, and the path GenerateSignedURL writes for
// it, with the "?" written as %3F and the "%" as %25.
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
t.Parallel()
signer := signature.New("example-signing-key-for-documentation")
req := signature.Request{
SourceHost: testHost,
SourcePath: "/img",
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Unix(1704067200, 0),
}
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
gotSig := signer.Sign(&req)
if gotSig != wantSig {
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
}
const wantPath = "/v1/image/cdn.example.com" +
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
if gotPath != wantPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
}
}
func TestParseParams(t *testing.T) {
t.Parallel()
+33 -12
View File
@@ -3,11 +3,13 @@
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes NOTHING is present (not git,
# make, or go). Node is used directly if installed; otherwise it is
# installed at a pinned version via nvm (installing nvm itself first,
# from a hash-verified release archive, never curl | sh). The linter is
# never installed on the host: golangci-lint runs only in the lint phase
# of the Dockerfile (see script/lint).
# make, or go). The installed node is used only when its major version
# is the pinned one; otherwise (node missing, or another major) the
# pinned version is installed via nvm (installing nvm itself first,
# from a hash-verified release archive, never curl | sh), and yarn and
# the packages are installed under it. The linter is never installed on
# the host: golangci-lint runs only in the lint phase of the Dockerfile
# (see script/lint).
#
# script/bootstrap git, make, Go, and Node, Yarn and the
# prettier in yarn.lock for script/fmt and
@@ -110,31 +112,50 @@ ensure_nvm() {
rm -rf "$tmp"
}
# True when the node on PATH has the pinned major version. Only the
# major is compared, as REPO_POLICIES.md asks. Another major is not
# used: the pinned yarn 1 prints a deprecation warning under node 24.
# script/fmt and script/fmt-check carry this function unchanged, to run
# the yarn installed here.
node_is_pinned_major() {
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
ensure_node() {
if ! missing node; then return 0; fi
if node_is_pinned_major; then
echo "bootstrap: using node $(node --version)"
return 0
fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
nvm_sh "nvm use $NODE_VERSION >/dev/null && \
echo \"bootstrap: using node \$(node --version)\""
}
ensure_yarn() {
if ! node_is_pinned_major; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
return 0
fi
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
if node_is_pinned_major; then
yarn install --frozen-lockfile
else
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
+13 -10
View File
@@ -2,9 +2,11 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -14,16 +16,17 @@ main() {
cd "$ROOT"
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore
# excludes .git, so `git describe` in a build stage yields an empty
# version without failing.
# The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
tag="$("$SCRIPT_DIR/projectname")"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
-t "$tag" .
}
main "$@"
+13 -9
View File
@@ -1,8 +1,11 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
# A new CACHEBUST value reruns every RUN step, since a cached gate phase
# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# first; a stage without that line is served from cache on an unchanged
# tree, with no warning.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -10,16 +13,17 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. VERSION is computed here because .dockerignore
# excludes .git, so `git describe` in a build stage yields an empty
# version without failing.
# The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
tag="$("$SCRIPT_DIR/projectname")"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
-t "$tag" .
}
main "$@"
+16 -6
View File
@@ -7,15 +7,25 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
# True when the node on PATH has the pinned major version: the same
# function as in script/bootstrap, which uses it to decide where it
# installs yarn.
node_is_pinned_major() {
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
# Run the yarn script/bootstrap installed: the yarn on PATH when the
# node on PATH has the pinned major version, and otherwise yarn under
# the pinned node in nvm, which bootstrap leaves off the PATH of the
# shell that called it. nvm is a bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
+16 -6
View File
@@ -8,15 +8,25 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
# True when the node on PATH has the pinned major version: the same
# function as in script/bootstrap, which uses it to decide where it
# installs yarn.
node_is_pinned_major() {
if ! command -v node >/dev/null 2>&1; then return 1; fi
installed="$(node --version)"
installed="${installed#v}"
[ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
}
# Run the yarn script/bootstrap installed: the yarn on PATH when the
# node on PATH has the pinned major version, and otherwise yarn under
# the pinned node in nvm, which bootstrap leaves off the PATH of the
# shell that called it. nvm is a bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
+8 -5
View File
@@ -5,9 +5,12 @@
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
# --target names it. A new CACHEBUST value reruns every RUN step, since
# a cached lint layer is a lint that did not run; unlike --no-cache, it
# keeps Go's build cache. This relies on every stage of the Dockerfile
# declaring ARG CACHEBUST first; a stage without that line is served
# from cache on an unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -15,9 +18,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
--output type=cacheonly .
}
main "$@"
+8 -4
View File
@@ -2,8 +2,12 @@
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
# named, and a new CACHEBUST value because a cached test layer is a test
# that did not run; unlike --no-cache, it keeps Go's build cache. This
# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
# a stage without that line is served from cache on an unchanged tree,
# with no warning. --output type=cacheonly writes no image, since
# nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -11,9 +15,9 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --no-cache \
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
--output type=cacheonly .
}
main "$@"