Author SHA1 Message Date
sneak 47a2c2360a Keep WebP, AVIF and JPEG XL encodes at the limits under 40 seconds (closes #272)
check / check (push) Waiting to run
WebP is saved at effort 2 and JPEG XL at effort 4, and an AVIF output of
more than 16,777,216 pixels is refused with 400 before anything is
decoded, as AVIF is already at the lowest effort govips can set. With
one libvips thread, at any quality, 8192x8192 random pixels with an
alpha channel take up to about 23 seconds as WebP and 27 as JPEG XL, and
4096x4096, 8192x2048 or 2048x8192 up to about 34 as AVIF. Resizing with
fit=cover already decodes the input, so the output's dimensions are now
worked out before resizing, in outputDimensions. Reading the input moves
into readInput to keep Process within the linter's limits.

Model: opus-5-5
2026-10-15 22:47:20 +00:00
sneak 84a86274cd Test lower WebP and JPEG XL effort and the AVIF pixel limit
The WebP test now expects effort 2 instead of libvips' default of 4, a
new test expects JPEG XL at effort 4 instead of 7, and new tests expect an
AVIF output of more than 4096x4096 pixels to be refused before anything
is decoded and answered 400 on both image routes. They fail until the
settings and the refusal are in.

Model: opus-5-5
2026-10-15 22:41:24 +00:00
clawbot 22bfc432b6 Document how a source query goes into an image URL, and accept it (closes #255)
check / check (push) Canceled after 0s
The README gave no way to put a source URL's query into a signed image
URL, and GenerateSignedURL built one that failed: the route parsed
chi's wildcard, which keeps the path as the client wrote it, so one
%2F, %3B or %2C left the %3F undecoded and the signature failed. The
route now parses the decoded path, GenerateSignedURL writes Go's own
encoding of it, and the README says the ? before the query, any ? in it
and every % must be percent-encoded, with a signed example. The path
traversal check no longer reads the query and accepts a literal %, while
a double-encoded .. stays refused. A percent-escape in a source path is
now decoded once.

Model: opus-5-5
2026-10-16 00:17:33 +02:00
clawbot f743930f71 Default max_concurrent_processing to half the CPUs pixa sees (closes #242)
check / check (push) Canceled after 0s
The default max_concurrent_processing was one image per CPU pixa sees,
so in a container with no CPU limit on a 48-CPU host pixa processed 48
images at once. It is now half the CPUs, rounded down and at least 1,
defined once in internal/config; PIXA_MAX_CONCURRENT_PROCESSING stays
the override, and a value that is not a whole number of at least 1
stops startup. "Running under upaas" gives the worst-case memory, 4.2
GiB per image (the largest peak measured over every source format and
kind at 8192x8192) plus the upstream buffers, and the temporary file
libvips keeps under /tmp. The two existing tests expect the new
default, as the owner allowed.

Model: opus-5-5
2026-10-15 23:07:22 +02:00
18 changed files with 831 additions and 168 deletions
+80 -14
View File
@@ -131,6 +131,34 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
the cache can grow to three quarters or more of the space they all have the cache can grow to three quarters or more of the space they all have
left, and to all of it when that is 500 MiB or less. left, and to all of it when that is 500 MiB or less.
- the rest are in the table under Configuration below - the rest are in the table under Configuration below
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
the images it processes at once, plus `upstream_connections` times
`upstream_max_response_size` for the source images it has fetched that wait
for processing. 4.2 GiB is the most resident memory was measured to grow while
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
largest it accepts, saving it at full size to each output format in turn, one
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
use.
- **Temporary files:** while pixa processes a large source, libvips writes its
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
written, so a listing of `/tmp` does not show it, but it keeps its space, on
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
well as written to that file, and the 4.2 GiB above includes that memory. For
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
This `/tmp` is in the container's own filesystem, not on the volume at
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
does not include these files; where `/tmp` is a tmpfs, they are memory on top
of it.
- **Health check:** the image's `HEALTHCHECK` requests - **Health check:** the image's `HEALTHCHECK` requests
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds `/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
after a deploy and marks the deploy failed unless it is `healthy`. The probe after a deploy and marks the deploy failed unless it is `healthy`. The probe
@@ -288,12 +316,32 @@ once, is refused with 400.
both optional (values under Signature Specification). Both are part of what is both optional (values under Signature Specification). Both are part of what is
cached, so each value of either is a separate cached image. cached, so each value of either is a separate cached image.
A source URL's query follows its path after `%3F`, the encoded `?`:
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
percent-escapes in the image URL's path once, then reads the host up to the
first `/`, the size and format after the last `/`, and between them the source
path up to the first `?` and the source query after it. So the `?` before the
query, any `?` in the query, and every `%` in the source path or query must be
percent-encoded, as `%3F` and `%25`. Any other character may be written as
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
example,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 WebP is
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
signed as shown under Signature Specification.
A size with one side `0` has that side scaled to keep the source's proportions, A size with one side `0` has that side scaled to keep the source's proportions,
and is refused with 400 when that side would come out above 8192, the most and is refused with 400 when that side would come out above 8192, the most
either side may be; so is a size with `fit=outside`, which scales the image to either side may be; so is a size with `fit=outside`, which scales the image to
cover the size and so past it on one side, when that side would come out cover the size and so past it on one side, when that side would come out
above 8192. above 8192.
An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
have, is refused with 400 before anything is decoded, as its encode could take
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
not `image/jxl` is refused such a size.
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
`Content-Type` must name its format, and the image's first bytes must match it. `Content-Type` must name its format, and the image's first bytes must match it.
An upstream answering with any other `Content-Type`, SVG included, is refused An upstream answering with any other `Content-Type`, SVG included, is refused
@@ -343,14 +391,18 @@ transcode that stops this way is answered 503 with the error
ended. govips gives pixa no way to stop libvips once an encode has begun, so an ended. govips gives pixa no way to stop libvips once an encode has begun, so an
encode keeps its slot until it finishes, which can be after encode keeps its slot until it finishes, which can be after
`downstream_timeout`. The limit of 8192 on each side of the source and of the `downstream_timeout`. The limit of 8192 on each side of the source and of the
output bounds how long that takes. With one libvips thread, as pixad runs it, output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG, With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16 takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are 16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28 encoders run below libvips' default effort, and AVIF at the lowest effort govips
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or can set, so that no encode takes more than 40 seconds of the default
JPEG XL encode can therefore hold its slot past the default `downstream_timeout` `downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
of 60 seconds. 8192x8192 random pixels with an alpha channel take up to about 23 seconds as
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
up to about 11 seconds.
The login form (`POST /`) is limited to 5 attempts per minute per client The login form (`POST /`) is limited to 5 attempts per minute per client
address, counting an IPv6 client by its /64; an attempt over the limit is address, counting an IPv6 client by its /64; an attempt over the limit is
@@ -402,7 +454,8 @@ nor change what it asks for.
sent with an empty format, or none, also makes a JPEG XL URL. Width and sent with an empty format, or none, also makes a JPEG XL URL. Width and
height both empty or `0` keep the original size; if only one of them is empty height both empty or `0` keep the original size; if only one of them is empty
or `0`, that side is scaled to keep the image's proportions. A URL whose or `0`, that side is scaled to keep the image's proportions. A URL whose
scaled side would come out above 8192 answers 400. scaled side would come out above 8192 answers 400, and so does an AVIF URL of
more than 16,777,216 pixels (see Routes).
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when 3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on, and the URL starts it expires. `<host>` is the host the page was opened on, and the URL starts
with `http` instead while `debug` is on. The name after the token is ignored with `http` instead while `debug` is on. The name after the token is ignored
@@ -507,6 +560,17 @@ For the same image at quality 40 with fit `contain`, the input ends in
and the URL is and the URL is
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`. `/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
For a source URL with a query,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
800x600 WebP with the same key and expiration and the default quality and fit:
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
`%25` (see Routes):
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
**Allowlist patterns:** **Allowlist patterns:**
- **Exact match**: `cdn.example.com` — matches only that host - **Exact match**: `cdn.example.com` — matches only that host
@@ -566,7 +630,7 @@ tell. With no file, pixa uses the environment and the defaults.
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` | | `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` | | `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` | | `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs | | `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` | | `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` | | `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB | | `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
@@ -652,10 +716,12 @@ Key settings in more detail:
does, and `downstream_timeout` has not ended first, the request is answered does, and `downstream_timeout` has not ended first, the request is answered
503 with the error `server busy, try again later` 503 with the error `server busy, try again later`
- `max_concurrent_processing` — the most images decoded and encoded at once; - `max_concurrent_processing` — the most images decoded and encoded at once;
default the number of CPUs pixa can use (`GOMAXPROCS`), which follows a default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's
container's CPU limit. A request that finds all of them in use waits up to 10 CPU limit), rounded down, and at least 1. A value that is not a whole number
seconds for one to free up; if none does, and `downstream_timeout` has not of at least 1 aborts startup. A request that finds all of them in use waits up
ended first, it is answered 503 the same way to 10 seconds for one to free up; if none does, and `downstream_timeout` has
not ended first, it is answered 503 the same way. Each one can take a lot of
memory: see "Running under upaas"
- `image_requests_per_minute` — how many requests one client may send to the - `image_requests_per_minute` — how many requests one client may send to the
image routes per minute (see the limit on them under Routes); default `600`. image routes per minute (see the limit on them under Routes); default `600`.
`0` turns the limit off, and a negative or non-integer value aborts startup. `0` turns the limit off, and a negative or non-integer value aborts startup.
+38
View File
@@ -30,6 +30,44 @@ P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-10-15 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
an alpha channel took about 61 seconds to encode as WebP, 96 as AVIF and 81 as
JPEG XL, so with the default `downstream_timeout` of 60 seconds such an image
was never delivered, yet its encode held a processing slot to the end. WebP is
now saved at effort 2 and JPEG XL at effort 4, which take up to about 23 and
27 seconds. AVIF was already at the lowest effort govips can set, so an AVIF
output of more than 16,777,216 pixels is refused with 400 before anything is
decoded; 4096x4096, 8192x2048 or 2048x8192 takes up to about 34 seconds. Each
time is the slowest at any quality. Resizing with `fit=cover` already decodes
the input, so the output's dimensions are worked out before resizing.
`README.md` gives the limit and the new times.
- 2026-10-15 the README says how a source URL with a query is written into an
image URL, and the image route accepts it (closes #255): the query follows the
source path after `%3F`. pixa now reads the image URL's path decoded once, so
the `?` before the query, any `?` in it and every `%` in the source path or
query must be percent-encoded, and any other character may be. The route used
to parse chi's wildcard, which holds the path as the client wrote it whenever
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
writes Go's own encoding of the path. The README has an example with a query
and its signature. The path traversal check no longer reads the source query,
and keeps a `%` that does not start a percent-escape as it is instead of
refusing the URL, so a source path or query holding `%25` is accepted, while a
double-encoded `..` next to such a `%` is still refused.
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
"Running under upaas" in `README.md` gives the worst-case memory, measured,
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
`/tmp` is a tmpfs.
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit - 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`, `e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`, `.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
+5 -4
View File
@@ -108,10 +108,11 @@ upstream_connections_per_host: 20
# downstream_timeout has ended first. # downstream_timeout has ended first.
upstream_connections: 64 upstream_connections: 64
# Maximum number of images decoded and encoded at once (default: the # Maximum number of images decoded and encoded at once (default: half the
# number of CPUs pixa can use, which follows a container's CPU limit). A # number of CPUs pixa can use, at least 1; the number of CPUs follows a
# request that finds none free waits up to 10 seconds for one, and if none # container's CPU limit). A request that finds none free waits up to 10
# frees up it is answered 503, unless downstream_timeout has ended first. # seconds for one, and if none frees up it is answered 503, unless
# downstream_timeout has ended first.
# max_concurrent_processing: 4 # max_concurrent_processing: 4
# How many requests to the image routes (/v1/image/ and /v1/e/) one client # How many requests to the image routes (/v1/image/ and /v1/e/) one client
@@ -1,6 +1,7 @@
package config package config
import ( import (
"fmt"
"runtime" "runtime"
"testing" "testing"
) )
@@ -12,8 +13,8 @@ const (
) )
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted // TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
// max_concurrent_processing is the number of CPUs Go uses and an omitted // max_concurrent_processing is half the CPUs Go uses, and at least 1, and
// upstream_connections is 64. // an omitted upstream_connections is 64.
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) { func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Parallel() t.Parallel()
@@ -22,9 +23,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
t.Fatalf("minimal config should be valid, got error: %v", err) t.Fatalf("minimal config should be valid, got error: %v", err)
} }
if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) { if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) {
t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU", t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs",
c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0)) c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1))
} }
if c.UpstreamConnections != 64 { if c.UpstreamConnections != 64 {
@@ -32,6 +33,36 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
} }
} }
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
// that an omitted max_concurrent_processing is half the CPUs Go uses,
// rounded down, and 1 when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
tc.cpus, c.MaxConcurrentProcessing)
if c.MaxConcurrentProcessing != tc.want {
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
tc.cpus, c.MaxConcurrentProcessing, tc.want)
}
}
}
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the // TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
// two limits are used as given. // two limits are used as given.
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) { func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
@@ -160,3 +191,19 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
}) })
} }
} }
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
// least 1, an empty one included, aborts startup naming the variable and
// the key instead of taking the default.
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
t.Setenv(testMaxConcurrentProcessingVar, value)
_, err := configFromYAML(t, signingKeyLine)
wantStartupError(t, err,
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
})
}
}
+9 -3
View File
@@ -37,6 +37,14 @@ const (
DefaultImageRequestsPerMinute = 600 DefaultImageRequestsPerMinute = 600
) )
// DefaultMaxConcurrentProcessing returns the most images processed at once
// when max_concurrent_processing is omitted: half the CPUs Go uses
// (runtime.GOMAXPROCS(0), which follows a container's CPU limit), rounded
// down, and at least 1.
func DefaultMaxConcurrentProcessing() int {
return max(runtime.GOMAXPROCS(0)/2, 1) //nolint:mnd // half the CPUs
}
// Configuration key names. // Configuration key names.
const ( const (
keyDebug = "debug" keyDebug = "debug"
@@ -297,10 +305,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost), keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
UpstreamConnections: loader.intVal( UpstreamConnections: loader.intVal(
keyUpstreamConnections, DefaultUpstreamConnections), keyUpstreamConnections, DefaultUpstreamConnections),
// Decoding and encoding are CPU-bound, so the default is one image
// per CPU Go uses, which follows a container's CPU limit.
MaxConcurrentProcessing: loader.intVal( MaxConcurrentProcessing: loader.intVal(
keyMaxConcurrentProcessing, runtime.GOMAXPROCS(0)), keyMaxConcurrentProcessing, DefaultMaxConcurrentProcessing()),
UpstreamFetchTimeout: loader.durationVal( UpstreamFetchTimeout: loader.durationVal(
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout), keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
UpstreamMaxResponseSize: loader.int64Val( UpstreamMaxResponseSize: loader.int64Val(
@@ -0,0 +1,49 @@
package handlers
import (
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"testing"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
// output above it, wrapped as the service wraps it. The request's source only
// goes into a log line, so it is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("image processing failed: %w: 4097x4096",
imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("%s: body = %q, want it to say %q",
name, rec.Body.String(), wantBody)
}
}
}
+16 -8
View File
@@ -9,7 +9,6 @@ import (
"strconv" "strconv"
"time" "time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher" "sneak.berlin/go/pixa/internal/httpfetcher"
@@ -80,20 +79,21 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
} }
} }
// parseImageRequest parses the wildcard path and query parameters into // parseImageRequest parses the image URL's path and query parameters into
// an ImageRequest. On invalid input it writes an error response and // an ImageRequest. On invalid input it writes an error response and
// returns false. // returns false.
func (s *Handlers) parseImageRequest( func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request, w http.ResponseWriter, r *http.Request,
) (*imgcache.ImageRequest, bool) { ) (*imgcache.ImageRequest, bool) {
// Get the wildcard path from chi // The path is parsed decoded once, as r.URL.Path holds it, and not from
pathParam := chi.URLParam(r, "*") // chi's wildcard: chi routes on the path as the client wrote it whenever
// that is not Go's own encoding of it, so its wildcard would still hold
// Parse the URL path // the %3F before a source query when, say, a "/" in the query is written
parsed, err := imgcache.ParseImagePath(pathParam) // as %2F
parsed, err := imgcache.ParseImageURL(r.URL.Path)
if err != nil { if err != nil {
s.log.Warn("failed to parse image URL", s.log.Warn("failed to parse image URL",
"path", pathParam, "path", r.URL.Path,
"error", err, "error", err,
) )
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest) s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
@@ -264,6 +264,14 @@ func (s *Handlers) respondImageError(
return return
} }
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError) s.respondError(w, "internal error", http.StatusInternalServerError)
} }
@@ -169,3 +169,166 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
}) })
} }
} }
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
// by the service's signed-URL generator for source URLs with a query to the
// route, through a chi router: chi routes on the path as the client wrote it
// whenever that is not Go's own encoding of it. A 502 means the signature
// passed, so the route read the source path and query that were signed.
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
query string
}{
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
{"semicolon", "a=1;b=2"},
{"comma", "crop=0,0,50,50"},
{"equals and ampersand", "a=1&b=2"},
{"percent-escape", "name=my%20cat"},
{"question mark", "next=/a?b=1"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: tt.query,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
// and writes it into the image URL in ways the README allows: the "?" before
// the query and each "%" in it percent-encoded, and every other character as
// itself or percent-encoded. A 502 means the signature passed.
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath,
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
tests := []struct {
name string
writtenQuery string
}{
{"only ? and % encoded",
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
{"every reserved character encoded",
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
{"lower-case hex digits",
"%3furl=https://origin.example.com/my%2520cat.jpg"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
// source query that each hold %25, the percent-escape of a literal "%", and
// sends each to the route written as the README says, with that "%" written as
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
t.Parallel()
tests := []struct {
name string
sourcePath string
sourceQuery string
// writtenPathAndQuery is the source path and query as the README says
// to write them into the image URL
writtenPathAndQuery string
}{
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
}
for _, tt := range tests {
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
t.Parallel()
expires := time.Now().Add(time.Hour)
signer := signature.New("test-signing-key-must-be-32-chars")
sig := signer.Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: expires,
})
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
status := getImage(t, setupTestHandler(t), target)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
t.Parallel()
fix := setupTestHandler(t)
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
SourceHost: signedHost,
SourcePath: tt.sourcePath,
SourceQuery: tt.sourceQuery,
Size: imgcache.Size{Width: 50, Height: 50},
Format: imgcache.FormatJPEG,
}, time.Hour)
if err != nil {
t.Fatalf("GenerateSignedURL() error = %v", err)
}
status := getImage(t, fix, signedURL)
if status != http.StatusBadGateway {
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
}
})
}
}
+4
View File
@@ -170,6 +170,10 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+ s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d", "computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest) imageprocessor.MaxOutputDimension), http.StatusBadRequest)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
default: default:
s.log.Error("image request failed", "error", err) s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError) s.respondError(w, "internal error", http.StatusInternalServerError)
@@ -0,0 +1,64 @@
package imageprocessor
import (
"bytes"
"errors"
"testing"
)
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
// before anything is decoded, whatever size and fit give it, and that an AVIF
// output of at most that many pixels, or one in another format, is not. The
// input is a 64x32 PNG cut off in its pixel data: libvips reads its size from
// its header but fails when it decodes it, so a request that gets that failure
// was not refused, and a refusal came before any decoding.
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
t.Parallel()
whole := createTestPNG(t, 64, 32)
cutOff := whole[:len(whole)/2]
tests := []struct {
name string
format Format
width, height int
fit FitMode
refused bool
}{
{"AVIF at the limit", FormatAVIF, 4096, 4096, FitCover, false},
{"AVIF above the limit", FormatAVIF, 4096, 4097, FitCover, true},
{"WebP above the AVIF limit", FormatWebP, 4096, 4097, FitCover, false},
// FitInside does not enlarge, so the output stays 64x32
{"AVIF inside a larger size", FormatAVIF, 8192, 8192, FitInside, false},
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
{"AVIF outside a smaller size", FormatAVIF, 8192, 1, FitOutside, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(cutOff),
&Request{
Size: Size{Width: tt.width, Height: tt.height},
Format: tt.format,
FitMode: tt.fit,
})
t.Logf("Process() error = %v", err)
if tt.refused {
if !errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
}
return
}
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
t.Errorf("Process() error = %v, want the failure to decode "+
"the cut-off input", err)
}
})
}
}
@@ -77,10 +77,10 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
} }
} }
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at // TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
// libvips' default effort, 4: the output is the size of the same image saved // not libvips' default of 4: the output is the size of the same image saved
// at that effort. // at effort 2.
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) { func TestImageProcessor_WebPAtEffort2(t *testing.T) {
t.Parallel() t.Parallel()
input := createTestJPEG(t, 200, 150) input := createTestJPEG(t, 200, 150)
@@ -90,14 +90,14 @@ func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{ want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
StripMetadata: true, StripMetadata: true,
Quality: 85, Quality: 85,
ReductionEffort: 4, ReductionEffort: 2,
}) })
if err != nil { if err != nil {
t.Fatalf("ExportWebp() error = %v", err) t.Fatalf("ExportWebp() error = %v", err)
} }
if size != int64(len(want)) { if size != int64(len(want)) {
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4", t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
size, len(want)) size, len(want))
} }
} }
@@ -143,3 +143,29 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
"and 8 bits", size, len(want)) "and 8 bits", size, len(want))
} }
} }
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
// effort 4, not libvips' default of 7: the output is the size of the same
// image saved at effort 4, with the distance exportJXL gives quality 85.
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, 200, 150)
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
params := vips.NewJxlExportParams()
params.Quality = 85
params.Distance = jxlDistance(85)
params.Effort = 4
want, _, err := decode(t, input).ExportJxl(params)
if err != nil {
t.Fatalf("ExportJxl() error = %v", err)
}
if size != int64(len(want)) {
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
size, len(want))
}
}
+147 -83
View File
@@ -7,11 +7,11 @@ import (
"errors" "errors"
"fmt" "fmt"
"io" "io"
"runtime"
"sync" "sync"
"time" "time"
"github.com/davidbyttow/govips/v2/vips" "github.com/davidbyttow/govips/v2/vips"
"sneak.berlin/go/pixa/internal/config"
) )
// vipsOnce ensures vips is initialized exactly once. // vipsOnce ensures vips is initialized exactly once.
@@ -20,11 +20,10 @@ import (
var vipsOnce sync.Once var vipsOnce sync.Once
// initVips initializes libvips with quiet logging, one worker thread per // initVips initializes libvips with quiet logging, one worker thread per
// image and no operation cache. Process already works on one image per CPU // image and no operation cache. Each request decodes different source
// by default, so more threads per image would only compete for the CPUs. // bytes, so the operation cache would rarely be hit and would hold memory
// Each request decodes different source bytes, so the operation cache // outside MaxConcurrentProcessing; repeated requests are served from
// would rarely be hit and would hold memory outside MaxConcurrentProcessing; // pixa's disk cache instead.
// repeated requests are served from pixa's disk cache instead.
func initVips() { func initVips() {
vipsOnce.Do(func() { vipsOnce.Do(func() {
vips.LoggingSettings(nil, vips.LogLevelError) vips.LoggingSettings(nil, vips.LogLevelError)
@@ -143,6 +142,19 @@ const MaxOutputDimension = 8192
// requested size. // requested size.
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum") var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
// encode takes at most 40 seconds of the default downstream_timeout of 60
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
// 2048x8192 random pixels with an alpha channel takes up to about 34
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
// 5120x4096 at 8 bits already takes about 36.
const MaxAVIFOutputPixels = 4096 * 4096
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
// output would have more than MaxAVIFOutputPixels pixels.
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
// ErrUnsupportedOutputFormat is returned when the requested output format is // ErrUnsupportedOutputFormat is returned when the requested output format is
// not supported. // not supported.
var ErrUnsupportedOutputFormat = errors.New("unsupported output format") var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
@@ -179,7 +191,7 @@ type Params struct {
// If <= 0, DefaultMaxInputBytes is used. // If <= 0, DefaultMaxInputBytes is used.
MaxInputBytes int64 MaxInputBytes int64
// MaxConcurrentProcessing is the most images processed at once. // MaxConcurrentProcessing is the most images processed at once.
// If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used. // If <= 0, config.DefaultMaxConcurrentProcessing() is used.
MaxConcurrentProcessing int MaxConcurrentProcessing int
} }
@@ -195,7 +207,7 @@ func New(params Params) *ImageProcessor {
maxConcurrentProcessing := params.MaxConcurrentProcessing maxConcurrentProcessing := params.MaxConcurrentProcessing
if maxConcurrentProcessing <= 0 { if maxConcurrentProcessing <= 0 {
maxConcurrentProcessing = runtime.GOMAXPROCS(0) maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing()
} }
return &ImageProcessor{ return &ImageProcessor{
@@ -211,7 +223,8 @@ func New(params Params) *ImageProcessor {
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it // ErrTooManyImages. When ctx has ended once Process has a slot, or before it
// encodes, it returns ctx's error and gives the slot back. govips gives pixa // encodes, it returns ctx's error and gives the slot back. govips gives pixa
// no way to stop libvips once an encode has begun, so an encode keeps its // no way to stop libvips once an encode has begun, so an encode keeps its
// slot until it finishes. // slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
// refused with ErrAVIFOutputTooLarge before anything is decoded.
func (p *ImageProcessor) Process( func (p *ImageProcessor) Process(
ctx context.Context, ctx context.Context,
input io.Reader, input io.Reader,
@@ -224,18 +237,9 @@ func (p *ImageProcessor) Process(
defer release() defer release()
// Read input with a size limit to prevent unbounded memory consumption. data, err := p.readInput(input)
// We read at most maxInputBytes+1 so we can detect if the input exceeds
// the limit without consuming additional memory.
limited := io.LimitReader(input, p.maxInputBytes+1)
data, err := io.ReadAll(limited)
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to read input: %w", err) return nil, err
}
if int64(len(data)) > p.maxInputBytes {
return nil, ErrInputDataTooLarge
} }
// govips picks the libvips loader from the input's first bytes with // govips picks the libvips loader from the input's first bytes with
@@ -278,20 +282,28 @@ func (p *ImageProcessor) Process(
return nil, err return nil, err
} }
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
// orig is the source's own format; encode refuses an empty format // orig is the source's own format; encode refuses an empty format
outputFormat := req.Format outputFormat := req.Format
if outputFormat == FormatOriginal { if outputFormat == FormatOriginal {
outputFormat = sourceFormat outputFormat = sourceFormat
} }
// Determine output dimensions; too large ones are refused before
// resizing, which with FitCover already decodes the input
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
targetWidth, targetHeight, req.FitMode, outputFormat)
if err != nil {
return nil, err
}
// Resize if needed
if targetWidth != origWidth || targetHeight != origHeight {
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
if err != nil {
return nil, fmt.Errorf("failed to resize: %w", err)
}
}
// Encode to target format // Encode to target format
output, err := p.encode(ctx, img, outputFormat, req.Quality) output, err := p.encode(ctx, img, outputFormat, req.Quality)
if err != nil { if err != nil {
@@ -444,6 +456,23 @@ func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
return release, nil return release, nil
} }
// readInput reads input with a size limit to prevent unbounded memory
// consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
// reads at most maxInputBytes+1 so it can detect if the input exceeds the
// limit without consuming additional memory.
func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
if err != nil {
return nil, fmt.Errorf("failed to read input: %w", err)
}
if int64(len(data)) > p.maxInputBytes {
return nil, ErrInputDataTooLarge
}
return data, nil
}
// detectFormat returns the format string for a govips image type. // detectFormat returns the format string for a govips image type.
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string { func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
switch imageType { switch imageType {
@@ -468,9 +497,61 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
} }
} }
// resize resizes the image according to the fit mode. With FitOutside one // outputDimensions returns the dimensions resize gives an image of imgWidth x
// side can come out larger than requested; when it would be above // imgHeight for the target dimensions and fit mode, without resizing it, and
// MaxOutputDimension, resize returns ErrOutputTooLarge without resizing. // refuses an output too large for format. With FitOutside one side can come
// out larger than the target; when it would be above MaxOutputDimension,
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
func outputDimensions(
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
) (int, int, error) {
scaleW := float64(width) / float64(imgWidth)
scaleH := float64(height) / float64(imgHeight)
outputWidth, outputHeight := width, height
switch fit {
case FitCover, FitFill, "":
// Cover crops, and fill stretches, to exactly the target dimensions
case FitContain:
// Fit within the target dimensions, maintaining aspect ratio
scale := min(scaleW, scaleH)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
case FitInside:
// Same as contain, but only shrink
scale := min(scaleW, scaleH, 1)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
case FitOutside:
// The smallest dimension fits, the other may exceed the target
scale := max(scaleW, scaleH)
outputWidth = int(float64(imgWidth) * scale)
outputHeight = int(float64(imgHeight) * scale)
default:
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
}
// The side FitOutside scales past the target is held to the maximum too
if max(outputWidth, outputHeight) > MaxOutputDimension {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
outputWidth, outputHeight, MaxOutputDimension)
}
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
}
return outputWidth, outputHeight, nil
}
// resize resizes the image to width x height, the dimensions
// outputDimensions gives for the fit mode.
func (p *ImageProcessor) resize( func (p *ImageProcessor) resize(
img *vips.ImageRef, width, height int, fit FitMode, img *vips.ImageRef, width, height int, fit FitMode,
) error { ) error {
@@ -479,52 +560,20 @@ func (p *ImageProcessor) resize(
// Resize and crop to fill exact dimensions (default) // Resize and crop to fill exact dimensions (default)
return img.Thumbnail(width, height, vips.InterestingCentre) return img.Thumbnail(width, height, vips.InterestingCentre)
case FitContain:
// Resize to fit within dimensions, maintaining aspect ratio
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone)
case FitFill: case FitFill:
// Resize to exact dimensions (may distort) // Resize to exact dimensions (may distort)
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce) return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
case FitInside: case FitInside:
// Same as contain, but only shrink // outputDimensions keeps the dimensions of an image that already fits
if img.Width() <= width && img.Height() <= height { if img.Width() == width && img.Height() == height {
return nil // Already fits return nil
} }
imgW, imgH := img.Width(), img.Height() return img.Thumbnail(width, height, vips.InterestingNone)
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := min(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
return img.Thumbnail(newW, newH, vips.InterestingNone) case FitContain, FitOutside:
return img.Thumbnail(width, height, vips.InterestingNone)
case FitOutside:
// Resize so smallest dimension fits, may exceed target on other dimension
imgW, imgH := img.Width(), img.Height()
scaleW := float64(width) / float64(imgW)
scaleH := float64(height) / float64(imgH)
scale := max(scaleW, scaleH)
newW := int(float64(imgW) * scale)
newH := int(float64(imgH) * scale)
// The dimension that exceeds the target is held to the maximum too
if newW > MaxOutputDimension || newH > MaxOutputDimension {
return fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
newW, newH, MaxOutputDimension)
}
return img.Thumbnail(newW, newH, vips.InterestingNone)
default: default:
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit) return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
@@ -628,11 +677,16 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
return output, err return output, err
} }
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6. // webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
const webpEffort = 4 // default of 4 so that an encode takes at most 40 seconds of the default
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
// saving an 8192x8192 image of random pixels with an alpha channel takes up
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
const webpEffort = 2
// exportWebP encodes img as lossy WebP at quality and libvips' default // exportWebP encodes img as lossy WebP at quality and webpEffort, without
// effort, without metadata. // metadata.
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) { func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
output, _, err := img.ExportWebp(&vips.WebpExportParams{ output, _, err := img.ExportWebp(&vips.WebpExportParams{
StripMetadata: true, StripMetadata: true,
@@ -644,12 +698,13 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
} }
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest // avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51 // govips can set. With one thread, as pixad runs libvips, 1 takes about 52
// seconds to save an 8192x8192 RGB image of random pixels, and about 110 // seconds to save an 8192x8192 RGB image of random pixels at quality 85,
// with an alpha channel, the worst case, against the default // and about 96 with an alpha channel, against the default
// downstream_timeout of 60 seconds. On an image of milder noise, which 1 // downstream_timeout of 60 seconds, so an AVIF output is held to
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default, // MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
// 4, takes minutes. // 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
// minutes.
const avifEffort = 1 const avifEffort = 1
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image. // avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
@@ -676,10 +731,18 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
// JPEG that names none. // JPEG that names none.
const jxlResolution = 72 / 25.4 const jxlResolution = 72 / 25.4
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort // jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
// and without metadata. govips sends libvips a distance, the JPEG XL // libvips' default of 7 so that an encode takes at most 40 seconds of the
// encoder's own measure of quality, along with the quality, and libvips then // default downstream_timeout of 60 seconds. With one thread, as pixad runs
// uses the distance alone, so the quality is also given as a distance. // libvips, saving an 8192x8192 image of random pixels with an alpha channel
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
const jxlEffort = 4
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
// measure of quality, along with the quality, and libvips then uses the
// distance alone, so the quality is also given as a distance.
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) { func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or // libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
// PNG, but cannot save one as JPEG XL. encode has already converted // PNG, but cannot save one as JPEG XL. encode has already converted
@@ -718,6 +781,7 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
params := vips.NewJxlExportParams() params := vips.NewJxlExportParams()
params.Quality = quality params.Quality = quality
params.Distance = jxlDistance(quality) params.Distance = jxlDistance(quality)
params.Effort = jxlEffort
output, _, err := toSave.ExportJxl(params) output, _, err := toSave.ExportJxl(params)
if err != nil { if err != nil {
@@ -115,14 +115,14 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
} }
} }
func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) { func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
t.Parallel() t.Parallel()
for _, limit := range []int{0, -1} { for _, limit := range []int{0, -1} {
proc := New(Params{MaxConcurrentProcessing: limit}) proc := New(Params{MaxConcurrentProcessing: limit})
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) { if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) {
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU", t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs",
limit, got, runtime.GOMAXPROCS(0)) limit, got, max(runtime.GOMAXPROCS(0)/2, 1))
} }
} }
@@ -132,6 +132,30 @@ func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
} }
} }
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
// rounded down, and one slot when Go uses one CPU.
//
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
cpusBefore := runtime.GOMAXPROCS(0)
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
for _, tc := range []struct{ cpus, want int }{
{1, 1}, {2, 1}, {3, 1}, {8, 4},
} {
runtime.GOMAXPROCS(tc.cpus)
got := cap(New(Params{}).processingSemaphore)
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
if got != tc.want {
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
}
}
}
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls // TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
// than MaxConcurrentProcessing allows and holds the first ones inside // than MaxConcurrentProcessing allows and holds the first ones inside
// Process until the test lets them go. No more than the limit may be // Process until the test lets them go. No more than the limit may be
+1 -1
View File
@@ -53,7 +53,7 @@ type ServiceConfig struct {
// Allowlist is the list of hosts that don't require signatures // Allowlist is the list of hosts that don't require signatures
Allowlist []string Allowlist []string
// MaxConcurrentProcessing is the most images processed at once; zero // MaxConcurrentProcessing is the most images processed at once; zero
// uses the image processor's default, one per CPU // uses the image processor's default, half the CPUs
MaxConcurrentProcessing int MaxConcurrentProcessing int
// Logger for logging // Logger for logging
Logger *slog.Logger Logger *slog.Logger
+29 -18
View File
@@ -47,6 +47,9 @@ func ValidateDimension(name string, value int) error {
var sizeFormatRegex = regexp.MustCompile( var sizeFormatRegex = regexp.MustCompile(
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`) `^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
// percentEscapeRegex matches one percent-escape, such as "%2e"
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
// ParsedURL contains the parsed components of an image proxy URL. // ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct { type ParsedURL struct {
// Host is the source origin host (e.g., "cdn.example.com") // Host is the source origin host (e.g., "cdn.example.com")
@@ -61,9 +64,8 @@ type ParsedURL struct {
Format ImageFormat Format ImageFormat
} }
// ParseImagePath parses the path captured by chi's wildcard: // ParseImagePath parses an image URL's path after /v1/image/, decoded once:
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL // <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
// This is the primary entry point when using chi routing.
// Examples: // Examples:
// - cdn.example.com/photos/cat.jpg/800x600.webp // - cdn.example.com/photos/cat.jpg/800x600.webp
// - cdn.example.com/photos/cat.jpg/0x0.jpeg // - cdn.example.com/photos/cat.jpg/0x0.jpeg
@@ -79,9 +81,10 @@ func ParseImagePath(path string) (*ParsedURL, error) {
return parseImageComponents(path) return parseImageComponents(path)
} }
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>, // ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
// or /v1/image/<host>/<path>/<size> for JPEG XL // it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
// Use ParseImagePath instead when working with chi's wildcard capture. // for JPEG XL. A source query follows the source path after a "?", which the
// URL writes as %3F.
func ParseImageURL(urlPath string) (*ParsedURL, error) { func ParseImageURL(urlPath string) (*ParsedURL, error) {
// Remove the /v1/image/ prefix // Remove the /v1/image/ prefix
const prefix = "/v1/image/" const prefix = "/v1/image/"
@@ -100,12 +103,6 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
// parseImageComponents parses <host>/<path>/<size>.<format>, or // parseImageComponents parses <host>/<path>/<size>.<format>, or
// <host>/<path>/<size> for JPEG XL. // <host>/<path>/<size> for JPEG XL.
func parseImageComponents(remainder string) (*ParsedURL, error) { func parseImageComponents(remainder string) (*ParsedURL, error) {
// Check for path traversal before any other processing
err := checkPathTraversal(remainder)
if err != nil {
return nil, err
}
// Find the last path segment, which holds "size" or "size.format" // Find the last path segment, which holds "size" or "size.format"
lastSlash := strings.LastIndex(remainder, "/") lastSlash := strings.LastIndex(remainder, "/")
if lastSlash == -1 { if lastSlash == -1 {
@@ -161,6 +158,13 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
path = path[:qIndex] path = path[:qIndex]
} }
// The source query names no file on the upstream, so only the host and
// the source path are checked for path traversal
err = checkPathTraversal(host + path)
if err != nil {
return nil, err
}
// Ensure path starts with / // Ensure path starts with /
if !strings.HasPrefix(path, "/") { if !strings.HasPrefix(path, "/") {
path = "/" + path path = "/" + path
@@ -186,12 +190,7 @@ func checkPathTraversal(path string) error {
// Decode multiple times to catch double-encoding // Decode multiple times to catch double-encoding
decoded := path decoded := path
for range 3 { for range 3 {
newDecoded, err := url.PathUnescape(decoded) newDecoded := decodePercentEscapes(decoded)
if err != nil {
// Malformed encoding is suspicious
return ErrPathTraversal
}
if newDecoded == decoded { if newDecoded == decoded {
break break
} }
@@ -232,6 +231,18 @@ func checkPathTraversal(path string) error {
return nil return nil
} }
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
// keeps a "%" that does not start one as it is, where url.PathUnescape would
// refuse the whole string. A source path holding %25, a literal "%", has such
// a "%" after one more decoding.
func decodePercentEscapes(s string) string {
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
decoded, _ := url.PathUnescape(escape)
return decoded
})
}
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600" // parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
func parseSizeFormat(s string) (Size, ImageFormat, error) { func parseSizeFormat(s string) (Size, ImageFormat, error) {
matches := sizeFormatRegex.FindStringSubmatch(s) matches := sizeFormatRegex.FindStringSubmatch(s)
@@ -407,6 +407,71 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
} }
} }
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
// a source query holding %25, the percent-escape of a literal "%", are
// accepted as they are, although decoding them once more leaves a "%" that
// does not start a percent-escape.
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
t.Parallel()
tests := []struct {
name string
input string
want *ParsedURL
}{
{
name: "path",
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: "/sale/50%25off.jpg",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
{
name: "query",
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
want: &ParsedURL{
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
got, err := ParseImageURL(tt.input)
if err != nil {
t.Fatalf("ParseImageURL() unexpected error = %v", err)
}
assertParsedURL(t, got, tt.want)
})
}
}
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
// double-encoded ".." is still refused when the path also holds a "%" that
// does not start a percent-escape, either as written or after one more
// decoding.
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
t.Parallel()
for _, input := range []string{
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
} {
_, err := ParseImageURL(input)
t.Logf("ParseImageURL(%q) error = %v", input, err)
if !errors.Is(err, ErrPathTraversal) {
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
input, err)
}
}
}
// errorIs checks if err matches target (handles wrapped errors). // errorIs checks if err matches target (handles wrapped errors).
func errorIs(err, target error) bool { func errorIs(err, target error) bool {
if errors.Is(err, target) { if errors.Is(err, target) {
+11 -22
View File
@@ -126,31 +126,20 @@ func (s *Signer) GenerateSignedURL(
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height) sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
} }
// Build the path. // Build the path. A source query follows the source path after a "?".
// When a source query is present, it is embedded as a path segment // The image route reads the path decoded once, so the whole path is
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract // written in Go's own percent-encoding of a URL path, which writes "?"
// it from the last-slash split. The "?" inside a path segment is // as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
// percent-encoded by clients but chi delivers it decoded, which is // are.
// exactly what the URL parser expects. sourcePathAndQuery := req.SourcePath
var path string
if req.SourceQuery != "" { if req.SourceQuery != "" {
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s", sourcePathAndQuery += "?" + req.SourceQuery
req.SourceHost,
req.SourcePath,
url.PathEscape(req.SourceQuery),
sizeStr,
req.Format,
)
} else {
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost,
req.SourcePath,
sizeStr,
req.Format,
)
} }
return path, sig, exp unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
} }
// buildSignatureData creates the string to be signed. // buildSignatureData creates the string to be signed.
+38
View File
@@ -475,6 +475,44 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
} }
} }
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
// source URL with a query, the image at
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
// signature with the README's key, and the path GenerateSignedURL writes for
// it, with the "?" written as %3F and the "%" as %25.
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
t.Parallel()
signer := signature.New("example-signing-key-for-documentation")
req := signature.Request{
SourceHost: testHost,
SourcePath: "/img",
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
Width: 800,
Height: 600,
Format: testFormatWebP,
Quality: 85,
FitMode: testFitCover,
Expires: time.Unix(1704067200, 0),
}
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
gotSig := signer.Sign(&req)
if gotSig != wantSig {
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
}
const wantPath = "/v1/image/cdn.example.com" +
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
if gotPath != wantPath {
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
}
}
func TestParseParams(t *testing.T) { func TestParseParams(t *testing.T) {
t.Parallel() t.Parallel()