Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
47a2c2360a | ||
|
|
84a86274cd | ||
|
|
22bfc432b6 | ||
|
|
f743930f71 |
@@ -131,6 +131,34 @@ What the [upaas](https://git.eeqj.de/sneak/upaas) app for pixa needs:
|
|||||||
the cache can grow to three quarters or more of the space they all have
|
the cache can grow to three quarters or more of the space they all have
|
||||||
left, and to all of it when that is 500 MiB or less.
|
left, and to all of it when that is 500 MiB or less.
|
||||||
- the rest are in the table under Configuration below
|
- the rest are in the table under Configuration below
|
||||||
|
- **Memory:** pixa can use up to `max_concurrent_processing` times 4.2 GiB for
|
||||||
|
the images it processes at once, plus `upstream_connections` times
|
||||||
|
`upstream_max_response_size` for the source images it has fetched that wait
|
||||||
|
for processing. 4.2 GiB is the most resident memory was measured to grow while
|
||||||
|
pixa, with pixad's libvips settings, processed one 8192x8192 source, the
|
||||||
|
largest it accepts, saving it at full size to each output format in turn, one
|
||||||
|
at a time. The sources were a JPEG, a GIF, a WebP, an 8-bit and a 16-bit PNG,
|
||||||
|
an 8-bit and a 12-bit AVIF, and an 8-bit, a 16-bit and a 32-bit floating-point
|
||||||
|
JPEG XL; memory grew most for the 16-bit JPEG XL, of random pixels, saved as
|
||||||
|
WebP. At the defaults that is half the CPUs pixa sees times 4.2 GiB, plus 64
|
||||||
|
times 50 MiB: on a host where pixa sees 48 CPUs, 24 times 4.2 GiB plus 3.1
|
||||||
|
GiB, about 104 GiB. Set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||||
|
`PIXA_UPSTREAM_CONNECTIONS` so that this fits the memory the container may
|
||||||
|
use.
|
||||||
|
- **Temporary files:** while pixa processes a large source, libvips writes its
|
||||||
|
decoded pixels to a file in `/tmp`. The file disappears from `/tmp` once it is
|
||||||
|
written, so a listing of `/tmp` does not show it, but it keeps its space, on
|
||||||
|
disk or, where `/tmp` is a tmpfs, in memory, until the image is done. For a
|
||||||
|
GIF, WebP, AVIF or JPEG XL source the decoded pixels are held in memory as
|
||||||
|
well as written to that file, and the 4.2 GiB above includes that memory. For
|
||||||
|
an 8192x8192 source the file was measured on pixad at 192 MiB for an 8-bit RGB
|
||||||
|
JPEG, 512 MiB for a 16-bit RGBA PNG or JPEG XL, and 1 GiB for a JPEG XL of
|
||||||
|
32-bit floating-point RGBA, the largest measured: up to 1 GiB per image
|
||||||
|
processed at once, 24 GiB at the defaults on a host where pixa sees 48 CPUs.
|
||||||
|
This `/tmp` is in the container's own filesystem, not on the volume at
|
||||||
|
`/var/lib/pixa`, and `cache_max_bytes` does not count it. The 4.2 GiB above
|
||||||
|
does not include these files; where `/tmp` is a tmpfs, they are memory on top
|
||||||
|
of it.
|
||||||
- **Health check:** the image's `HEALTHCHECK` requests
|
- **Health check:** the image's `HEALTHCHECK` requests
|
||||||
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
`/.well-known/healthcheck.json`. upaas reads the container's health 60 seconds
|
||||||
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
after a deploy and marks the deploy failed unless it is `healthy`. The probe
|
||||||
@@ -288,12 +316,32 @@ once, is refused with 400.
|
|||||||
both optional (values under Signature Specification). Both are part of what is
|
both optional (values under Signature Specification). Both are part of what is
|
||||||
cached, so each value of either is a separate cached image.
|
cached, so each value of either is a separate cached image.
|
||||||
|
|
||||||
|
A source URL's query follows its path after `%3F`, the encoded `?`:
|
||||||
|
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
|
||||||
|
percent-escapes in the image URL's path once, then reads the host up to the
|
||||||
|
first `/`, the size and format after the last `/`, and between them the source
|
||||||
|
path up to the first `?` and the source query after it. So the `?` before the
|
||||||
|
query, any `?` in the query, and every `%` in the source path or query must be
|
||||||
|
percent-encoded, as `%3F` and `%25`. Any other character may be written as
|
||||||
|
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
|
||||||
|
example,
|
||||||
|
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
|
||||||
|
800x600 WebP is
|
||||||
|
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
|
||||||
|
signed as shown under Signature Specification.
|
||||||
|
|
||||||
A size with one side `0` has that side scaled to keep the source's proportions,
|
A size with one side `0` has that side scaled to keep the source's proportions,
|
||||||
and is refused with 400 when that side would come out above 8192, the most
|
and is refused with 400 when that side would come out above 8192, the most
|
||||||
either side may be; so is a size with `fit=outside`, which scales the image to
|
either side may be; so is a size with `fit=outside`, which scales the image to
|
||||||
cover the size and so past it on one side, when that side would come out
|
cover the size and so past it on one side, when that side would come out
|
||||||
above 8192.
|
above 8192.
|
||||||
|
|
||||||
|
An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
|
||||||
|
have, is refused with 400 before anything is decoded, as its encode could take
|
||||||
|
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
|
||||||
|
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
|
||||||
|
not `image/jxl` is refused such a size.
|
||||||
|
|
||||||
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
|
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
|
||||||
`Content-Type` must name its format, and the image's first bytes must match it.
|
`Content-Type` must name its format, and the image's first bytes must match it.
|
||||||
An upstream answering with any other `Content-Type`, SVG included, is refused
|
An upstream answering with any other `Content-Type`, SVG included, is refused
|
||||||
@@ -343,14 +391,18 @@ transcode that stops this way is answered 503 with the error
|
|||||||
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
|
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
|
||||||
encode keeps its slot until it finishes, which can be after
|
encode keeps its slot until it finishes, which can be after
|
||||||
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
||||||
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
|
||||||
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
|
||||||
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16
|
takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
|
||||||
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are
|
16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
|
||||||
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28
|
encoders run below libvips' default effort, and AVIF at the lowest effort govips
|
||||||
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or
|
can set, so that no encode takes more than 40 seconds of the default
|
||||||
JPEG XL encode can therefore hold its slot past the default `downstream_timeout`
|
`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
|
||||||
of 60 seconds.
|
8192x8192 random pixels with an alpha channel take up to about 23 seconds as
|
||||||
|
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
|
||||||
|
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
|
||||||
|
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
|
||||||
|
up to about 11 seconds.
|
||||||
|
|
||||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||||
@@ -402,7 +454,8 @@ nor change what it asks for.
|
|||||||
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
||||||
height both empty or `0` keep the original size; if only one of them is empty
|
height both empty or `0` keep the original size; if only one of them is empty
|
||||||
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
||||||
scaled side would come out above 8192 answers 400.
|
scaled side would come out above 8192 answers 400, and so does an AVIF URL of
|
||||||
|
more than 16,777,216 pixels (see Routes).
|
||||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||||
with `http` instead while `debug` is on. The name after the token is ignored
|
with `http` instead while `debug` is on. The name after the token is ignored
|
||||||
@@ -507,6 +560,17 @@ For the same image at quality 40 with fit `contain`, the input ends in
|
|||||||
and the URL is
|
and the URL is
|
||||||
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
||||||
|
|
||||||
|
For a source URL with a query,
|
||||||
|
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
|
||||||
|
800x600 WebP with the same key and expiration and the default quality and fit:
|
||||||
|
|
||||||
|
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
|
||||||
|
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
|
||||||
|
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
|
||||||
|
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
|
||||||
|
`%25` (see Routes):
|
||||||
|
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
|
||||||
|
|
||||||
**Allowlist patterns:**
|
**Allowlist patterns:**
|
||||||
|
|
||||||
- **Exact match**: `cdn.example.com` — matches only that host
|
- **Exact match**: `cdn.example.com` — matches only that host
|
||||||
@@ -566,7 +630,7 @@ tell. With no file, pixa uses the environment and the defaults.
|
|||||||
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
|
||||||
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
|
||||||
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
|
||||||
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs |
|
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default half the CPUs, at least 1 |
|
||||||
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
|
||||||
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
|
||||||
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
|
||||||
@@ -652,10 +716,12 @@ Key settings in more detail:
|
|||||||
does, and `downstream_timeout` has not ended first, the request is answered
|
does, and `downstream_timeout` has not ended first, the request is answered
|
||||||
503 with the error `server busy, try again later`
|
503 with the error `server busy, try again later`
|
||||||
- `max_concurrent_processing` — the most images decoded and encoded at once;
|
- `max_concurrent_processing` — the most images decoded and encoded at once;
|
||||||
default the number of CPUs pixa can use (`GOMAXPROCS`), which follows a
|
default half the CPUs pixa can use (`GOMAXPROCS`, which follows a container's
|
||||||
container's CPU limit. A request that finds all of them in use waits up to 10
|
CPU limit), rounded down, and at least 1. A value that is not a whole number
|
||||||
seconds for one to free up; if none does, and `downstream_timeout` has not
|
of at least 1 aborts startup. A request that finds all of them in use waits up
|
||||||
ended first, it is answered 503 the same way
|
to 10 seconds for one to free up; if none does, and `downstream_timeout` has
|
||||||
|
not ended first, it is answered 503 the same way. Each one can take a lot of
|
||||||
|
memory: see "Running under upaas"
|
||||||
- `image_requests_per_minute` — how many requests one client may send to the
|
- `image_requests_per_minute` — how many requests one client may send to the
|
||||||
image routes per minute (see the limit on them under Routes); default `600`.
|
image routes per minute (see the limit on them under Routes); default `600`.
|
||||||
`0` turns the limit off, and a negative or non-integer value aborts startup.
|
`0` turns the limit off, and a negative or non-integer value aborts startup.
|
||||||
|
|||||||
@@ -30,6 +30,44 @@ P2: security: per-origin rate limiting
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-10-15 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
|
||||||
|
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
|
||||||
|
an alpha channel took about 61 seconds to encode as WebP, 96 as AVIF and 81 as
|
||||||
|
JPEG XL, so with the default `downstream_timeout` of 60 seconds such an image
|
||||||
|
was never delivered, yet its encode held a processing slot to the end. WebP is
|
||||||
|
now saved at effort 2 and JPEG XL at effort 4, which take up to about 23 and
|
||||||
|
27 seconds. AVIF was already at the lowest effort govips can set, so an AVIF
|
||||||
|
output of more than 16,777,216 pixels is refused with 400 before anything is
|
||||||
|
decoded; 4096x4096, 8192x2048 or 2048x8192 takes up to about 34 seconds. Each
|
||||||
|
time is the slowest at any quality. Resizing with `fit=cover` already decodes
|
||||||
|
the input, so the output's dimensions are worked out before resizing.
|
||||||
|
`README.md` gives the limit and the new times.
|
||||||
|
- 2026-10-15 the README says how a source URL with a query is written into an
|
||||||
|
image URL, and the image route accepts it (closes #255): the query follows the
|
||||||
|
source path after `%3F`. pixa now reads the image URL's path decoded once, so
|
||||||
|
the `?` before the query, any `?` in it and every `%` in the source path or
|
||||||
|
query must be percent-encoded, and any other character may be. The route used
|
||||||
|
to parse chi's wildcard, which holds the path as the client wrote it whenever
|
||||||
|
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
|
||||||
|
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
|
||||||
|
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
|
||||||
|
writes Go's own encoding of the path. The README has an example with a query
|
||||||
|
and its signature. The path traversal check no longer reads the source query,
|
||||||
|
and keeps a `%` that does not start a percent-escape as it is instead of
|
||||||
|
refusing the URL, so a source path or query holding `%25` is accepted, while a
|
||||||
|
double-encoded `..` next to such a `%` is still refused.
|
||||||
|
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||||
|
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||||
|
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||||
|
host pixa processed 48 images at once, each of which can take about 4.2 GiB.
|
||||||
|
`config.DefaultMaxConcurrentProcessing` gives the default to the config and to
|
||||||
|
`imageprocessor.New`; `PIXA_MAX_CONCURRENT_PROCESSING` still overrides it.
|
||||||
|
"Running under upaas" in `README.md` gives the worst-case memory, measured,
|
||||||
|
and tells the operator to set `PIXA_MAX_CONCURRENT_PROCESSING` and
|
||||||
|
`PIXA_UPSTREAM_CONNECTIONS` to fit the container's memory. It also gives the
|
||||||
|
file libvips writes to `/tmp` for each large source, up to 1 GiB, measured,
|
||||||
|
which is outside `/var/lib/pixa` and `cache_max_bytes` and is memory where
|
||||||
|
`/tmp` is a tmpfs.
|
||||||
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
|
- 2026-10-15 the canonical files are re-vendored from `sneak/prompts` commit
|
||||||
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
|
`e3f79c4` (closes #221): `.dockerignore`, `.editorconfig`,
|
||||||
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
|
`.gitea/workflows/check.yml`, `.gitignore`, `.golangci.yml`,
|
||||||
|
|||||||
@@ -108,10 +108,11 @@ upstream_connections_per_host: 20
|
|||||||
# downstream_timeout has ended first.
|
# downstream_timeout has ended first.
|
||||||
upstream_connections: 64
|
upstream_connections: 64
|
||||||
|
|
||||||
# Maximum number of images decoded and encoded at once (default: the
|
# Maximum number of images decoded and encoded at once (default: half the
|
||||||
# number of CPUs pixa can use, which follows a container's CPU limit). A
|
# number of CPUs pixa can use, at least 1; the number of CPUs follows a
|
||||||
# request that finds none free waits up to 10 seconds for one, and if none
|
# container's CPU limit). A request that finds none free waits up to 10
|
||||||
# frees up it is answered 503, unless downstream_timeout has ended first.
|
# seconds for one, and if none frees up it is answered 503, unless
|
||||||
|
# downstream_timeout has ended first.
|
||||||
# max_concurrent_processing: 4
|
# max_concurrent_processing: 4
|
||||||
|
|
||||||
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
|
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package config
|
package config
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"fmt"
|
||||||
"runtime"
|
"runtime"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
@@ -12,8 +13,8 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
|
// TestOmittedConcurrencyLimitsUseDefaults checks that an omitted
|
||||||
// max_concurrent_processing is the number of CPUs Go uses and an omitted
|
// max_concurrent_processing is half the CPUs Go uses, and at least 1, and
|
||||||
// upstream_connections is 64.
|
// an omitted upstream_connections is 64.
|
||||||
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -22,9 +23,9 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
|||||||
t.Fatalf("minimal config should be valid, got error: %v", err)
|
t.Fatalf("minimal config should be valid, got error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.MaxConcurrentProcessing != runtime.GOMAXPROCS(0) {
|
if c.MaxConcurrentProcessing != max(runtime.GOMAXPROCS(0)/2, 1) {
|
||||||
t.Errorf("MaxConcurrentProcessing = %d, want %d, one per CPU",
|
t.Errorf("MaxConcurrentProcessing = %d, want %d, half the CPUs",
|
||||||
c.MaxConcurrentProcessing, runtime.GOMAXPROCS(0))
|
c.MaxConcurrentProcessing, max(runtime.GOMAXPROCS(0)/2, 1))
|
||||||
}
|
}
|
||||||
|
|
||||||
if c.UpstreamConnections != 64 {
|
if c.UpstreamConnections != 64 {
|
||||||
@@ -32,6 +33,36 @@ func TestOmittedConcurrencyLimitsUseDefaults(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne checks
|
||||||
|
// that an omitted max_concurrent_processing is half the CPUs Go uses,
|
||||||
|
// rounded down, and 1 when Go uses one CPU.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
|
||||||
|
func TestOmittedMaxConcurrentProcessingIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
|
||||||
|
cpusBefore := runtime.GOMAXPROCS(0)
|
||||||
|
|
||||||
|
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
|
||||||
|
|
||||||
|
for _, tc := range []struct{ cpus, want int }{
|
||||||
|
{1, 1}, {2, 1}, {3, 1}, {8, 4},
|
||||||
|
} {
|
||||||
|
runtime.GOMAXPROCS(tc.cpus)
|
||||||
|
|
||||||
|
c, err := configFromYAML(t, signingKeyLine)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("minimal config should be valid, got error: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Logf("GOMAXPROCS %d: MaxConcurrentProcessing %d",
|
||||||
|
tc.cpus, c.MaxConcurrentProcessing)
|
||||||
|
|
||||||
|
if c.MaxConcurrentProcessing != tc.want {
|
||||||
|
t.Errorf("GOMAXPROCS %d: MaxConcurrentProcessing = %d, want %d",
|
||||||
|
tc.cpus, c.MaxConcurrentProcessing, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
|
// TestExplicitConcurrencyLimitsAreUsed checks that valid values for the
|
||||||
// two limits are used as given.
|
// two limits are used as given.
|
||||||
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
|
func TestExplicitConcurrencyLimitsAreUsed(t *testing.T) {
|
||||||
@@ -160,3 +191,19 @@ func TestConcurrencyLimitsFromEnvironment(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup checks
|
||||||
|
// that a PIXA_MAX_CONCURRENT_PROCESSING that is not a whole number of at
|
||||||
|
// least 1, an empty one included, aborts startup naming the variable and
|
||||||
|
// the key instead of taking the default.
|
||||||
|
func TestInvalidMaxConcurrentProcessingFromEnvironmentAbortsStartup(t *testing.T) {
|
||||||
|
for _, value := range []string{"", " ", "0.5", "half", "-1", "4 images"} {
|
||||||
|
t.Run(fmt.Sprintf("value %q", value), func(t *testing.T) {
|
||||||
|
t.Setenv(testMaxConcurrentProcessingVar, value)
|
||||||
|
|
||||||
|
_, err := configFromYAML(t, signingKeyLine)
|
||||||
|
wantStartupError(t, err,
|
||||||
|
testMaxConcurrentProcessingVar, keyMaxConcurrentProcessing)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -37,6 +37,14 @@ const (
|
|||||||
DefaultImageRequestsPerMinute = 600
|
DefaultImageRequestsPerMinute = 600
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// DefaultMaxConcurrentProcessing returns the most images processed at once
|
||||||
|
// when max_concurrent_processing is omitted: half the CPUs Go uses
|
||||||
|
// (runtime.GOMAXPROCS(0), which follows a container's CPU limit), rounded
|
||||||
|
// down, and at least 1.
|
||||||
|
func DefaultMaxConcurrentProcessing() int {
|
||||||
|
return max(runtime.GOMAXPROCS(0)/2, 1) //nolint:mnd // half the CPUs
|
||||||
|
}
|
||||||
|
|
||||||
// Configuration key names.
|
// Configuration key names.
|
||||||
const (
|
const (
|
||||||
keyDebug = "debug"
|
keyDebug = "debug"
|
||||||
@@ -297,10 +305,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
|
|||||||
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
|
keyUpstreamConnectionsPerHost, DefaultUpstreamConnectionsPerHost),
|
||||||
UpstreamConnections: loader.intVal(
|
UpstreamConnections: loader.intVal(
|
||||||
keyUpstreamConnections, DefaultUpstreamConnections),
|
keyUpstreamConnections, DefaultUpstreamConnections),
|
||||||
// Decoding and encoding are CPU-bound, so the default is one image
|
|
||||||
// per CPU Go uses, which follows a container's CPU limit.
|
|
||||||
MaxConcurrentProcessing: loader.intVal(
|
MaxConcurrentProcessing: loader.intVal(
|
||||||
keyMaxConcurrentProcessing, runtime.GOMAXPROCS(0)),
|
keyMaxConcurrentProcessing, DefaultMaxConcurrentProcessing()),
|
||||||
UpstreamFetchTimeout: loader.durationVal(
|
UpstreamFetchTimeout: loader.durationVal(
|
||||||
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
|
keyUpstreamFetchTimeout, DefaultUpstreamFetchTimeout),
|
||||||
UpstreamMaxResponseSize: loader.int64Val(
|
UpstreamMaxResponseSize: loader.int64Val(
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
package handlers
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||||
|
"sneak.berlin/go/pixa/internal/imgcache"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
|
||||||
|
// handleImageError, which answer the errors of the two image routes, answer
|
||||||
|
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
|
||||||
|
// output above it, wrapped as the service wraps it. The request's source only
|
||||||
|
// goes into a log line, so it is left empty.
|
||||||
|
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||||
|
err := fmt.Errorf("image processing failed: %w: 4097x4096",
|
||||||
|
imageprocessor.ErrAVIFOutputTooLarge)
|
||||||
|
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||||
|
|
||||||
|
plain := httptest.NewRecorder()
|
||||||
|
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
||||||
|
|
||||||
|
encrypted := httptest.NewRecorder()
|
||||||
|
h.handleImageError(encrypted, err)
|
||||||
|
|
||||||
|
for name, rec := range map[string]*httptest.ResponseRecorder{
|
||||||
|
"respondImageError": plain, "handleImageError": encrypted,
|
||||||
|
} {
|
||||||
|
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
||||||
|
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("%s: status = %d, want %d",
|
||||||
|
name, rec.Code, http.StatusBadRequest)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(rec.Body.String(), wantBody) {
|
||||||
|
t.Errorf("%s: body = %q, want it to say %q",
|
||||||
|
name, rec.Body.String(), wantBody)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,7 +9,6 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/go-chi/chi/v5"
|
|
||||||
"github.com/go-chi/chi/v5/middleware"
|
"github.com/go-chi/chi/v5/middleware"
|
||||||
"sneak.berlin/go/pixa/internal/encurl"
|
"sneak.berlin/go/pixa/internal/encurl"
|
||||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||||
@@ -80,20 +79,21 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// parseImageRequest parses the wildcard path and query parameters into
|
// parseImageRequest parses the image URL's path and query parameters into
|
||||||
// an ImageRequest. On invalid input it writes an error response and
|
// an ImageRequest. On invalid input it writes an error response and
|
||||||
// returns false.
|
// returns false.
|
||||||
func (s *Handlers) parseImageRequest(
|
func (s *Handlers) parseImageRequest(
|
||||||
w http.ResponseWriter, r *http.Request,
|
w http.ResponseWriter, r *http.Request,
|
||||||
) (*imgcache.ImageRequest, bool) {
|
) (*imgcache.ImageRequest, bool) {
|
||||||
// Get the wildcard path from chi
|
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
||||||
pathParam := chi.URLParam(r, "*")
|
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
||||||
|
// that is not Go's own encoding of it, so its wildcard would still hold
|
||||||
// Parse the URL path
|
// the %3F before a source query when, say, a "/" in the query is written
|
||||||
parsed, err := imgcache.ParseImagePath(pathParam)
|
// as %2F
|
||||||
|
parsed, err := imgcache.ParseImageURL(r.URL.Path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.log.Warn("failed to parse image URL",
|
s.log.Warn("failed to parse image URL",
|
||||||
"path", pathParam,
|
"path", r.URL.Path,
|
||||||
"error", err,
|
"error", err,
|
||||||
)
|
)
|
||||||
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
||||||
@@ -264,6 +264,14 @@ func (s *Handlers) respondImageError(
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
|
||||||
|
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||||
|
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||||
|
http.StatusBadRequest)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -169,3 +169,166 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
|
||||||
|
// by the service's signed-URL generator for source URLs with a query to the
|
||||||
|
// route, through a chi router: chi routes on the path as the client wrote it
|
||||||
|
// whenever that is not Go's own encoding of it. A 502 means the signature
|
||||||
|
// passed, so the route read the source path and query that were signed.
|
||||||
|
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
query string
|
||||||
|
}{
|
||||||
|
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
|
||||||
|
{"semicolon", "a=1;b=2"},
|
||||||
|
{"comma", "crop=0,0,50,50"},
|
||||||
|
{"equals and ampersand", "a=1&b=2"},
|
||||||
|
{"percent-escape", "name=my%20cat"},
|
||||||
|
{"question mark", "next=/a?b=1"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fix := setupTestHandler(t)
|
||||||
|
|
||||||
|
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
SourceQuery: tt.query,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatJPEG,
|
||||||
|
}, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := getImage(t, fix, signedURL)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
|
||||||
|
// and writes it into the image URL in ways the README allows: the "?" before
|
||||||
|
// the query and each "%" in it percent-encoded, and every other character as
|
||||||
|
// itself or percent-encoded. A 502 means the signature passed.
|
||||||
|
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
expires := time.Now().Add(time.Hour)
|
||||||
|
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||||
|
sig := signer.Sign(&signature.Request{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: photoPath,
|
||||||
|
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: expires,
|
||||||
|
})
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
writtenQuery string
|
||||||
|
}{
|
||||||
|
{"only ? and % encoded",
|
||||||
|
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
|
||||||
|
{"every reserved character encoded",
|
||||||
|
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
|
||||||
|
{"lower-case hex digits",
|
||||||
|
"%3furl=https://origin.example.com/my%2520cat.jpg"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||||
|
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
|
||||||
|
|
||||||
|
status := getImage(t, setupTestHandler(t), target)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
|
||||||
|
// source query that each hold %25, the percent-escape of a literal "%", and
|
||||||
|
// sends each to the route written as the README says, with that "%" written as
|
||||||
|
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
|
||||||
|
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
sourcePath string
|
||||||
|
sourceQuery string
|
||||||
|
// writtenPathAndQuery is the source path and query as the README says
|
||||||
|
// to write them into the image URL
|
||||||
|
writtenPathAndQuery string
|
||||||
|
}{
|
||||||
|
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
|
||||||
|
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
expires := time.Now().Add(time.Hour)
|
||||||
|
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||||
|
sig := signer.Sign(&signature.Request{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: tt.sourcePath,
|
||||||
|
SourceQuery: tt.sourceQuery,
|
||||||
|
Width: 50,
|
||||||
|
Height: 50,
|
||||||
|
Format: string(imgcache.FormatJPEG),
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: string(imgcache.FitCover),
|
||||||
|
Expires: expires,
|
||||||
|
})
|
||||||
|
|
||||||
|
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||||
|
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
|
||||||
|
|
||||||
|
status := getImage(t, setupTestHandler(t), target)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fix := setupTestHandler(t)
|
||||||
|
|
||||||
|
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||||
|
SourceHost: signedHost,
|
||||||
|
SourcePath: tt.sourcePath,
|
||||||
|
SourceQuery: tt.sourceQuery,
|
||||||
|
Size: imgcache.Size{Width: 50, Height: 50},
|
||||||
|
Format: imgcache.FormatJPEG,
|
||||||
|
}, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
status := getImage(t, fix, signedURL)
|
||||||
|
if status != http.StatusBadGateway {
|
||||||
|
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -170,6 +170,10 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
|
|||||||
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
||||||
"computed from its proportions is above %d",
|
"computed from its proportions is above %d",
|
||||||
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
||||||
|
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
|
||||||
|
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||||
|
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||||
|
http.StatusBadRequest)
|
||||||
default:
|
default:
|
||||||
s.log.Error("image request failed", "error", err)
|
s.log.Error("image request failed", "error", err)
|
||||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
package imageprocessor
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
|
||||||
|
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
|
||||||
|
// before anything is decoded, whatever size and fit give it, and that an AVIF
|
||||||
|
// output of at most that many pixels, or one in another format, is not. The
|
||||||
|
// input is a 64x32 PNG cut off in its pixel data: libvips reads its size from
|
||||||
|
// its header but fails when it decodes it, so a request that gets that failure
|
||||||
|
// was not refused, and a refusal came before any decoding.
|
||||||
|
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
whole := createTestPNG(t, 64, 32)
|
||||||
|
cutOff := whole[:len(whole)/2]
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
format Format
|
||||||
|
width, height int
|
||||||
|
fit FitMode
|
||||||
|
refused bool
|
||||||
|
}{
|
||||||
|
{"AVIF at the limit", FormatAVIF, 4096, 4096, FitCover, false},
|
||||||
|
{"AVIF above the limit", FormatAVIF, 4096, 4097, FitCover, true},
|
||||||
|
{"WebP above the AVIF limit", FormatWebP, 4096, 4097, FitCover, false},
|
||||||
|
// FitInside does not enlarge, so the output stays 64x32
|
||||||
|
{"AVIF inside a larger size", FormatAVIF, 8192, 8192, FitInside, false},
|
||||||
|
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||||
|
{"AVIF outside a smaller size", FormatAVIF, 8192, 1, FitOutside, true},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(cutOff),
|
||||||
|
&Request{
|
||||||
|
Size: Size{Width: tt.width, Height: tt.height},
|
||||||
|
Format: tt.format,
|
||||||
|
FitMode: tt.fit,
|
||||||
|
})
|
||||||
|
t.Logf("Process() error = %v", err)
|
||||||
|
|
||||||
|
if tt.refused {
|
||||||
|
if !errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||||
|
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
|
||||||
|
}
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||||
|
t.Errorf("Process() error = %v, want the failure to decode "+
|
||||||
|
"the cut-off input", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -77,10 +77,10 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at
|
// TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
|
||||||
// libvips' default effort, 4: the output is the size of the same image saved
|
// not libvips' default of 4: the output is the size of the same image saved
|
||||||
// at that effort.
|
// at effort 2.
|
||||||
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
|
func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
input := createTestJPEG(t, 200, 150)
|
input := createTestJPEG(t, 200, 150)
|
||||||
@@ -90,14 +90,14 @@ func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
|
|||||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||||
StripMetadata: true,
|
StripMetadata: true,
|
||||||
Quality: 85,
|
Quality: 85,
|
||||||
ReductionEffort: 4,
|
ReductionEffort: 2,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("ExportWebp() error = %v", err)
|
t.Fatalf("ExportWebp() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
if size != int64(len(want)) {
|
if size != int64(len(want)) {
|
||||||
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4",
|
t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
|
||||||
size, len(want))
|
size, len(want))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -143,3 +143,29 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
|||||||
"and 8 bits", size, len(want))
|
"and 8 bits", size, len(want))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
|
||||||
|
// effort 4, not libvips' default of 7: the output is the size of the same
|
||||||
|
// image saved at effort 4, with the distance exportJXL gives quality 85.
|
||||||
|
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
input := createTestJPEG(t, 200, 150)
|
||||||
|
|
||||||
|
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
|
||||||
|
|
||||||
|
params := vips.NewJxlExportParams()
|
||||||
|
params.Quality = 85
|
||||||
|
params.Distance = jxlDistance(85)
|
||||||
|
params.Effort = 4
|
||||||
|
|
||||||
|
want, _, err := decode(t, input).ExportJxl(params)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ExportJxl() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if size != int64(len(want)) {
|
||||||
|
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
|
||||||
|
size, len(want))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -7,11 +7,11 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"runtime"
|
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/davidbyttow/govips/v2/vips"
|
"github.com/davidbyttow/govips/v2/vips"
|
||||||
|
"sneak.berlin/go/pixa/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
// vipsOnce ensures vips is initialized exactly once.
|
// vipsOnce ensures vips is initialized exactly once.
|
||||||
@@ -20,11 +20,10 @@ import (
|
|||||||
var vipsOnce sync.Once
|
var vipsOnce sync.Once
|
||||||
|
|
||||||
// initVips initializes libvips with quiet logging, one worker thread per
|
// initVips initializes libvips with quiet logging, one worker thread per
|
||||||
// image and no operation cache. Process already works on one image per CPU
|
// image and no operation cache. Each request decodes different source
|
||||||
// by default, so more threads per image would only compete for the CPUs.
|
// bytes, so the operation cache would rarely be hit and would hold memory
|
||||||
// Each request decodes different source bytes, so the operation cache
|
// outside MaxConcurrentProcessing; repeated requests are served from
|
||||||
// would rarely be hit and would hold memory outside MaxConcurrentProcessing;
|
// pixa's disk cache instead.
|
||||||
// repeated requests are served from pixa's disk cache instead.
|
|
||||||
func initVips() {
|
func initVips() {
|
||||||
vipsOnce.Do(func() {
|
vipsOnce.Do(func() {
|
||||||
vips.LoggingSettings(nil, vips.LogLevelError)
|
vips.LoggingSettings(nil, vips.LogLevelError)
|
||||||
@@ -143,6 +142,19 @@ const MaxOutputDimension = 8192
|
|||||||
// requested size.
|
// requested size.
|
||||||
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
|
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
|
||||||
|
|
||||||
|
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
|
||||||
|
// encode takes at most 40 seconds of the default downstream_timeout of 60
|
||||||
|
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
|
||||||
|
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
|
||||||
|
// 2048x8192 random pixels with an alpha channel takes up to about 34
|
||||||
|
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
|
||||||
|
// 5120x4096 at 8 bits already takes about 36.
|
||||||
|
const MaxAVIFOutputPixels = 4096 * 4096
|
||||||
|
|
||||||
|
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
|
||||||
|
// output would have more than MaxAVIFOutputPixels pixels.
|
||||||
|
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
|
||||||
|
|
||||||
// ErrUnsupportedOutputFormat is returned when the requested output format is
|
// ErrUnsupportedOutputFormat is returned when the requested output format is
|
||||||
// not supported.
|
// not supported.
|
||||||
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
||||||
@@ -179,7 +191,7 @@ type Params struct {
|
|||||||
// If <= 0, DefaultMaxInputBytes is used.
|
// If <= 0, DefaultMaxInputBytes is used.
|
||||||
MaxInputBytes int64
|
MaxInputBytes int64
|
||||||
// MaxConcurrentProcessing is the most images processed at once.
|
// MaxConcurrentProcessing is the most images processed at once.
|
||||||
// If <= 0, the number of CPUs Go uses (runtime.GOMAXPROCS(0)) is used.
|
// If <= 0, config.DefaultMaxConcurrentProcessing() is used.
|
||||||
MaxConcurrentProcessing int
|
MaxConcurrentProcessing int
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -195,7 +207,7 @@ func New(params Params) *ImageProcessor {
|
|||||||
|
|
||||||
maxConcurrentProcessing := params.MaxConcurrentProcessing
|
maxConcurrentProcessing := params.MaxConcurrentProcessing
|
||||||
if maxConcurrentProcessing <= 0 {
|
if maxConcurrentProcessing <= 0 {
|
||||||
maxConcurrentProcessing = runtime.GOMAXPROCS(0)
|
maxConcurrentProcessing = config.DefaultMaxConcurrentProcessing()
|
||||||
}
|
}
|
||||||
|
|
||||||
return &ImageProcessor{
|
return &ImageProcessor{
|
||||||
@@ -211,7 +223,8 @@ func New(params Params) *ImageProcessor {
|
|||||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
||||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
// no way to stop libvips once an encode has begun, so an encode keeps its
|
||||||
// slot until it finishes.
|
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
|
||||||
|
// refused with ErrAVIFOutputTooLarge before anything is decoded.
|
||||||
func (p *ImageProcessor) Process(
|
func (p *ImageProcessor) Process(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
input io.Reader,
|
input io.Reader,
|
||||||
@@ -224,18 +237,9 @@ func (p *ImageProcessor) Process(
|
|||||||
|
|
||||||
defer release()
|
defer release()
|
||||||
|
|
||||||
// Read input with a size limit to prevent unbounded memory consumption.
|
data, err := p.readInput(input)
|
||||||
// We read at most maxInputBytes+1 so we can detect if the input exceeds
|
|
||||||
// the limit without consuming additional memory.
|
|
||||||
limited := io.LimitReader(input, p.maxInputBytes+1)
|
|
||||||
|
|
||||||
data, err := io.ReadAll(limited)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("failed to read input: %w", err)
|
return nil, err
|
||||||
}
|
|
||||||
|
|
||||||
if int64(len(data)) > p.maxInputBytes {
|
|
||||||
return nil, ErrInputDataTooLarge
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// govips picks the libvips loader from the input's first bytes with
|
// govips picks the libvips loader from the input's first bytes with
|
||||||
@@ -278,20 +282,28 @@ func (p *ImageProcessor) Process(
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// Resize if needed
|
|
||||||
if targetWidth != origWidth || targetHeight != origHeight {
|
|
||||||
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// orig is the source's own format; encode refuses an empty format
|
// orig is the source's own format; encode refuses an empty format
|
||||||
outputFormat := req.Format
|
outputFormat := req.Format
|
||||||
if outputFormat == FormatOriginal {
|
if outputFormat == FormatOriginal {
|
||||||
outputFormat = sourceFormat
|
outputFormat = sourceFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Determine output dimensions; too large ones are refused before
|
||||||
|
// resizing, which with FitCover already decodes the input
|
||||||
|
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
|
||||||
|
targetWidth, targetHeight, req.FitMode, outputFormat)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resize if needed
|
||||||
|
if targetWidth != origWidth || targetHeight != origHeight {
|
||||||
|
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Encode to target format
|
// Encode to target format
|
||||||
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -444,6 +456,23 @@ func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
|
|||||||
return release, nil
|
return release, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// readInput reads input with a size limit to prevent unbounded memory
|
||||||
|
// consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
|
||||||
|
// reads at most maxInputBytes+1 so it can detect if the input exceeds the
|
||||||
|
// limit without consuming additional memory.
|
||||||
|
func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
|
||||||
|
data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to read input: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if int64(len(data)) > p.maxInputBytes {
|
||||||
|
return nil, ErrInputDataTooLarge
|
||||||
|
}
|
||||||
|
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
// detectFormat returns the format string for a govips image type.
|
// detectFormat returns the format string for a govips image type.
|
||||||
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||||
switch imageType {
|
switch imageType {
|
||||||
@@ -468,9 +497,61 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// resize resizes the image according to the fit mode. With FitOutside one
|
// outputDimensions returns the dimensions resize gives an image of imgWidth x
|
||||||
// side can come out larger than requested; when it would be above
|
// imgHeight for the target dimensions and fit mode, without resizing it, and
|
||||||
// MaxOutputDimension, resize returns ErrOutputTooLarge without resizing.
|
// refuses an output too large for format. With FitOutside one side can come
|
||||||
|
// out larger than the target; when it would be above MaxOutputDimension,
|
||||||
|
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
|
||||||
|
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
|
||||||
|
func outputDimensions(
|
||||||
|
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
|
||||||
|
) (int, int, error) {
|
||||||
|
scaleW := float64(width) / float64(imgWidth)
|
||||||
|
scaleH := float64(height) / float64(imgHeight)
|
||||||
|
outputWidth, outputHeight := width, height
|
||||||
|
|
||||||
|
switch fit {
|
||||||
|
case FitCover, FitFill, "":
|
||||||
|
// Cover crops, and fill stretches, to exactly the target dimensions
|
||||||
|
|
||||||
|
case FitContain:
|
||||||
|
// Fit within the target dimensions, maintaining aspect ratio
|
||||||
|
scale := min(scaleW, scaleH)
|
||||||
|
outputWidth = int(float64(imgWidth) * scale)
|
||||||
|
outputHeight = int(float64(imgHeight) * scale)
|
||||||
|
|
||||||
|
case FitInside:
|
||||||
|
// Same as contain, but only shrink
|
||||||
|
scale := min(scaleW, scaleH, 1)
|
||||||
|
outputWidth = int(float64(imgWidth) * scale)
|
||||||
|
outputHeight = int(float64(imgHeight) * scale)
|
||||||
|
|
||||||
|
case FitOutside:
|
||||||
|
// The smallest dimension fits, the other may exceed the target
|
||||||
|
scale := max(scaleW, scaleH)
|
||||||
|
outputWidth = int(float64(imgWidth) * scale)
|
||||||
|
outputHeight = int(float64(imgHeight) * scale)
|
||||||
|
|
||||||
|
default:
|
||||||
|
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The side FitOutside scales past the target is held to the maximum too
|
||||||
|
if max(outputWidth, outputHeight) > MaxOutputDimension {
|
||||||
|
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
||||||
|
outputWidth, outputHeight, MaxOutputDimension)
|
||||||
|
}
|
||||||
|
|
||||||
|
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
|
||||||
|
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
|
||||||
|
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
|
||||||
|
}
|
||||||
|
|
||||||
|
return outputWidth, outputHeight, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// resize resizes the image to width x height, the dimensions
|
||||||
|
// outputDimensions gives for the fit mode.
|
||||||
func (p *ImageProcessor) resize(
|
func (p *ImageProcessor) resize(
|
||||||
img *vips.ImageRef, width, height int, fit FitMode,
|
img *vips.ImageRef, width, height int, fit FitMode,
|
||||||
) error {
|
) error {
|
||||||
@@ -479,52 +560,20 @@ func (p *ImageProcessor) resize(
|
|||||||
// Resize and crop to fill exact dimensions (default)
|
// Resize and crop to fill exact dimensions (default)
|
||||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||||
|
|
||||||
case FitContain:
|
|
||||||
// Resize to fit within dimensions, maintaining aspect ratio
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
|
||||||
scaleW := float64(width) / float64(imgW)
|
|
||||||
scaleH := float64(height) / float64(imgH)
|
|
||||||
scale := min(scaleW, scaleH)
|
|
||||||
newW := int(float64(imgW) * scale)
|
|
||||||
newH := int(float64(imgH) * scale)
|
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
|
||||||
|
|
||||||
case FitFill:
|
case FitFill:
|
||||||
// Resize to exact dimensions (may distort)
|
// Resize to exact dimensions (may distort)
|
||||||
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
|
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
|
||||||
|
|
||||||
case FitInside:
|
case FitInside:
|
||||||
// Same as contain, but only shrink
|
// outputDimensions keeps the dimensions of an image that already fits
|
||||||
if img.Width() <= width && img.Height() <= height {
|
if img.Width() == width && img.Height() == height {
|
||||||
return nil // Already fits
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||||
scaleW := float64(width) / float64(imgW)
|
|
||||||
scaleH := float64(height) / float64(imgH)
|
|
||||||
scale := min(scaleW, scaleH)
|
|
||||||
newW := int(float64(imgW) * scale)
|
|
||||||
newH := int(float64(imgH) * scale)
|
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
case FitContain, FitOutside:
|
||||||
|
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||||
case FitOutside:
|
|
||||||
// Resize so smallest dimension fits, may exceed target on other dimension
|
|
||||||
imgW, imgH := img.Width(), img.Height()
|
|
||||||
scaleW := float64(width) / float64(imgW)
|
|
||||||
scaleH := float64(height) / float64(imgH)
|
|
||||||
scale := max(scaleW, scaleH)
|
|
||||||
newW := int(float64(imgW) * scale)
|
|
||||||
newH := int(float64(imgH) * scale)
|
|
||||||
|
|
||||||
// The dimension that exceeds the target is held to the maximum too
|
|
||||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
|
||||||
return fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
|
||||||
newW, newH, MaxOutputDimension)
|
|
||||||
}
|
|
||||||
|
|
||||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
|
||||||
|
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||||
@@ -628,11 +677,16 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
|
|||||||
return output, err
|
return output, err
|
||||||
}
|
}
|
||||||
|
|
||||||
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6.
|
// webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
|
||||||
const webpEffort = 4
|
// default of 4 so that an encode takes at most 40 seconds of the default
|
||||||
|
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
|
||||||
|
// saving an 8192x8192 image of random pixels with an alpha channel takes up
|
||||||
|
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
|
||||||
|
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
|
||||||
|
const webpEffort = 2
|
||||||
|
|
||||||
// exportWebP encodes img as lossy WebP at quality and libvips' default
|
// exportWebP encodes img as lossy WebP at quality and webpEffort, without
|
||||||
// effort, without metadata.
|
// metadata.
|
||||||
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||||
output, _, err := img.ExportWebp(&vips.WebpExportParams{
|
output, _, err := img.ExportWebp(&vips.WebpExportParams{
|
||||||
StripMetadata: true,
|
StripMetadata: true,
|
||||||
@@ -644,12 +698,13 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
||||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
|
// govips can set. With one thread, as pixad runs libvips, 1 takes about 52
|
||||||
// seconds to save an 8192x8192 RGB image of random pixels, and about 110
|
// seconds to save an 8192x8192 RGB image of random pixels at quality 85,
|
||||||
// with an alpha channel, the worst case, against the default
|
// and about 96 with an alpha channel, against the default
|
||||||
// downstream_timeout of 60 seconds. On an image of milder noise, which 1
|
// downstream_timeout of 60 seconds, so an AVIF output is held to
|
||||||
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default,
|
// MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
|
||||||
// 4, takes minutes.
|
// 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
|
||||||
|
// minutes.
|
||||||
const avifEffort = 1
|
const avifEffort = 1
|
||||||
|
|
||||||
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
||||||
@@ -676,10 +731,18 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
|
|||||||
// JPEG that names none.
|
// JPEG that names none.
|
||||||
const jxlResolution = 72 / 25.4
|
const jxlResolution = 72 / 25.4
|
||||||
|
|
||||||
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
|
// jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
|
||||||
// and without metadata. govips sends libvips a distance, the JPEG XL
|
// libvips' default of 7 so that an encode takes at most 40 seconds of the
|
||||||
// encoder's own measure of quality, along with the quality, and libvips then
|
// default downstream_timeout of 60 seconds. With one thread, as pixad runs
|
||||||
// uses the distance alone, so the quality is also given as a distance.
|
// libvips, saving an 8192x8192 image of random pixels with an alpha channel
|
||||||
|
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
|
||||||
|
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
|
||||||
|
const jxlEffort = 4
|
||||||
|
|
||||||
|
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
|
||||||
|
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
|
||||||
|
// measure of quality, along with the quality, and libvips then uses the
|
||||||
|
// distance alone, so the quality is also given as a distance.
|
||||||
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||||
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
|
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
|
||||||
// PNG, but cannot save one as JPEG XL. encode has already converted
|
// PNG, but cannot save one as JPEG XL. encode has already converted
|
||||||
@@ -718,6 +781,7 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
|||||||
params := vips.NewJxlExportParams()
|
params := vips.NewJxlExportParams()
|
||||||
params.Quality = quality
|
params.Quality = quality
|
||||||
params.Distance = jxlDistance(quality)
|
params.Distance = jxlDistance(quality)
|
||||||
|
params.Effort = jxlEffort
|
||||||
|
|
||||||
output, _, err := toSave.ExportJxl(params)
|
output, _, err := toSave.ExportJxl(params)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -115,14 +115,14 @@ func waitForEntries(t *testing.T, entered <-chan struct{}, count int) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
|
func TestNewDefaultsMaxConcurrentProcessingToHalfTheCPUs(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
for _, limit := range []int{0, -1} {
|
for _, limit := range []int{0, -1} {
|
||||||
proc := New(Params{MaxConcurrentProcessing: limit})
|
proc := New(Params{MaxConcurrentProcessing: limit})
|
||||||
if got := cap(proc.processingSemaphore); got != runtime.GOMAXPROCS(0) {
|
if got := cap(proc.processingSemaphore); got != max(runtime.GOMAXPROCS(0)/2, 1) {
|
||||||
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, one per CPU",
|
t.Errorf("MaxConcurrentProcessing %d: %d slots, want %d, half the CPUs",
|
||||||
limit, got, runtime.GOMAXPROCS(0))
|
limit, got, max(runtime.GOMAXPROCS(0)/2, 1))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -132,6 +132,30 @@ func TestNewDefaultsMaxConcurrentProcessingToCPUs(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestNewDefaultIsHalfTheCPUsAndAtLeastOne checks that New, given no
|
||||||
|
// MaxConcurrentProcessing, makes one slot for every two CPUs Go uses,
|
||||||
|
// rounded down, and one slot when Go uses one CPU.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // it sets GOMAXPROCS, which other tests read
|
||||||
|
func TestNewDefaultIsHalfTheCPUsAndAtLeastOne(t *testing.T) {
|
||||||
|
cpusBefore := runtime.GOMAXPROCS(0)
|
||||||
|
|
||||||
|
t.Cleanup(func() { runtime.GOMAXPROCS(cpusBefore) })
|
||||||
|
|
||||||
|
for _, tc := range []struct{ cpus, want int }{
|
||||||
|
{1, 1}, {2, 1}, {3, 1}, {8, 4},
|
||||||
|
} {
|
||||||
|
runtime.GOMAXPROCS(tc.cpus)
|
||||||
|
|
||||||
|
got := cap(New(Params{}).processingSemaphore)
|
||||||
|
t.Logf("GOMAXPROCS %d: %d slots", tc.cpus, got)
|
||||||
|
|
||||||
|
if got != tc.want {
|
||||||
|
t.Errorf("GOMAXPROCS %d: %d slots, want %d", tc.cpus, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
|
// TestProcessNeverExceedsMaxConcurrentProcessing starts more Process calls
|
||||||
// than MaxConcurrentProcessing allows and holds the first ones inside
|
// than MaxConcurrentProcessing allows and holds the first ones inside
|
||||||
// Process until the test lets them go. No more than the limit may be
|
// Process until the test lets them go. No more than the limit may be
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ type ServiceConfig struct {
|
|||||||
// Allowlist is the list of hosts that don't require signatures
|
// Allowlist is the list of hosts that don't require signatures
|
||||||
Allowlist []string
|
Allowlist []string
|
||||||
// MaxConcurrentProcessing is the most images processed at once; zero
|
// MaxConcurrentProcessing is the most images processed at once; zero
|
||||||
// uses the image processor's default, one per CPU
|
// uses the image processor's default, half the CPUs
|
||||||
MaxConcurrentProcessing int
|
MaxConcurrentProcessing int
|
||||||
// Logger for logging
|
// Logger for logging
|
||||||
Logger *slog.Logger
|
Logger *slog.Logger
|
||||||
|
|||||||
@@ -47,6 +47,9 @@ func ValidateDimension(name string, value int) error {
|
|||||||
var sizeFormatRegex = regexp.MustCompile(
|
var sizeFormatRegex = regexp.MustCompile(
|
||||||
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
||||||
|
|
||||||
|
// percentEscapeRegex matches one percent-escape, such as "%2e"
|
||||||
|
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
|
||||||
|
|
||||||
// ParsedURL contains the parsed components of an image proxy URL.
|
// ParsedURL contains the parsed components of an image proxy URL.
|
||||||
type ParsedURL struct {
|
type ParsedURL struct {
|
||||||
// Host is the source origin host (e.g., "cdn.example.com")
|
// Host is the source origin host (e.g., "cdn.example.com")
|
||||||
@@ -61,9 +64,8 @@ type ParsedURL struct {
|
|||||||
Format ImageFormat
|
Format ImageFormat
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImagePath parses the path captured by chi's wildcard:
|
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
||||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
|
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
||||||
// This is the primary entry point when using chi routing.
|
|
||||||
// Examples:
|
// Examples:
|
||||||
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
||||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||||
@@ -79,9 +81,10 @@ func ParseImagePath(path string) (*ParsedURL, error) {
|
|||||||
return parseImageComponents(path)
|
return parseImageComponents(path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
|
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
|
||||||
// or /v1/image/<host>/<path>/<size> for JPEG XL
|
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
||||||
// Use ParseImagePath instead when working with chi's wildcard capture.
|
// for JPEG XL. A source query follows the source path after a "?", which the
|
||||||
|
// URL writes as %3F.
|
||||||
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||||
// Remove the /v1/image/ prefix
|
// Remove the /v1/image/ prefix
|
||||||
const prefix = "/v1/image/"
|
const prefix = "/v1/image/"
|
||||||
@@ -100,12 +103,6 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
|||||||
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
||||||
// <host>/<path>/<size> for JPEG XL.
|
// <host>/<path>/<size> for JPEG XL.
|
||||||
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||||
// Check for path traversal before any other processing
|
|
||||||
err := checkPathTraversal(remainder)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
// Find the last path segment, which holds "size" or "size.format"
|
// Find the last path segment, which holds "size" or "size.format"
|
||||||
lastSlash := strings.LastIndex(remainder, "/")
|
lastSlash := strings.LastIndex(remainder, "/")
|
||||||
if lastSlash == -1 {
|
if lastSlash == -1 {
|
||||||
@@ -161,6 +158,13 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
|||||||
path = path[:qIndex]
|
path = path[:qIndex]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The source query names no file on the upstream, so only the host and
|
||||||
|
// the source path are checked for path traversal
|
||||||
|
err = checkPathTraversal(host + path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
// Ensure path starts with /
|
// Ensure path starts with /
|
||||||
if !strings.HasPrefix(path, "/") {
|
if !strings.HasPrefix(path, "/") {
|
||||||
path = "/" + path
|
path = "/" + path
|
||||||
@@ -186,12 +190,7 @@ func checkPathTraversal(path string) error {
|
|||||||
// Decode multiple times to catch double-encoding
|
// Decode multiple times to catch double-encoding
|
||||||
decoded := path
|
decoded := path
|
||||||
for range 3 {
|
for range 3 {
|
||||||
newDecoded, err := url.PathUnescape(decoded)
|
newDecoded := decodePercentEscapes(decoded)
|
||||||
if err != nil {
|
|
||||||
// Malformed encoding is suspicious
|
|
||||||
return ErrPathTraversal
|
|
||||||
}
|
|
||||||
|
|
||||||
if newDecoded == decoded {
|
if newDecoded == decoded {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -232,6 +231,18 @@ func checkPathTraversal(path string) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
|
||||||
|
// keeps a "%" that does not start one as it is, where url.PathUnescape would
|
||||||
|
// refuse the whole string. A source path holding %25, a literal "%", has such
|
||||||
|
// a "%" after one more decoding.
|
||||||
|
func decodePercentEscapes(s string) string {
|
||||||
|
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
|
||||||
|
decoded, _ := url.PathUnescape(escape)
|
||||||
|
|
||||||
|
return decoded
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||||
|
|||||||
@@ -407,6 +407,71 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
|
||||||
|
// a source query holding %25, the percent-escape of a literal "%", are
|
||||||
|
// accepted as they are, although decoding them once more leaves a "%" that
|
||||||
|
// does not start a percent-escape.
|
||||||
|
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
input string
|
||||||
|
want *ParsedURL
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "path",
|
||||||
|
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
|
||||||
|
want: &ParsedURL{
|
||||||
|
Host: testHostCDN, Path: "/sale/50%25off.jpg",
|
||||||
|
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "query",
|
||||||
|
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
|
||||||
|
want: &ParsedURL{
|
||||||
|
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
||||||
|
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
got, err := ParseImageURL(tt.input)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertParsedURL(t, got, tt.want)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
|
||||||
|
// double-encoded ".." is still refused when the path also holds a "%" that
|
||||||
|
// does not start a percent-escape, either as written or after one more
|
||||||
|
// decoding.
|
||||||
|
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, input := range []string{
|
||||||
|
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
|
||||||
|
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
|
||||||
|
} {
|
||||||
|
_, err := ParseImageURL(input)
|
||||||
|
t.Logf("ParseImageURL(%q) error = %v", input, err)
|
||||||
|
|
||||||
|
if !errors.Is(err, ErrPathTraversal) {
|
||||||
|
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
|
||||||
|
input, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// errorIs checks if err matches target (handles wrapped errors).
|
// errorIs checks if err matches target (handles wrapped errors).
|
||||||
func errorIs(err, target error) bool {
|
func errorIs(err, target error) bool {
|
||||||
if errors.Is(err, target) {
|
if errors.Is(err, target) {
|
||||||
|
|||||||
@@ -126,31 +126,20 @@ func (s *Signer) GenerateSignedURL(
|
|||||||
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Build the path.
|
// Build the path. A source query follows the source path after a "?".
|
||||||
// When a source query is present, it is embedded as a path segment
|
// The image route reads the path decoded once, so the whole path is
|
||||||
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract
|
// written in Go's own percent-encoding of a URL path, which writes "?"
|
||||||
// it from the last-slash split. The "?" inside a path segment is
|
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
|
||||||
// percent-encoded by clients but chi delivers it decoded, which is
|
// are.
|
||||||
// exactly what the URL parser expects.
|
sourcePathAndQuery := req.SourcePath
|
||||||
var path string
|
|
||||||
if req.SourceQuery != "" {
|
if req.SourceQuery != "" {
|
||||||
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
|
sourcePathAndQuery += "?" + req.SourceQuery
|
||||||
req.SourceHost,
|
|
||||||
req.SourcePath,
|
|
||||||
url.PathEscape(req.SourceQuery),
|
|
||||||
sizeStr,
|
|
||||||
req.Format,
|
|
||||||
)
|
|
||||||
} else {
|
|
||||||
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
|
||||||
req.SourceHost,
|
|
||||||
req.SourcePath,
|
|
||||||
sizeStr,
|
|
||||||
req.Format,
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return path, sig, exp
|
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||||
|
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
|
||||||
|
|
||||||
|
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildSignatureData creates the string to be signed.
|
// buildSignatureData creates the string to be signed.
|
||||||
|
|||||||
@@ -475,6 +475,44 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
|
||||||
|
// source URL with a query, the image at
|
||||||
|
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
|
||||||
|
// signature with the README's key, and the path GenerateSignedURL writes for
|
||||||
|
// it, with the "?" written as %3F and the "%" as %25.
|
||||||
|
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
signer := signature.New("example-signing-key-for-documentation")
|
||||||
|
|
||||||
|
req := signature.Request{
|
||||||
|
SourceHost: testHost,
|
||||||
|
SourcePath: "/img",
|
||||||
|
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||||
|
Width: 800,
|
||||||
|
Height: 600,
|
||||||
|
Format: testFormatWebP,
|
||||||
|
Quality: 85,
|
||||||
|
FitMode: testFitCover,
|
||||||
|
Expires: time.Unix(1704067200, 0),
|
||||||
|
}
|
||||||
|
|
||||||
|
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
|
||||||
|
|
||||||
|
gotSig := signer.Sign(&req)
|
||||||
|
if gotSig != wantSig {
|
||||||
|
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
|
||||||
|
}
|
||||||
|
|
||||||
|
const wantPath = "/v1/image/cdn.example.com" +
|
||||||
|
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
|
||||||
|
|
||||||
|
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
|
||||||
|
if gotPath != wantPath {
|
||||||
|
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestParseParams(t *testing.T) {
|
func TestParseParams(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user