Author SHA1 Message Date
sneak 5506858834 Test the generator refusals and a "'" in the source query
check / check (push) Waiting to run
Through the router: a source query holding "'" must reach the upstream
as %27. The generator page must answer 400, show the URL pixa would
fetch and make no encrypted URL for a source path holding %2F next to a
space, a "|" or a letter outside ASCII, and for a source URL holding a
user name and password. It must also refuse a source path holding a ".."
segment, as /v1/image/ does. These fail on the branch; the fix follows.

Model: opus-5-5
2026-10-16 00:56:05 +00:00
sneak 545507e9d4 Fetch the source path and query URL-encoded, not joined as text (closes #288)
The source path, decoded, is fetched percent-encoded again by net/url,
so 100%25.png fetches /100%25.png, and my%20cat.png and my%2520cat.png
fetch different paths. The source query stays as the source URL writes
it; the characters a browser percent-encodes in a query, such as a
space, are percent-encoded when the image URL is read, so the query is
signed and cached as it is sent. The generator page answers 400 for a
source URL its image URL would fetch written another way, such as with
%2F in the path, and shows the URL pixa would fetch. README.md says how
the source path and query are fetched.

Model: opus-5-5
2026-10-16 00:47:13 +00:00
sneak af5ce499b9 Test that the source URL is fetched with its path and query encoded
Through the router, against a test upstream that records each request
it receives: a space in the source query, 100%25.png, the source paths
written my%20cat.png and my%2520cat.png, one source query written two
ways, and the generator page with %20, %25 and %23 in the source URL,
and with %2F in its path, which the page must refuse. These fail on
next; the fix for #288 follows.

Model: opus-5-5
2026-10-16 00:47:13 +00:00
clawbot 9a96740463 Crop the centre before resizing for fit=cover (closes #282)
check / check (push) Waiting to run
For fit=cover, the default fit, libvips' thumbnail scaled the whole
source to cover the requested size and copied that into memory before
cutting out the centre, so an 8192x64 source at 8192x8192 asked for
about 24 GiB, and 8192x1 failed with 500. resize now first cuts the
centre of the source, widened to whole source pixels, and gives only
that to thumbnail, which scales both sides by the same amount as before;
the image it holds exceeds the output by less than one scaled source
pixel on each side. Output size and scale are unchanged; its position
can move by up to one output pixel.

Model: opus-5-5
2026-10-16 02:23:18 +02:00
clawbot 1fd5cf8bd0 Refuse an image URL whose source path or query holds a # (closes #286)
check / check (push) Canceled after 0s
A # written as %23 in the source path or query reached the parser
decoded and stayed in the source URL, where Go reads it as the start of
a fragment and drops it before fetching. So each text after the # was
another unsigned spelling of one upstream request, cached under its own
keys. An image URL whose decoded source path or query holds a # is now
refused with 400, signed or not, and README.md's rule for writing a
source path and query says a # cannot be part of either.

Model: opus-5-5
2026-10-16 01:36:56 +02:00
clawbot 62b4f4791e Refuse an image URL whose host is not a host name or IP address (closes #275)
check / check (push) Canceled after 0s
A ? written into the host segment as %3F reached the parser decoded,
so cdn.example.com?x=1 passed the allowlist as cdn.example.com, fetched
the host's root path with that query, and was cached under its own keys:
another unsigned spelling of one upstream request. The host segment is
now accepted only as a host name, an IPv4 address or a bracketed IPv6
address, with an optional port, and anything else is refused with 400.
An IPv6 address without brackets is now refused instead of needing a
signature; a host name outside ASCII is refused, its punycode accepted.

Model: opus-5-5
2026-10-16 01:14:51 +02:00
9 changed files with 958 additions and 25 deletions
+34 -9
View File
@@ -323,13 +323,28 @@ first `/`, the size and format after the last `/`, and between them the source
path up to the first `?` and the source query after it. So the `?` before the
query, any `?` in the query, and every `%` in the source path or query must be
percent-encoded, as `%3F` and `%25`. Any other character may be written as
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
example,
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. Neither
the source path nor the source query may hold a `#`, and an image URL that
writes one as `%23` is refused with 400. For example,
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
800x600 WebP is
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
signed as shown under Signature Specification.
The source path is the path with its percent-escapes decoded, and pixa fetches
it percent-encoded again: every character other than letters, digits, `-`, `.`,
`_`, `~`, `/`, `$`, `&`, `+`, `,`, `:`, `;`, `=` and `@` is sent as a
percent-escape. So `my%20cat.jpg` in the image URL is the source path
`/my cat.jpg`, fetched as `/my%20cat.jpg`; `my%2520cat.jpg` is the source path
`/my%20cat.jpg`, fetched as `/my%2520cat.jpg`; and `100%25.png` is fetched as
`/100%25.png`. The source query is the source URL's query as that URL writes it,
its percent-escapes included, and is sent as it is, except that the characters a
browser percent-encodes in a query (a space, `"`, `<`, `>`, control characters
and characters outside ASCII) and a `%` that does not begin a percent-escape are
percent-encoded. pixa signs and caches the query in that form, so
`img%3Fname=my%20cat` and `img%3Fname=my%2520cat` in the image URL are the same
source query, `name=my%20cat`.
A size with one side `0` has that side scaled to keep the source's proportions,
and is refused with 400 when that side would come out above 8192, the most
either side may be; so is a size with `fit=outside`, which scales the image to
@@ -444,7 +459,11 @@ nor change what it asks for.
sent with an empty format, or none, also makes a JPEG XL URL. Width and
height both empty or `0` keep the original size; if only one of them is empty
or `0`, that side is scaled to keep the image's proportions. A URL whose
scaled side would come out above 8192 answers 400.
scaled side would come out above 8192 answers 400. So does a source URL that
the image URL would fetch written another way, such as one whose path holds
`%2F`, fetched as `/`, or `(`, fetched as `%28`, as pixa keeps the source
path decoded and fetches it percent-encoded again (see Routes); the page then
shows the URL pixa would fetch.
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
it expires. `<host>` is the host the page was opened on, and the URL starts
with `http` instead while `debug` is on. The name after the token is ignored
@@ -506,11 +525,13 @@ Where:
- `host` — source host as the URL writes it up to the first `/`, in lower case,
keeping an IPv6 address's brackets and any `:` and port after it (e.g.
`cdn.example.com`, `cdn.example.com:8443`, `[2001:db8::1]:8443`)
- `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
- `query` — source query string, empty string if none, with `%` and `:` written
as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
`/a` with query `b:` are signed differently
- `path` — source path, with its percent-escapes decoded as under Routes (e.g.
`/photos/cat.jpg`, or `/photos/my cat.jpg` for `/photos/my%20cat.jpg`), with
each `%` written as `%25` and each `:` as `%3A`, so `/photos/a:b.jpg` is
signed as `/photos/a%3Ab.jpg`
- `query` — source query string as under Routes, empty string if none, with `%`
and `:` written as in `path`; neither then holds a `:`, so path `/a:b` with no
query and path `/a` with query `b:` are signed differently
- `width` — requested width in pixels, `0` for original
- `height` — requested height in pixels, `0` for original
- `format` — output format, one of those listed under Routes, with `original`
@@ -575,7 +596,11 @@ pixa lower-cases the host of a `/v1/image/` URL before anything else uses it, so
`CDN.Example.com` is matched, fetched, cached and counted against
`upstream_connections_per_host` as `cdn.example.com`. Such a URL whose host has
a port, as in `cdn.example.com:8443`, or a `:` with no port, needs a signature
even when the host is allowlisted. A host holding `@` is refused with 400.
even when the host is allowlisted. The host must be a host name or an IPv4
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
brackets, either one optionally followed by `:` and a port. Any other host, such
as one holding `@`, or a `?`, `#`, `%`, `\` or space written percent-encoded, is
refused with 400.
### Configuration
+39
View File
@@ -30,6 +30,45 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-15 a thin source requested with `fit=cover` no longer makes libvips
build a huge image (closes #282): libvips' thumbnail scaled the whole source
to cover the requested size and copied that into memory before it cut out the
centre, so an 8192x64 source at 8192x8192 made it allocate about 24 GiB, and
an 8192x1 source answered 500. The image processor now first cuts the centre
of the source, widened to whole source pixels, and thumbnail scales and cuts
only that, by the same amount on both sides as before. What it copies into
memory is then larger than the output by less than one scaled source pixel on
each side. With the runtime image's libvips and JPEG output at 8192x8192,
resident memory peaked at about 242 MiB for the 8192x64 source and 431 MiB for
the 8192x1 one. The output can sit a fraction of an output pixel from where it
was.
- 2026-10-16 the source URL a request fetches is built with `net/url` from the
source host, path and query, not joined as text (closes #288). The source
path, decoded, is fetched percent-encoded again, so `100%25.png` fetches
`/100%25.png` where it answered 401, and `my%20cat.png` and `my%2520cat.png`
fetch different paths where both fetched `/my%20cat.png` under two cache keys.
The source query stays as the source URL writes it; the characters a browser
percent-encodes in a query, such as a space, which was sent raw and answered
502, are percent-encoded when the image URL is read, so the query is signed
and cached as it is sent. The generator page answers 400, showing the URL pixa
would fetch, for a source URL its image URL would fetch written another way,
such as with `%2F` in the path.
- 2026-10-15 a `/v1/image/` URL whose source path or source query holds a `#`,
written as `%23`, is refused with 400 (closes #286): the source URL built from
it ended at the `#` when fetched, so each text after it fetched the same
upstream image, without a signature for an allowlisted host, and cached it
under keys of its own. `README.md` says neither may hold a `#`.
- 2026-10-15 the host of a `/v1/image/` URL must be a host name or an IPv4
address (letters, digits, hyphens, underscores and dots) or an IPv6 address in
brackets, either one optionally followed by `:` and a port, and any other host
is refused with 400 (closes #275): only `@` was refused, so a `?` written as
`%3F` gave the host `cdn.example.com?x=1`, which Go's URL parser reads as
`cdn.example.com` with a query. Such a URL passed the allowlist, and each text
after the `?` fetched the host's root path and cached the answer under keys of
its own. A `#`, `%`, `\` or space written percent-encoded is refused the same
way, and so is an IPv6 address without brackets, which needed a signature
before. Underscores are kept because `allowlist_hosts` accepts host names
holding them.
- 2026-10-15 the README says how a source URL with a query is written into an
image URL, and the image route accepts it (closes #255): the query follows the
source path after `%3F`. pixa now reads the image URL's path decoded once, so
+16
View File
@@ -141,6 +141,22 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
// The image URL must fetch the source URL as it was written. The
// token keeps the path decoded and pixa fetches it percent-encoded
// again, so a path written another way, such as with %2F, would fetch
// another URL. The scheme is not kept: only path and query are compared
sourceRequest := payload.ToImageRequest()
sourceRequest.AllowHTTP = s.config.AllowHTTP
fetchURL, err := url.Parse(sourceRequest.SourceURL())
if err != nil || fetchURL.RequestURI() != parsed.RequestURI() {
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
"written: pixa would fetch "+sourceRequest.SourceURL(),
r.Form, http.StatusBadRequest)
return
}
// Generate encrypted token
token, err := s.encGen.Generate(payload)
if err != nil {
@@ -264,3 +264,127 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
})
}
}
// TestHandleImage_InvalidHost_Returns400 sends image URLs whose host no URL
// host can be, and checks that each is refused with 400 naming the host as
// decoded: allowlistedHost with a character after it that a URL host cannot
// hold, percent-encoded, an IPv6 address without brackets, and brackets
// around something that is not an IPv6 address. A "?" or "#" after
// allowlistedHost would otherwise end the host when the source URL is read,
// so the request would pass the allowlist and be fetched and cached under a
// host that is not the one fetched.
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost)
tests := []struct {
name string
writtenHost string
decodedHost string
}{
{"? written as %3F", allowlistedHost + "%3Fx=1", allowlistedHost + "?x=1"},
{"# written as %23", allowlistedHost + "%23x", allowlistedHost + "#x"},
{"% written as %25", allowlistedHost + "%253F", allowlistedHost + "%3F"},
{`\ written as %5C`, allowlistedHost + "%5Cx", allowlistedHost + `\x`},
{"space written as %20", allowlistedHost + "%20", allowlistedHost + " "},
{"IPv6 address without brackets", "2001:db8::1", "2001:db8::1"},
{"IPv4 address in brackets", "[203.0.113.5]", "[203.0.113.5]"},
{"no address in brackets", "[abc]", "[abc]"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), photoURL(tt.writtenHost))
checkErrorBody(t, rec, http.StatusBadRequest,
fmt.Sprintf("invalid image URL: invalid source host: %q",
tt.decodedHost))
})
}
}
// TestHandleImage_HostForms_Served checks that the image route serves, with
// a valid signature, the image at photoPath on a host name, an IPv4 address,
// an IPv6 address in brackets, and a host followed by ":" with or without a
// port.
func TestHandleImage_HostForms_Served(t *testing.T) {
t.Parallel()
hosts := []string{
allowlistedHost,
"img_1.example.com",
"203.0.113.5",
"[2001:db8::1]",
allowlistedHost + ":8443",
"[2001:db8::1]:8443",
allowlistedHost + ":",
}
photos := newPhotoFetcher(t, hosts...)
exp := time.Now().Add(time.Hour)
for _, host := range hosts {
t.Run(host, func(t *testing.T) {
t.Parallel()
sig := photoSignature(testSigningKey, host, exp)
rec := sendGet(t, newImageRoute(t, photos),
photoURLWithSig(host, sig, exp))
if rec.Code != http.StatusOK {
t.Errorf("status = %d, want %d: %s",
rec.Code, http.StatusOK, rec.Body)
}
})
}
}
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
// source path or source query holds a "#", written as %23, and checks that
// each is refused with 400, the one signed for that source path included. The
// source URL would otherwise end at the "#" when it is fetched, so each text
// after it would fetch the image at photoPath and cache it under keys of its
// own.
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
t.Parallel()
photos := newPhotoFetcher(t, allowlistedHost, signedHost)
exp := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(&signature.Request{
SourceHost: signedHost,
SourcePath: photoPath + "#x",
Width: 50,
Height: 50,
Format: string(imgcache.FormatJPEG),
Quality: 85,
FitMode: string(imgcache.FitCover),
Expires: exp,
})
tests := []struct {
name string
target string
}{
{"# in the source path",
"/v1/image/" + allowlistedHost + photoPath + "%23x/50x50.jpeg"},
{"# in the source query",
"/v1/image/" + allowlistedHost + photoPath + "%3Fv=1%23x/50x50.jpeg"},
{"# in the source path, signed",
fmt.Sprintf("/v1/image/%s%s%%23x/50x50.jpeg?sig=%s&exp=%d",
signedHost, photoPath, sig, exp.Unix())},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
rec := sendGet(t, newImageRoute(t, photos), tt.target)
checkErrorBody(t, rec, http.StatusBadRequest,
`invalid image URL: invalid image path: `+
`the source path or query holds a "#"`)
})
}
}
@@ -0,0 +1,268 @@
package imageprocessor
import (
"bytes"
"image"
"image/color"
"image/png"
"math"
"testing"
)
// TestImageProcessor_FitCoverServesThinSource verifies that FitCover serves
// a thin source at the requested size. An 8192x1 source scaled to cover
// 8192x8192 is 8192 times as long as the output on its long side. The centre
// of an 8192x1 source with the proportions of 1x8192 is 1/8192 of a pixel
// wide. Each case is also turned on its side.
func TestImageProcessor_FitCoverServesThinSource(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
size Size
}{
{"8192x1 at 8192x8192", 8192, 1, Size{Width: 8192, Height: 8192}},
{"8192x1 at 1x8192", 8192, 1, Size{Width: 1, Height: 8192}},
{"1x8192 at 8192x8192", 1, 8192, Size{Width: 8192, Height: 8192}},
{"1x8192 at 8192x1", 1, 8192, Size{Width: 8192, Height: 1}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createTestJPEG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatJPEG, FitMode: FitCover})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
t.Logf("Process() made %dx%d", result.Width, result.Height)
if result.Width != tt.size.Width || result.Height != tt.size.Height {
t.Errorf("Process() made %dx%d, want %dx%d",
result.Width, result.Height, tt.size.Width, tt.size.Height)
}
})
}
}
// createBandsPNG creates a PNG of three equal bands, red, green and blue,
// side by side when it is wider than tall and top to bottom otherwise.
func createBandsPNG(t *testing.T, width, height int) []byte {
t.Helper()
bands := []color.RGBA{
{R: 255, A: 255},
{G: 255, A: 255},
{B: 255, A: 255},
}
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
band := 3 * y / height
if width > height {
band = 3 * x / width
}
img.Set(x, y, bands[band])
}
}
return encodePNG(t, img)
}
// TestImageProcessor_FitCoverCropsTheCentre verifies that FitCover keeps the
// centre of the source: a 300x100 source of red, green and blue bands side
// by side, or a 100x300 one with them top to bottom, comes out at 50x50 as
// its green band.
func TestImageProcessor_FitCoverCropsTheCentre(t *testing.T) {
t.Parallel()
tests := []struct {
name string
inputWidth, inputHeight int
}{
{"bands side by side", 300, 100},
{"bands top to bottom", 100, 300},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createBandsPNG(t, tt.inputWidth, tt.inputHeight)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{
Size: Size{Width: 50, Height: 50},
Format: FormatPNG,
FitMode: FitCover,
})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
t.Logf("Process() made %dx%d", bounds.Dx(), bounds.Dy())
if bounds.Dx() != 50 || bounds.Dy() != 50 {
t.Fatalf("Process() made %dx%d, want 50x50",
bounds.Dx(), bounds.Dy())
}
// Resampling may blend a little of the bands beside the
// green one into the output's edges.
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
for x := bounds.Min.X; x < bounds.Max.X; x++ {
r, g, b, _ := output.At(x, y).RGBA()
if r>>8 > 64 || g>>8 < 192 || b>>8 > 64 {
t.Fatalf("output pixel %d,%d is %d,%d,%d, want green",
x, y, r>>8, g>>8, b>>8)
}
}
}
})
}
}
// The greys of createSquarePNG, and the grey halfway between them.
const (
backgroundGrey = 192
squareGrey = 64
halfwayGrey = (backgroundGrey + squareGrey) / 2
)
// createSquarePNG creates a grey PNG side pixels square with a darker grey
// square squareSide pixels wide in its centre.
func createSquarePNG(t *testing.T, side, squareSide int) []byte {
t.Helper()
first := (side - squareSide) / 2
last := first + squareSide - 1
img := image.NewGray(image.Rect(0, 0, side, side))
for y := range side {
for x := range side {
grey := uint8(backgroundGrey)
if x >= first && x <= last && y >= first && y <= last {
grey = squareGrey
}
img.SetGray(x, y, color.Gray{Y: grey})
}
}
return encodePNG(t, img)
}
// squareLength returns how long the square is along greys, the greys of a
// row or column of the output that crosses it: the distance between the
// first and the last point where greys passes halfwayGrey, or 0 when there
// is no such point. Resampling blurs the square's edges over several
// pixels, and each point is placed between the two pixels on either side
// of it in proportion to their greys, so the length is measured to a
// fraction of a pixel.
func squareLength(greys []float64) float64 {
first, last := -1.0, -1.0
for i := 1; i < len(greys); i++ {
before, after := greys[i-1], greys[i]
if (before > halfwayGrey) == (after > halfwayGrey) {
continue
}
point := float64(i-1) + (before-halfwayGrey)/(before-after)
if first < 0 {
first = point
}
last = point
}
return last - first
}
// TestImageProcessor_FitCoverKeepsProportions verifies that FitCover scales
// both sides of the source by the same amount: a square in the centre of
// the source comes out square. Its width and height are measured along the
// output's middle row and column.
func TestImageProcessor_FitCoverKeepsProportions(t *testing.T) {
t.Parallel()
tests := []struct {
name string
side, squareSide int
size Size
}{
{"16x16 at 100x60", 16, 8, Size{Width: 100, Height: 60}},
{"16x16 at 60x100", 16, 8, Size{Width: 60, Height: 100}},
{"10x10 at 100x35", 10, 2, Size{Width: 100, Height: 35}},
{"10x10 at 35x100", 10, 2, Size{Width: 35, Height: 100}},
{"64x64 at 300x100", 64, 16, Size{Width: 300, Height: 100}},
{"64x64 at 100x300", 64, 16, Size{Width: 100, Height: 300}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
input := createSquarePNG(t, tt.side, tt.squareSide)
result, err := New(Params{}).Process(t.Context(),
bytes.NewReader(input),
&Request{Size: tt.size, Format: FormatPNG, FitMode: FitCover})
if err != nil {
t.Fatalf("Process() error = %v", err)
}
defer func() { _ = result.Content.Close() }()
output, err := png.Decode(result.Content)
if err != nil {
t.Fatalf("failed to decode output: %v", err)
}
bounds := output.Bounds()
if bounds.Dx() != tt.size.Width || bounds.Dy() != tt.size.Height {
t.Fatalf("Process() made %dx%d, want %dx%d",
bounds.Dx(), bounds.Dy(), tt.size.Width, tt.size.Height)
}
row := make([]float64, bounds.Dx())
for x := range row {
r, _, _, _ := output.At(x, bounds.Dy()/2).RGBA()
row[x] = float64(r >> 8)
}
column := make([]float64, bounds.Dy())
for y := range column {
r, _, _, _ := output.At(bounds.Dx()/2, y).RGBA()
column[y] = float64(r >> 8)
}
width, height := squareLength(row), squareLength(column)
t.Logf("square is %.2f wide and %.2f high", width, height)
if width == 0 || math.Abs(width-height) > 0.5 {
t.Errorf("square is %.2f wide and %.2f high, want it square",
width, height)
}
})
}
}
+34 -2
View File
@@ -475,8 +475,8 @@ func (p *ImageProcessor) resize(
) error {
switch fit {
case FitCover, "":
// Resize and crop to fill exact dimensions (default)
return img.Thumbnail(width, height, vips.InterestingCentre)
// Crop the centre and resize to fill exact dimensions (default)
return cropCentreAndResize(img, width, height)
case FitContain:
// Resize to fit within dimensions, maintaining aspect ratio
@@ -530,6 +530,38 @@ func (p *ImageProcessor) resize(
}
}
// cropCentreAndResize scales img by the same amount on both sides, the
// least that makes it cover width x height, and cuts the centre width x
// height out of it, as libvips' thumbnail does with InterestingCentre.
// thumbnail alone scales the whole image and copies the result into memory
// before it cuts, and for a thin image that is far larger than the image or
// the output: 8192x64 at 8192x8192 is 1048576x8192, about 24 GiB. So the
// centre is cut from img first, widened to whole pixels of img, which adds
// less than one pixel of img on each side, and thumbnail scales and cuts
// only that.
func cropCentreAndResize(img *vips.ImageRef, width, height int) error {
imgW, imgH := img.Width(), img.Height()
// An image wider than the requested proportions keeps its height and
// loses width down to imgH*width/height, any other keeps its width and
// loses height down to imgW*height/width. Half of what is lost comes
// off each side, rounded down to whole pixels, so the part kept holds
// all of the centre.
left, top := 0, 0
if imgW*height > imgH*width {
left = (imgW*height - imgH*width) / (2 * height) //nolint:mnd // each side
} else {
top = (imgH*width - imgW*height) / (2 * width) //nolint:mnd // each side
}
err := img.ExtractArea(left, top, imgW-2*left, imgH-2*top)
if err != nil {
return err
}
return img.Thumbnail(width, height, vips.InterestingCentre)
}
const defaultQuality = 85
// encode encodes an image to the specified format. govips gives pixa no way
+16 -8
View File
@@ -5,6 +5,7 @@ import (
"context"
"errors"
"io"
"net/url"
"time"
)
@@ -86,9 +87,11 @@ func ValidateImageRequest(req *ImageRequest) error {
type ImageRequest struct {
// SourceHost is the origin host (e.g., "cdn.example.com")
SourceHost string
// SourcePath is the path on the origin (e.g., "/photos/cat.jpg")
// SourcePath is the path on the origin with its percent-escapes decoded,
// as url.URL's Path holds it (e.g., "/photos/my cat.jpg")
SourcePath string
// SourceQuery is the optional query string for the origin URL
// SourceQuery is the optional query string for the origin URL as the URL
// writes it, percent-escapes included, as url.URL's RawQuery holds it
SourceQuery string
// Size is the requested output dimensions
Size Size
@@ -107,20 +110,25 @@ type ImageRequest struct {
AllowHTTP bool
}
// SourceURL returns the full upstream URL to fetch.
// Uses http:// scheme when AllowHTTP is true, otherwise https://.
// SourceURL returns the full upstream URL to fetch: the source path
// percent-encoded again, as net/url writes a path, so each source path is one
// upstream path, and the source query as it is, with the characters
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
// true, otherwise https://.
func (r *ImageRequest) SourceURL() string {
scheme := "https"
if r.AllowHTTP {
scheme = "http"
}
url := scheme + "://" + r.SourceHost + r.SourcePath
if r.SourceQuery != "" {
url += "?" + r.SourceQuery
sourceURL := url.URL{
Scheme: scheme,
Host: r.SourceHost,
Path: r.SourcePath,
RawQuery: escapeQuery(r.SourceQuery),
}
return url
return sourceURL.String()
}
// ImageResponse represents a processed image ready to serve
+56 -6
View File
@@ -3,6 +3,7 @@ package imgcache
import (
"errors"
"fmt"
"net/netip"
"net/url"
"regexp"
"strconv"
@@ -50,6 +51,20 @@ var sizeFormatRegex = regexp.MustCompile(
// percentEscapeRegex matches one percent-escape, such as "%2e"
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
// queryEscapeRegex matches a percent-escape, or one character that a browser
// percent-encodes in a URL's query: one outside the printable ASCII characters
// "!" to "~", such as a space, a control character or a character outside
// ASCII, or one of `"#<>`. A "%" that does not begin a percent-escape matches too.
var queryEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}|[^!-~]|["#<>%]`)
// hostRegex matches the forms the host of an image URL may take: a host name
// or an IPv4 address (letters, digits, "-", "_" and "."), or an IPv6 address
// in brackets (hex digits, ":" and "."), either one optionally followed by ":"
// and a port, which may be empty. Its one group holds the text in brackets,
// which parseImageComponents also checks is an IPv6 address.
var hostRegex = regexp.MustCompile(
`^(?:[A-Za-z0-9_.-]+|\[([0-9A-Fa-f:.]+)\])(?::[0-9]*)?$`)
// ParsedURL contains the parsed components of an image proxy URL.
type ParsedURL struct {
// Host is the source origin host (e.g., "cdn.example.com")
@@ -141,10 +156,20 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
return nil, ErrMissingHost
}
// A URL reads what comes before "@" as a user name and password, and the
// rest as the host
if strings.Contains(host, "@") {
return nil, fmt.Errorf("%w: %q holds @", ErrInvalidHost, host)
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
// the host would make Go's URL parser, which the allowlist check and the
// fetcher use, read a different host from the one allowlisted and cached
hostMatch := hostRegex.FindStringSubmatch(host)
if hostMatch == nil {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
// Go's URL parser refuses brackets around anything but an IPv6 address
if inBrackets := hostMatch[1]; inBrackets != "" {
addr, err := netip.ParseAddr(inBrackets)
if err != nil || !addr.Is6() {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
}
// Host names do not depend on letter case, so the allowlist, the upstream
@@ -152,9 +177,20 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
// host
host = strings.ToLower(host)
// Extract query string if present in path
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
// fragment and drops the rest, so each text after it would be another
// spelling, cached under its own keys, of one upstream request
if strings.Contains(path, "#") {
return nil, fmt.Errorf(
`%w: the source path or query holds a "#"`, ErrInvalidPath)
}
// Extract query string if present in path. It is the source URL's query
// as that URL writes it. A character a URL's query cannot hold as it is,
// such as a space written %20 in the image URL, is percent-encoded here,
// so the query is signed and cached as it is sent upstream
if qIndex := strings.Index(path, "?"); qIndex != -1 {
query = path[qIndex+1:]
query = escapeQuery(path[qIndex+1:])
path = path[:qIndex]
}
@@ -243,6 +279,20 @@ func decodePercentEscapes(s string) string {
})
}
// escapeQuery returns a source query as a URL's query holds it: each
// character that queryEscapeRegex names, such as a space, percent-encoded,
// and the rest, the query's percent-escapes included, kept as they are. The
// result is unchanged by a second escapeQuery.
func escapeQuery(query string) string {
return queryEscapeRegex.ReplaceAllStringFunc(query, func(match string) string {
if percentEscapeRegex.MatchString(match) {
return match
}
return url.PathEscape(match)
})
}
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
func parseSizeFormat(s string) (Size, ImageFormat, error) {
matches := sizeFormatRegex.FindStringSubmatch(s)
+371
View File
@@ -0,0 +1,371 @@
package server
import (
"net/http"
"net/http/httptest"
"net/url"
"regexp"
"slices"
"strings"
"sync"
"testing"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/session"
)
// percentSignPath is a source path holding a "%", which a URL writes %25.
const percentSignPath = "/100%25.png"
// generateFormPattern extracts the CSRF token rendered into the generator
// page's form.
var generateFormPattern = regexp.MustCompile(
`(?s)action="/generate".*?name="gorilla\.csrf\.Token" value="([^"]+)"`)
// generatedURLPattern extracts the path of the encrypted URL the generator
// page shows.
var generatedURLPattern = regexp.MustCompile(`value="[^"]*(/v1/e/[^"]+)"`)
// recordingUpstream is a test upstream that answers every request with a PNG
// and records the request target, the path and query as sent, of each request
// it receives.
type recordingUpstream struct {
*httptest.Server
mu sync.Mutex
received []string
}
// startRecordingUpstream starts a recordingUpstream, which is closed when the
// test ends.
func startRecordingUpstream(t *testing.T) *recordingUpstream {
t.Helper()
source := encodeTestPNG(t, 64, 48)
upstream := &recordingUpstream{}
upstream.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
upstream.mu.Lock()
upstream.received = append(upstream.received, r.RequestURI)
upstream.mu.Unlock()
w.Header().Set("Content-Type", "image/png")
_, _ = w.Write(source)
}))
t.Cleanup(upstream.Close)
return upstream
}
// requestTargets returns the request targets the upstream has received, in
// the order it received them.
func (u *recordingUpstream) requestTargets() []string {
u.mu.Lock()
defer u.mu.Unlock()
return slices.Clone(u.received)
}
// pageOrigin is the origin of the login and generator pages below, opened
// over HTTPS; httptest gives every request the host example.com.
const pageOrigin = "https://example.com"
// openRoot sends GET / to s over HTTPS, with cookie when it is not nil, and
// returns the page: the login page, or the generator page for a login session.
func openRoot(
t *testing.T, s *Server, cookie *http.Cookie,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet,
pageOrigin+"/", nil)
if cookie != nil {
req.AddCookie(cookie)
}
page := httptest.NewRecorder()
s.ServeHTTP(page, req)
return page
}
// submitForm posts form to path on s as a browser submits it from page over
// HTTPS: with the CSRF token tokenPattern finds in page, the cookies page set,
// cookie when it is not nil, and a Referer naming the page, which pixa needs
// while debug is off, as it is in startImageProxy's config.
func submitForm(
t *testing.T, s *Server, page *httptest.ResponseRecorder,
tokenPattern *regexp.Regexp, path string, form url.Values,
cookie *http.Cookie,
) *httptest.ResponseRecorder {
t.Helper()
match := tokenPattern.FindStringSubmatch(page.Body.String())
if match == nil {
t.Fatalf("the page has no form for %s with a CSRF token", path)
}
form.Set("gorilla.csrf.Token", match[1])
req := httptest.NewRequestWithContext(t.Context(), http.MethodPost,
pageOrigin+path, strings.NewReader(form.Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("Referer", pageOrigin+"/")
for _, c := range page.Result().Cookies() {
req.AddCookie(c)
}
if cookie != nil {
req.AddCookie(cookie)
}
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
return rec
}
// postGenerate logs in to s with the signing key, submits the generator page's
// form with sourceURL, and returns the response.
func postGenerate(
t *testing.T, s *Server, sourceURL string,
) *httptest.ResponseRecorder {
t.Helper()
login := submitForm(t, s, openRoot(t, s, nil), csrfFieldPattern, "/",
url.Values{"key": {testSigningKey}}, nil)
var sessionCookie *http.Cookie
for _, c := range login.Result().Cookies() {
if c.Name == session.CookieName {
sessionCookie = c
}
}
if sessionCookie == nil {
t.Fatalf("logging in answered %d and set no session cookie", login.Code)
}
rec := submitForm(t, s, openRoot(t, s, sessionCookie), generateFormPattern,
"/generate", url.Values{"url": {sourceURL}}, sessionCookie)
t.Logf("POST /generate with url %s: %d", sourceURL, rec.Code)
return rec
}
// TestImageURLFetchesItsSourcePathAndQuery requests image URLs whose source
// query holds a space, written percent-encoded, or a "'", and whose source
// path holds a "%", written percent-encoded, and checks the one request the
// upstream receives: the source path percent-encoded again, and the source
// query with the space written %20 and the "'" written %27, as a browser
// writes them in a query.
func TestImageURLFetchesItsSourcePathAndQuery(t *testing.T) {
t.Parallel()
tests := []struct {
name string
// pathAndQuery is the source path and query as the image URL writes
// them
pathAndQuery string
// wantRequest is the path and query the upstream must receive
wantRequest string
}{
{"space in the query", "/photo.png%3Fname=my%20cat",
"/photo.png?name=my%20cat"},
{"apostrophe in the query", "/photo.png%3Fname=it's",
"/photo.png?name=it%27s"},
{"percent sign in the path", percentSignPath, percentSignPath},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
getImage(t, s, "/v1/image/"+upstreamHost+tt.pathAndQuery+"/orig.png")
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.wantRequest}) {
t.Errorf("upstream received %q, want [%q]", got, tt.wantRequest)
}
})
}
}
// TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart requests twice each
// of the source paths /my cat.png and /my%20cat.png, which the image URL
// writes my%20cat.png and my%2520cat.png. Each is fetched once, as a request
// of its own, and its second request is served from the cache.
func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
imageURLs := []string{
"/v1/image/" + upstreamHost + "/my%20cat.png/orig.png",
"/v1/image/" + upstreamHost + "/my%2520cat.png/orig.png",
}
for _, imageURL := range imageURLs {
getImage(t, s, imageURL)
}
for _, imageURL := range imageURLs {
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) {
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
imageURL, cacheStatus)
}
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
want := []string{"/my%20cat.png", "/my%2520cat.png"}
if !slices.Equal(got, want) {
t.Errorf("upstream received %q, want %q", got, want)
}
}
// TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest requests the source query
// name=my%20cat written as the README says, name=my%2520cat, and with the
// space written %20, name=my%20cat. Both are one source query: the upstream
// receives one request, and the second image URL is served from the cache.
func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
getImage(t, s, "/v1/image/"+upstreamHost+"/photo.png%3Fname=my%2520cat/orig.png")
second := getImage(t, s,
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
want := []string{"/photo.png?name=my%20cat"}
if !slices.Equal(got, want) {
t.Errorf("upstream received %q, want %q", got, want)
}
}
// TestGeneratedURLFetchesTheSourceURLAsEntered makes an encrypted URL on the
// generator page for source URLs holding %20, %25 and %23 in the path or the
// query, requests it, and checks that the upstream receives the path and
// query exactly as entered.
func TestGeneratedURLFetchesTheSourceURLAsEntered(t *testing.T) {
t.Parallel()
// The names hold no "%": each test's state directory is named after it
// and goes into the database URL, which decodes percent-escapes
tests := []struct {
name string
pathAndQuery string
}{
{"space in the path", "/my%20cat.png"},
{"percent sign in the path", percentSignPath},
{"number sign in the path", "/a%23b.png"},
{"all three in the query", "/photo.png?name=my%20cat%25%23"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
upstream := startRecordingUpstream(t)
s, _, _ := startImageProxy(t, upstream.Server)
rec := postGenerate(t, s, "https://"+upstreamHost+tt.pathAndQuery)
match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
if rec.Code != http.StatusOK || match == nil {
t.Fatalf("POST /generate: %d with no encrypted URL; body %s",
rec.Code, rec.Body.String())
}
getImage(t, s, match[1])
got := upstream.requestTargets()
t.Logf("upstream received %q", got)
if !slices.Equal(got, []string{tt.pathAndQuery}) {
t.Errorf("upstream received %q, want [%q]", got, tt.pathAndQuery)
}
})
}
}
// TestGeneratorRefusesSourceURLItCannotFetchAsWritten submits source URLs the
// generator page must make no encrypted URL for, and checks that it answers
// 400 and shows why. pixa keeps the source path decoded and fetches it
// percent-encoded again, so a path holding %2F would be fetched with a "/"
// there, whatever else the path holds. pixa sends no user name or password,
// and no "?" with no query after it. For these the page shows the URL pixa
// would fetch. A source path holding a ".." segment is refused, as
// /v1/image/ refuses it.
func TestGeneratorRefusesSourceURLItCannotFetchAsWritten(t *testing.T) {
t.Parallel()
// The test config sets allow_http, so pixa would fetch over plain HTTP
const fetchedFrom = "http://" + upstreamHost
tests := []struct {
name string
// sourceURL is the source URL after "https://"
sourceURL string
// wantShown is what the page must show: the URL pixa would fetch, or
// why it makes no encrypted URL
wantShown string
}{
{"encoded slash", upstreamHost + "/a%2Fb.png",
fetchedFrom + "/a/b.png"},
{"encoded slash and a space", upstreamHost + "/a%2Fb c.png",
fetchedFrom + "/a/b%20c.png"},
{"encoded slash and a bar", upstreamHost + "/a%2Fb|c.png",
fetchedFrom + "/a/b%7Cc.png"},
{"encoded slash and a letter outside ASCII", upstreamHost + "/café%2Fx.png",
fetchedFrom + "/caf%C3%A9/x.png"},
{"user name and password", "user:pass@" + upstreamHost + "/a.png",
fetchedFrom + "/a.png"},
{"question mark with no query", upstreamHost + "/a.png?",
fetchedFrom + "/a.png"},
{"dot dot segment", upstreamHost + "/a/../b.png",
"cannot hold a .. segment"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
s, _, _ := startImageProxy(t, startRecordingUpstream(t).Server)
rec := postGenerate(t, s, "https://"+tt.sourceURL)
body := rec.Body.String()
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
if strings.Contains(body, "/v1/e/") {
t.Error("the page shows an encrypted URL")
}
if !strings.Contains(body, tt.wantShown) {
t.Errorf("the page does not show %s; body %s", tt.wantShown, body)
}
})
}
}