Compare commits
10
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a9e4d76b5f | ||
|
|
cead3d57bf | ||
|
|
b12cb4565c | ||
|
|
80e336c937 | ||
|
|
64d96aab87 | ||
|
|
e8fdd49c80 | ||
|
|
5b89f928d7 | ||
|
|
ebdecf9aba | ||
|
|
de7ecfdb5e | ||
|
|
ae7564b769 |
@@ -316,32 +316,12 @@ once, is refused with 400.
|
||||
both optional (values under Signature Specification). Both are part of what is
|
||||
cached, so each value of either is a separate cached image.
|
||||
|
||||
A source URL's query follows its path after `%3F`, the encoded `?`:
|
||||
`/v1/image/<host>/<path>%3F<query>/<size>.<format>`. pixa decodes the
|
||||
percent-escapes in the image URL's path once, then reads the host up to the
|
||||
first `/`, the size and format after the last `/`, and between them the source
|
||||
path up to the first `?` and the source query after it. So the `?` before the
|
||||
query, any `?` in the query, and every `%` in the source path or query must be
|
||||
percent-encoded, as `%3F` and `%25`. Any other character may be written as
|
||||
itself or percent-encoded: `/`, `;`, `,`, `=` and `&` work either way. For
|
||||
example,
|
||||
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg` at
|
||||
800x600 WebP is
|
||||
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp`,
|
||||
signed as shown under Signature Specification.
|
||||
|
||||
A size with one side `0` has that side scaled to keep the source's proportions,
|
||||
and is refused with 400 when that side would come out above 8192, the most
|
||||
either side may be; so is a size with `fit=outside`, which scales the image to
|
||||
cover the size and so past it on one side, when that side would come out
|
||||
above 8192.
|
||||
|
||||
An AVIF image of more than 16,777,216 pixels, the number 4096x4096 and 8192x2048
|
||||
have, is refused with 400 before anything is decoded, as its encode could take
|
||||
more than 40 seconds (see below). That includes `orig` of an AVIF source, and
|
||||
`auto` when it chooses AVIF, so a client whose `Accept` names `image/avif` but
|
||||
not `image/jxl` is refused such a size.
|
||||
|
||||
The source image may be JPEG, PNG, GIF, WebP, AVIF or JPEG XL. The upstream's
|
||||
`Content-Type` must name its format, and the image's first bytes must match it.
|
||||
An upstream answering with any other `Content-Type`, SVG included, is refused
|
||||
@@ -391,18 +371,14 @@ transcode that stops this way is answered 503 with the error
|
||||
ended. govips gives pixa no way to stop libvips once an encode has begun, so an
|
||||
encode keeps its slot until it finishes, which can be after
|
||||
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
||||
output, and for AVIF the limit of 16,777,216 pixels, bound how long that takes.
|
||||
With one libvips thread, as pixad runs it, encoding 8192x8192 random pixels
|
||||
takes up to about 1 second as JPEG, 15 as PNG and 16 as GIF, the slowest of 8 or
|
||||
16 bits per sample, with or without an alpha channel. The WebP and JPEG XL
|
||||
encoders run below libvips' default effort, and AVIF at the lowest effort govips
|
||||
can set, so that no encode takes more than 40 seconds of the default
|
||||
`downstream_timeout` of 60: at any quality and 8 or 16 bits per sample,
|
||||
8192x8192 random pixels with an alpha channel take up to about 23 seconds as
|
||||
WebP and 27 as JPEG XL, and 4096x4096, 8192x2048 or 2048x8192 up to about 34 as
|
||||
AVIF, which at quality 85 would take about 96 at 8192x8192. Without an alpha
|
||||
channel they take up to about 16, 13 and 22 seconds. Decoding such a source adds
|
||||
up to about 11 seconds.
|
||||
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
||||
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
||||
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16
|
||||
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are
|
||||
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28
|
||||
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or
|
||||
JPEG XL encode can therefore hold its slot past the default `downstream_timeout`
|
||||
of 60 seconds.
|
||||
|
||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||
@@ -454,8 +430,7 @@ nor change what it asks for.
|
||||
sent with an empty format, or none, also makes a JPEG XL URL. Width and
|
||||
height both empty or `0` keep the original size; if only one of them is empty
|
||||
or `0`, that side is scaled to keep the image's proportions. A URL whose
|
||||
scaled side would come out above 8192 answers 400, and so does an AVIF URL of
|
||||
more than 16,777,216 pixels (see Routes).
|
||||
scaled side would come out above 8192 answers 400.
|
||||
3. The page shows the URL, `https://<host>/v1/e/<token>/img.<format>`, and when
|
||||
it expires. `<host>` is the host the page was opened on, and the URL starts
|
||||
with `http` instead while `debug` is on. The name after the token is ignored
|
||||
@@ -560,17 +535,6 @@ For the same image at quality 40 with fit `contain`, the input ends in
|
||||
and the URL is
|
||||
`/v1/image/cdn.example.com/photos/cat.jpg/800x600.webp?sig=5IwXUx6vf7yefhaUvFzgXZvG2o0Df4RJxPTK3pKq5VU=&exp=1704067200&q=40&fit=contain`.
|
||||
|
||||
For a source URL with a query,
|
||||
`https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg`, at
|
||||
800x600 WebP with the same key and expiration and the default quality and fit:
|
||||
|
||||
1. Build input, with the query's `:` written as `%3A` and its `%` as `%25`:
|
||||
`cdn.example.com:/img:url=https%3A//origin.example.com/my%2520cat.jpg:800:600:webp:1704067200:85:cover`
|
||||
2. The signature is `37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=`
|
||||
3. URL, with the `?` before the query written as `%3F` and the query's `%` as
|
||||
`%25` (see Routes):
|
||||
`/v1/image/cdn.example.com/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp?sig=37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA=&exp=1704067200`
|
||||
|
||||
**Allowlist patterns:**
|
||||
|
||||
- **Exact match**: `cdn.example.com` — matches only that host
|
||||
|
||||
@@ -30,32 +30,6 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-15 a WebP, AVIF or JPEG XL encode at the limits takes at most about 34
|
||||
seconds (closes #272): with one libvips thread, 8192x8192 random pixels with
|
||||
an alpha channel took about 61 seconds to encode as WebP, 96 as AVIF and 81 as
|
||||
JPEG XL, so with the default `downstream_timeout` of 60 seconds such an image
|
||||
was never delivered, yet its encode held a processing slot to the end. WebP is
|
||||
now saved at effort 2 and JPEG XL at effort 4, which take up to about 23 and
|
||||
27 seconds. AVIF was already at the lowest effort govips can set, so an AVIF
|
||||
output of more than 16,777,216 pixels is refused with 400 before anything is
|
||||
decoded; 4096x4096, 8192x2048 or 2048x8192 takes up to about 34 seconds. Each
|
||||
time is the slowest at any quality. Resizing with `fit=cover` already decodes
|
||||
the input, so the output's dimensions are worked out before resizing.
|
||||
`README.md` gives the limit and the new times.
|
||||
- 2026-10-15 the README says how a source URL with a query is written into an
|
||||
image URL, and the image route accepts it (closes #255): the query follows the
|
||||
source path after `%3F`. pixa now reads the image URL's path decoded once, so
|
||||
the `?` before the query, any `?` in it and every `%` in the source path or
|
||||
query must be percent-encoded, and any other character may be. The route used
|
||||
to parse chi's wildcard, which holds the path as the client wrote it whenever
|
||||
that is not Go's own encoding of it; a `/`, `;` or `,` in the query written as
|
||||
`%2F`, `%3B` or `%2C`, as `GenerateSignedURL` wrote them, kept the `%3F` from
|
||||
being read as `?`, and the request failed with 401. `GenerateSignedURL` now
|
||||
writes Go's own encoding of the path. The README has an example with a query
|
||||
and its signature. The path traversal check no longer reads the source query,
|
||||
and keeps a `%` that does not start a percent-escape as it is instead of
|
||||
refusing the URL, so a source path or query holding `%25` is accepted, while a
|
||||
double-encoded `..` next to such a `%` is still refused.
|
||||
- 2026-10-15 `max_concurrent_processing` defaults to half the CPUs pixa sees,
|
||||
rounded down, and at least 1 (closes #242): it defaulted to one image per CPU,
|
||||
and a container with no CPU limit sees every CPU of the host, so on a 48-CPU
|
||||
|
||||
@@ -1,49 +0,0 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/imgcache"
|
||||
)
|
||||
|
||||
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
|
||||
// handleImageError, which answer the errors of the two image routes, answer
|
||||
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
|
||||
// output above it, wrapped as the service wraps it. The request's source only
|
||||
// goes into a log line, so it is left empty.
|
||||
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
err := fmt.Errorf("image processing failed: %w: 4097x4096",
|
||||
imageprocessor.ErrAVIFOutputTooLarge)
|
||||
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
|
||||
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, &imgcache.ImageRequest{}, err)
|
||||
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
for name, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"respondImageError": plain, "handleImageError": encrypted,
|
||||
} {
|
||||
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status = %d, want %d",
|
||||
name, rec.Code, http.StatusBadRequest)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), wantBody) {
|
||||
t.Errorf("%s: body = %q, want it to say %q",
|
||||
name, rec.Body.String(), wantBody)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"sneak.berlin/go/pixa/internal/encurl"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
@@ -79,21 +80,20 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
|
||||
}
|
||||
}
|
||||
|
||||
// parseImageRequest parses the image URL's path and query parameters into
|
||||
// parseImageRequest parses the wildcard path and query parameters into
|
||||
// an ImageRequest. On invalid input it writes an error response and
|
||||
// returns false.
|
||||
func (s *Handlers) parseImageRequest(
|
||||
w http.ResponseWriter, r *http.Request,
|
||||
) (*imgcache.ImageRequest, bool) {
|
||||
// The path is parsed decoded once, as r.URL.Path holds it, and not from
|
||||
// chi's wildcard: chi routes on the path as the client wrote it whenever
|
||||
// that is not Go's own encoding of it, so its wildcard would still hold
|
||||
// the %3F before a source query when, say, a "/" in the query is written
|
||||
// as %2F
|
||||
parsed, err := imgcache.ParseImageURL(r.URL.Path)
|
||||
// Get the wildcard path from chi
|
||||
pathParam := chi.URLParam(r, "*")
|
||||
|
||||
// Parse the URL path
|
||||
parsed, err := imgcache.ParseImagePath(pathParam)
|
||||
if err != nil {
|
||||
s.log.Warn("failed to parse image URL",
|
||||
"path", r.URL.Path,
|
||||
"path", pathParam,
|
||||
"error", err,
|
||||
)
|
||||
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
|
||||
@@ -264,14 +264,6 @@ func (s *Handlers) respondImageError(
|
||||
return
|
||||
}
|
||||
|
||||
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
}
|
||||
|
||||
|
||||
@@ -169,166 +169,3 @@ func TestHandleImage_InvalidExp_Returns400(t *testing.T) {
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies sends URLs built
|
||||
// by the service's signed-URL generator for source URLs with a query to the
|
||||
// route, through a chi router: chi routes on the path as the client wrote it
|
||||
// whenever that is not Go's own encoding of it. A 502 means the signature
|
||||
// passed, so the route read the source path and query that were signed.
|
||||
func TestHandleImage_GeneratedSignedURLWithSourceQueryVerifies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
query string
|
||||
}{
|
||||
{"slash and colon", "url=https://origin.example.com/cat.jpg"},
|
||||
{"semicolon", "a=1;b=2"},
|
||||
{"comma", "crop=0,0,50,50"},
|
||||
{"equals and ampersand", "a=1&b=2"},
|
||||
{"percent-escape", "name=my%20cat"},
|
||||
{"question mark", "next=/a?b=1"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
SourceQuery: tt.query,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
status := getImage(t, fix, signedURL)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_SourceQueryEncodedAnyWay signs one source URL with a query
|
||||
// and writes it into the image URL in ways the README allows: the "?" before
|
||||
// the query and each "%" in it percent-encoded, and every other character as
|
||||
// itself or percent-encoded. A 502 means the signature passed.
|
||||
func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
expires := time.Now().Add(time.Hour)
|
||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||
sig := signer.Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: photoPath,
|
||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
writtenQuery string
|
||||
}{
|
||||
{"only ? and % encoded",
|
||||
"%3Furl=https://origin.example.com/my%2520cat.jpg"},
|
||||
{"every reserved character encoded",
|
||||
"%3Furl%3Dhttps%3A%2F%2Forigin.example.com%2Fmy%2520cat.jpg"},
|
||||
{"lower-case hex digits",
|
||||
"%3furl=https://origin.example.com/my%2520cat.jpg"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
target := fmt.Sprintf("/v1/image/%s%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, photoPath, tt.writtenQuery, sig, expires.Unix())
|
||||
|
||||
status := getImage(t, setupTestHandler(t), target)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleImage_SourceURLHoldingPercent25Verifies signs a source path and a
|
||||
// source query that each hold %25, the percent-escape of a literal "%", and
|
||||
// sends each to the route written as the README says, with that "%" written as
|
||||
// %25, and as GenerateSignedURL writes it. A 502 means the signature passed.
|
||||
func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
sourcePath string
|
||||
sourceQuery string
|
||||
// writtenPathAndQuery is the source path and query as the README says
|
||||
// to write them into the image URL
|
||||
writtenPathAndQuery string
|
||||
}{
|
||||
{"path", "/sale/50%25off.jpg", "", "/sale/50%2525off.jpg"},
|
||||
{"query", photoPath, "discount=50%25", photoPath + "%3Fdiscount=50%2525"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name+" written by the README rule", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
expires := time.Now().Add(time.Hour)
|
||||
signer := signature.New("test-signing-key-must-be-32-chars")
|
||||
sig := signer.Sign(&signature.Request{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: tt.sourcePath,
|
||||
SourceQuery: tt.sourceQuery,
|
||||
Width: 50,
|
||||
Height: 50,
|
||||
Format: string(imgcache.FormatJPEG),
|
||||
Quality: 85,
|
||||
FitMode: string(imgcache.FitCover),
|
||||
Expires: expires,
|
||||
})
|
||||
|
||||
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
|
||||
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
|
||||
|
||||
status := getImage(t, setupTestHandler(t), target)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run(tt.name+" written by GenerateSignedURL", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
fix := setupTestHandler(t)
|
||||
|
||||
signedURL, err := fix.service.GenerateSignedURL("", &imgcache.ImageRequest{
|
||||
SourceHost: signedHost,
|
||||
SourcePath: tt.sourcePath,
|
||||
SourceQuery: tt.sourceQuery,
|
||||
Size: imgcache.Size{Width: 50, Height: 50},
|
||||
Format: imgcache.FormatJPEG,
|
||||
}, time.Hour)
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateSignedURL() error = %v", err)
|
||||
}
|
||||
|
||||
status := getImage(t, fix, signedURL)
|
||||
if status != http.StatusBadGateway {
|
||||
t.Errorf("status = %d, want %d", status, http.StatusBadGateway)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,10 +170,6 @@ func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
|
||||
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
|
||||
"computed from its proportions is above %d",
|
||||
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
|
||||
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
|
||||
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
|
||||
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
|
||||
http.StatusBadRequest)
|
||||
default:
|
||||
s.log.Error("image request failed", "error", err)
|
||||
s.respondError(w, "internal error", http.StatusInternalServerError)
|
||||
|
||||
@@ -1,64 +0,0 @@
|
||||
package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestImageProcessor_RefusesAVIFAboveMaxPixels verifies that an AVIF output
|
||||
// of more than MaxAVIFOutputPixels is refused with ErrAVIFOutputTooLarge
|
||||
// before anything is decoded, whatever size and fit give it, and that an AVIF
|
||||
// output of at most that many pixels, or one in another format, is not. The
|
||||
// input is a 64x32 PNG cut off in its pixel data: libvips reads its size from
|
||||
// its header but fails when it decodes it, so a request that gets that failure
|
||||
// was not refused, and a refusal came before any decoding.
|
||||
func TestImageProcessor_RefusesAVIFAboveMaxPixels(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
whole := createTestPNG(t, 64, 32)
|
||||
cutOff := whole[:len(whole)/2]
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
format Format
|
||||
width, height int
|
||||
fit FitMode
|
||||
refused bool
|
||||
}{
|
||||
{"AVIF at the limit", FormatAVIF, 4096, 4096, FitCover, false},
|
||||
{"AVIF above the limit", FormatAVIF, 4096, 4097, FitCover, true},
|
||||
{"WebP above the AVIF limit", FormatWebP, 4096, 4097, FitCover, false},
|
||||
// FitInside does not enlarge, so the output stays 64x32
|
||||
{"AVIF inside a larger size", FormatAVIF, 8192, 8192, FitInside, false},
|
||||
// FitOutside scales 64x32 by 128 to cover 8192x1: 8192x4096
|
||||
{"AVIF outside a smaller size", FormatAVIF, 8192, 1, FitOutside, true},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
_, err := New(Params{}).Process(t.Context(), bytes.NewReader(cutOff),
|
||||
&Request{
|
||||
Size: Size{Width: tt.width, Height: tt.height},
|
||||
Format: tt.format,
|
||||
FitMode: tt.fit,
|
||||
})
|
||||
t.Logf("Process() error = %v", err)
|
||||
|
||||
if tt.refused {
|
||||
if !errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||
t.Errorf("Process() error = %v, want %v", err, ErrAVIFOutputTooLarge)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if err == nil || errors.Is(err, ErrAVIFOutputTooLarge) {
|
||||
t.Errorf("Process() error = %v, want the failure to decode "+
|
||||
"the cut-off input", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -77,10 +77,10 @@ func TestImageProcessor_PNGIsCompressed(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_WebPAtEffort2 verifies that WebP is saved at effort 2,
|
||||
// not libvips' default of 4: the output is the size of the same image saved
|
||||
// at effort 2.
|
||||
func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
// TestImageProcessor_WebPAtDefaultEffort verifies that WebP is saved at
|
||||
// libvips' default effort, 4: the output is the size of the same image saved
|
||||
// at that effort.
|
||||
func TestImageProcessor_WebPAtDefaultEffort(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
@@ -90,14 +90,14 @@ func TestImageProcessor_WebPAtEffort2(t *testing.T) {
|
||||
want, _, err := decode(t, input).ExportWebp(&vips.WebpExportParams{
|
||||
StripMetadata: true,
|
||||
Quality: 85,
|
||||
ReductionEffort: 2,
|
||||
ReductionEffort: 4,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ExportWebp() error = %v", err)
|
||||
}
|
||||
|
||||
if size != int64(len(want)) {
|
||||
t.Errorf("WebP output is %d bytes, want %d, the size at effort 2",
|
||||
t.Errorf("WebP output is %d bytes, want %d, the size at effort 4",
|
||||
size, len(want))
|
||||
}
|
||||
}
|
||||
@@ -143,29 +143,3 @@ func TestImageProcessor_AVIFAtEffort1(t *testing.T) {
|
||||
"and 8 bits", size, len(want))
|
||||
}
|
||||
}
|
||||
|
||||
// TestImageProcessor_JPEGXLAtEffort4 verifies that JPEG XL is saved at
|
||||
// effort 4, not libvips' default of 7: the output is the size of the same
|
||||
// image saved at effort 4, with the distance exportJXL gives quality 85.
|
||||
func TestImageProcessor_JPEGXLAtEffort4(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, 200, 150)
|
||||
|
||||
size := processedSize(t, input, &Request{Format: FormatJXL, Quality: 85})
|
||||
|
||||
params := vips.NewJxlExportParams()
|
||||
params.Quality = 85
|
||||
params.Distance = jxlDistance(85)
|
||||
params.Effort = 4
|
||||
|
||||
want, _, err := decode(t, input).ExportJxl(params)
|
||||
if err != nil {
|
||||
t.Fatalf("ExportJxl() error = %v", err)
|
||||
}
|
||||
|
||||
if size != int64(len(want)) {
|
||||
t.Errorf("JPEG XL output is %d bytes, want %d, the size at effort 4",
|
||||
size, len(want))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -142,19 +142,6 @@ const MaxOutputDimension = 8192
|
||||
// requested size.
|
||||
var ErrOutputTooLarge = errors.New("output image dimensions exceed maximum")
|
||||
|
||||
// MaxAVIFOutputPixels is the most pixels an AVIF output may have, so that an
|
||||
// encode takes at most 40 seconds of the default downstream_timeout of 60
|
||||
// seconds; AVIF is already at the lowest effort govips can set (avifEffort).
|
||||
// With one thread, as pixad runs libvips, saving 4096x4096, 8192x2048 or
|
||||
// 2048x8192 random pixels with an alpha channel takes up to about 34
|
||||
// seconds, the slowest at any quality and at 8 or 16 bits per sample, and
|
||||
// 5120x4096 at 8 bits already takes about 36.
|
||||
const MaxAVIFOutputPixels = 4096 * 4096
|
||||
|
||||
// ErrAVIFOutputTooLarge is returned, before anything is decoded, when an AVIF
|
||||
// output would have more than MaxAVIFOutputPixels pixels.
|
||||
var ErrAVIFOutputTooLarge = errors.New("AVIF output has too many pixels")
|
||||
|
||||
// ErrUnsupportedOutputFormat is returned when the requested output format is
|
||||
// not supported.
|
||||
var ErrUnsupportedOutputFormat = errors.New("unsupported output format")
|
||||
@@ -223,8 +210,7 @@ func New(params Params) *ImageProcessor {
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
// encodes, it returns ctx's error and gives the slot back. govips gives pixa
|
||||
// no way to stop libvips once an encode has begun, so an encode keeps its
|
||||
// slot until it finishes. An AVIF output of more than MaxAVIFOutputPixels is
|
||||
// refused with ErrAVIFOutputTooLarge before anything is decoded.
|
||||
// slot until it finishes.
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
@@ -237,9 +223,18 @@ func (p *ImageProcessor) Process(
|
||||
|
||||
defer release()
|
||||
|
||||
data, err := p.readInput(input)
|
||||
// Read input with a size limit to prevent unbounded memory consumption.
|
||||
// We read at most maxInputBytes+1 so we can detect if the input exceeds
|
||||
// the limit without consuming additional memory.
|
||||
limited := io.LimitReader(input, p.maxInputBytes+1)
|
||||
|
||||
data, err := io.ReadAll(limited)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("failed to read input: %w", err)
|
||||
}
|
||||
|
||||
if int64(len(data)) > p.maxInputBytes {
|
||||
return nil, ErrInputDataTooLarge
|
||||
}
|
||||
|
||||
// govips picks the libvips loader from the input's first bytes with
|
||||
@@ -282,28 +277,20 @@ func (p *ImageProcessor) Process(
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if targetWidth != origWidth || targetHeight != origHeight {
|
||||
err := p.resize(img, targetWidth, targetHeight, req.FitMode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// orig is the source's own format; encode refuses an empty format
|
||||
outputFormat := req.Format
|
||||
if outputFormat == FormatOriginal {
|
||||
outputFormat = sourceFormat
|
||||
}
|
||||
|
||||
// Determine output dimensions; too large ones are refused before
|
||||
// resizing, which with FitCover already decodes the input
|
||||
outputWidth, outputHeight, err := outputDimensions(origWidth, origHeight,
|
||||
targetWidth, targetHeight, req.FitMode, outputFormat)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if targetWidth != origWidth || targetHeight != origHeight {
|
||||
err := p.resize(img, outputWidth, outputHeight, req.FitMode)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to resize: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Encode to target format
|
||||
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
||||
if err != nil {
|
||||
@@ -456,23 +443,6 @@ func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// readInput reads input with a size limit to prevent unbounded memory
|
||||
// consumption, refusing more than maxInputBytes with ErrInputDataTooLarge. It
|
||||
// reads at most maxInputBytes+1 so it can detect if the input exceeds the
|
||||
// limit without consuming additional memory.
|
||||
func (p *ImageProcessor) readInput(input io.Reader) ([]byte, error) {
|
||||
data, err := io.ReadAll(io.LimitReader(input, p.maxInputBytes+1))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read input: %w", err)
|
||||
}
|
||||
|
||||
if int64(len(data)) > p.maxInputBytes {
|
||||
return nil, ErrInputDataTooLarge
|
||||
}
|
||||
|
||||
return data, nil
|
||||
}
|
||||
|
||||
// detectFormat returns the format string for a govips image type.
|
||||
func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
switch imageType {
|
||||
@@ -497,61 +467,9 @@ func (p *ImageProcessor) detectFormat(imageType vips.ImageType) string {
|
||||
}
|
||||
}
|
||||
|
||||
// outputDimensions returns the dimensions resize gives an image of imgWidth x
|
||||
// imgHeight for the target dimensions and fit mode, without resizing it, and
|
||||
// refuses an output too large for format. With FitOutside one side can come
|
||||
// out larger than the target; when it would be above MaxOutputDimension,
|
||||
// outputDimensions returns ErrOutputTooLarge. An AVIF output of more than
|
||||
// MaxAVIFOutputPixels is ErrAVIFOutputTooLarge.
|
||||
func outputDimensions(
|
||||
imgWidth, imgHeight, width, height int, fit FitMode, format Format,
|
||||
) (int, int, error) {
|
||||
scaleW := float64(width) / float64(imgWidth)
|
||||
scaleH := float64(height) / float64(imgHeight)
|
||||
outputWidth, outputHeight := width, height
|
||||
|
||||
switch fit {
|
||||
case FitCover, FitFill, "":
|
||||
// Cover crops, and fill stretches, to exactly the target dimensions
|
||||
|
||||
case FitContain:
|
||||
// Fit within the target dimensions, maintaining aspect ratio
|
||||
scale := min(scaleW, scaleH)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
case FitInside:
|
||||
// Same as contain, but only shrink
|
||||
scale := min(scaleW, scaleH, 1)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
case FitOutside:
|
||||
// The smallest dimension fits, the other may exceed the target
|
||||
scale := max(scaleW, scaleH)
|
||||
outputWidth = int(float64(imgWidth) * scale)
|
||||
outputHeight = int(float64(imgHeight) * scale)
|
||||
|
||||
default:
|
||||
return 0, 0, fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
}
|
||||
|
||||
// The side FitOutside scales past the target is held to the maximum too
|
||||
if max(outputWidth, outputHeight) > MaxOutputDimension {
|
||||
return 0, 0, fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
||||
outputWidth, outputHeight, MaxOutputDimension)
|
||||
}
|
||||
|
||||
if format == FormatAVIF && outputWidth*outputHeight > MaxAVIFOutputPixels {
|
||||
return 0, 0, fmt.Errorf("%w: %dx%d is above %d pixels",
|
||||
ErrAVIFOutputTooLarge, outputWidth, outputHeight, MaxAVIFOutputPixels)
|
||||
}
|
||||
|
||||
return outputWidth, outputHeight, nil
|
||||
}
|
||||
|
||||
// resize resizes the image to width x height, the dimensions
|
||||
// outputDimensions gives for the fit mode.
|
||||
// resize resizes the image according to the fit mode. With FitOutside one
|
||||
// side can come out larger than requested; when it would be above
|
||||
// MaxOutputDimension, resize returns ErrOutputTooLarge without resizing.
|
||||
func (p *ImageProcessor) resize(
|
||||
img *vips.ImageRef, width, height int, fit FitMode,
|
||||
) error {
|
||||
@@ -560,20 +478,52 @@ func (p *ImageProcessor) resize(
|
||||
// Resize and crop to fill exact dimensions (default)
|
||||
return img.Thumbnail(width, height, vips.InterestingCentre)
|
||||
|
||||
case FitContain:
|
||||
// Resize to fit within dimensions, maintaining aspect ratio
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
case FitFill:
|
||||
// Resize to exact dimensions (may distort)
|
||||
return img.ThumbnailWithSize(width, height, vips.InterestingNone, vips.SizeForce)
|
||||
|
||||
case FitInside:
|
||||
// outputDimensions keeps the dimensions of an image that already fits
|
||||
if img.Width() == width && img.Height() == height {
|
||||
return nil
|
||||
// Same as contain, but only shrink
|
||||
if img.Width() <= width && img.Height() <= height {
|
||||
return nil // Already fits
|
||||
}
|
||||
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := min(scaleW, scaleH)
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
case FitContain, FitOutside:
|
||||
return img.Thumbnail(width, height, vips.InterestingNone)
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
case FitOutside:
|
||||
// Resize so smallest dimension fits, may exceed target on other dimension
|
||||
imgW, imgH := img.Width(), img.Height()
|
||||
scaleW := float64(width) / float64(imgW)
|
||||
scaleH := float64(height) / float64(imgH)
|
||||
scale := max(scaleW, scaleH)
|
||||
newW := int(float64(imgW) * scale)
|
||||
newH := int(float64(imgH) * scale)
|
||||
|
||||
// The dimension that exceeds the target is held to the maximum too
|
||||
if newW > MaxOutputDimension || newH > MaxOutputDimension {
|
||||
return fmt.Errorf("%w: %dx%d is above %d", ErrOutputTooLarge,
|
||||
newW, newH, MaxOutputDimension)
|
||||
}
|
||||
|
||||
return img.Thumbnail(newW, newH, vips.InterestingNone)
|
||||
|
||||
default:
|
||||
return fmt.Errorf("%w: %s", ErrInvalidFitMode, fit)
|
||||
@@ -677,16 +627,11 @@ func exportGIF(img *vips.ImageRef) ([]byte, error) {
|
||||
return output, err
|
||||
}
|
||||
|
||||
// webpEffort is the WebP effort, from 0 (fastest) to 6. It is below libvips'
|
||||
// default of 4 so that an encode takes at most 40 seconds of the default
|
||||
// downstream_timeout of 60 seconds. With one thread, as pixad runs libvips,
|
||||
// saving an 8192x8192 image of random pixels with an alpha channel takes up
|
||||
// to about 23 seconds at 2, and up to about 16 without one, at any quality;
|
||||
// with an alpha channel, 3 takes about 50 seconds and 4 about 61.
|
||||
const webpEffort = 2
|
||||
// webpEffort is libvips' default WebP effort, from 0 (fastest) to 6.
|
||||
const webpEffort = 4
|
||||
|
||||
// exportWebP encodes img as lossy WebP at quality and webpEffort, without
|
||||
// metadata.
|
||||
// exportWebP encodes img as lossy WebP at quality and libvips' default
|
||||
// effort, without metadata.
|
||||
func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
output, _, err := img.ExportWebp(&vips.WebpExportParams{
|
||||
StripMetadata: true,
|
||||
@@ -698,13 +643,12 @@ func exportWebP(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
}
|
||||
|
||||
// avifEffort is the AVIF effort, from 0 (fastest) to 9; 1 is the lowest
|
||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 52
|
||||
// seconds to save an 8192x8192 RGB image of random pixels at quality 85,
|
||||
// and about 96 with an alpha channel, against the default
|
||||
// downstream_timeout of 60 seconds, so an AVIF output is held to
|
||||
// MaxAVIFOutputPixels. On an image of milder noise, which 1 saves in about
|
||||
// 12 seconds, 2 takes nearly a minute and libvips' default, 4, takes
|
||||
// minutes.
|
||||
// govips can set. With one thread, as pixad runs libvips, 1 takes about 51
|
||||
// seconds to save an 8192x8192 RGB image of random pixels, and about 110
|
||||
// with an alpha channel, the worst case, against the default
|
||||
// downstream_timeout of 60 seconds. On an image of milder noise, which 1
|
||||
// saves in about 12 seconds, 2 takes nearly a minute and libvips' default,
|
||||
// 4, takes minutes.
|
||||
const avifEffort = 1
|
||||
|
||||
// avifBitdepth is the AVIF bit depth, 8 bits per sample for every image.
|
||||
@@ -731,18 +675,10 @@ func exportAVIF(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
// JPEG that names none.
|
||||
const jxlResolution = 72 / 25.4
|
||||
|
||||
// jxlEffort is the JPEG XL effort, from 1 (fastest) to 9. It is below
|
||||
// libvips' default of 7 so that an encode takes at most 40 seconds of the
|
||||
// default downstream_timeout of 60 seconds. With one thread, as pixad runs
|
||||
// libvips, saving an 8192x8192 image of random pixels with an alpha channel
|
||||
// takes up to about 27 seconds at 4, and up to about 13 without one, at any
|
||||
// quality; with an alpha channel, 5 takes about 43 seconds and 7 about 81.
|
||||
const jxlEffort = 4
|
||||
|
||||
// exportJXL encodes img as JPEG XL at quality and jxlEffort, without
|
||||
// metadata. govips sends libvips a distance, the JPEG XL encoder's own
|
||||
// measure of quality, along with the quality, and libvips then uses the
|
||||
// distance alone, so the quality is also given as a distance.
|
||||
// exportJXL encodes img as JPEG XL at quality, with libvips' default effort
|
||||
// and without metadata. govips sends libvips a distance, the JPEG XL
|
||||
// encoder's own measure of quality, along with the quality, and libvips then
|
||||
// uses the distance alone, so the quality is also given as a distance.
|
||||
func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
// libvips converts a CMYK image to sRGB before it saves WebP, AVIF or
|
||||
// PNG, but cannot save one as JPEG XL. encode has already converted
|
||||
@@ -781,7 +717,6 @@ func exportJXL(img *vips.ImageRef, quality int) ([]byte, error) {
|
||||
params := vips.NewJxlExportParams()
|
||||
params.Quality = quality
|
||||
params.Distance = jxlDistance(quality)
|
||||
params.Effort = jxlEffort
|
||||
|
||||
output, _, err := toSave.ExportJxl(params)
|
||||
if err != nil {
|
||||
|
||||
@@ -47,9 +47,6 @@ func ValidateDimension(name string, value int) error {
|
||||
var sizeFormatRegex = regexp.MustCompile(
|
||||
`^(\d+)x(\d+)(?:\.(\w+))?$|^(orig)(?:\.(\w+))?$`)
|
||||
|
||||
// percentEscapeRegex matches one percent-escape, such as "%2e"
|
||||
var percentEscapeRegex = regexp.MustCompile(`%[0-9A-Fa-f]{2}`)
|
||||
|
||||
// ParsedURL contains the parsed components of an image proxy URL.
|
||||
type ParsedURL struct {
|
||||
// Host is the source origin host (e.g., "cdn.example.com")
|
||||
@@ -64,8 +61,9 @@ type ParsedURL struct {
|
||||
Format ImageFormat
|
||||
}
|
||||
|
||||
// ParseImagePath parses an image URL's path after /v1/image/, decoded once:
|
||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL.
|
||||
// ParseImagePath parses the path captured by chi's wildcard:
|
||||
// <host>/<path>/<size>.<format>, or <host>/<path>/<size> for JPEG XL
|
||||
// This is the primary entry point when using chi routing.
|
||||
// Examples:
|
||||
// - cdn.example.com/photos/cat.jpg/800x600.webp
|
||||
// - cdn.example.com/photos/cat.jpg/0x0.jpeg
|
||||
@@ -81,10 +79,9 @@ func ParseImagePath(path string) (*ParsedURL, error) {
|
||||
return parseImageComponents(path)
|
||||
}
|
||||
|
||||
// ParseImageURL parses an image URL's path, decoded once as r.URL.Path holds
|
||||
// it: /v1/image/<host>/<path>/<size>.<format>, or /v1/image/<host>/<path>/<size>
|
||||
// for JPEG XL. A source query follows the source path after a "?", which the
|
||||
// URL writes as %3F.
|
||||
// ParseImageURL parses a full URL path like /v1/image/<host>/<path>/<size>.<format>,
|
||||
// or /v1/image/<host>/<path>/<size> for JPEG XL
|
||||
// Use ParseImagePath instead when working with chi's wildcard capture.
|
||||
func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||
// Remove the /v1/image/ prefix
|
||||
const prefix = "/v1/image/"
|
||||
@@ -103,6 +100,12 @@ func ParseImageURL(urlPath string) (*ParsedURL, error) {
|
||||
// parseImageComponents parses <host>/<path>/<size>.<format>, or
|
||||
// <host>/<path>/<size> for JPEG XL.
|
||||
func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
// Check for path traversal before any other processing
|
||||
err := checkPathTraversal(remainder)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Find the last path segment, which holds "size" or "size.format"
|
||||
lastSlash := strings.LastIndex(remainder, "/")
|
||||
if lastSlash == -1 {
|
||||
@@ -158,13 +161,6 @@ func parseImageComponents(remainder string) (*ParsedURL, error) {
|
||||
path = path[:qIndex]
|
||||
}
|
||||
|
||||
// The source query names no file on the upstream, so only the host and
|
||||
// the source path are checked for path traversal
|
||||
err = checkPathTraversal(host + path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Ensure path starts with /
|
||||
if !strings.HasPrefix(path, "/") {
|
||||
path = "/" + path
|
||||
@@ -190,7 +186,12 @@ func checkPathTraversal(path string) error {
|
||||
// Decode multiple times to catch double-encoding
|
||||
decoded := path
|
||||
for range 3 {
|
||||
newDecoded := decodePercentEscapes(decoded)
|
||||
newDecoded, err := url.PathUnescape(decoded)
|
||||
if err != nil {
|
||||
// Malformed encoding is suspicious
|
||||
return ErrPathTraversal
|
||||
}
|
||||
|
||||
if newDecoded == decoded {
|
||||
break
|
||||
}
|
||||
@@ -231,18 +232,6 @@ func checkPathTraversal(path string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// decodePercentEscapes decodes every percent-escape in s, such as %2e, and
|
||||
// keeps a "%" that does not start one as it is, where url.PathUnescape would
|
||||
// refuse the whole string. A source path holding %25, a literal "%", has such
|
||||
// a "%" after one more decoding.
|
||||
func decodePercentEscapes(s string) string {
|
||||
return percentEscapeRegex.ReplaceAllStringFunc(s, func(escape string) string {
|
||||
decoded, _ := url.PathUnescape(escape)
|
||||
|
||||
return decoded
|
||||
})
|
||||
}
|
||||
|
||||
// parseSizeFormat parses strings like "800x600.webp", "orig.png" or "800x600"
|
||||
func parseSizeFormat(s string) (Size, ImageFormat, error) {
|
||||
matches := sizeFormatRegex.FindStringSubmatch(s)
|
||||
|
||||
@@ -407,71 +407,6 @@ func TestParseImagePath_PathTraversal(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImageURL_SourceURLHoldingPercent25 verifies that a source path and
|
||||
// a source query holding %25, the percent-escape of a literal "%", are
|
||||
// accepted as they are, although decoding them once more leaves a "%" that
|
||||
// does not start a percent-escape.
|
||||
func TestParseImageURL_SourceURLHoldingPercent25(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
want *ParsedURL
|
||||
}{
|
||||
{
|
||||
name: "path",
|
||||
input: "/v1/image/cdn.example.com/sale/50%25off.jpg/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: "/sale/50%25off.jpg",
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "query",
|
||||
input: "/v1/image/cdn.example.com/photos/cat.jpg?discount=50%25/800x600.webp",
|
||||
want: &ParsedURL{
|
||||
Host: testHostCDN, Path: testPathCat, Query: "discount=50%25",
|
||||
Size: Size{Width: 800, Height: 600}, Format: FormatWebP,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
got, err := ParseImageURL(tt.input)
|
||||
if err != nil {
|
||||
t.Fatalf("ParseImageURL() unexpected error = %v", err)
|
||||
}
|
||||
|
||||
assertParsedURL(t, got, tt.want)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestParseImageURL_DoubleEncodedTraversalNextToPercent verifies that a
|
||||
// double-encoded ".." is still refused when the path also holds a "%" that
|
||||
// does not start a percent-escape, either as written or after one more
|
||||
// decoding.
|
||||
func TestParseImageURL_DoubleEncodedTraversalNextToPercent(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, input := range []string{
|
||||
"/v1/image/cdn.example.com/photos/%252e%252e/secret/50%25off.jpg/800x600.jpeg",
|
||||
"/v1/image/cdn.example.com/photos/%252e%252e/secret/100%/800x600.jpeg",
|
||||
} {
|
||||
_, err := ParseImageURL(input)
|
||||
t.Logf("ParseImageURL(%q) error = %v", input, err)
|
||||
|
||||
if !errors.Is(err, ErrPathTraversal) {
|
||||
t.Errorf("ParseImageURL(%q) error = %v, want ErrPathTraversal",
|
||||
input, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// errorIs checks if err matches target (handles wrapped errors).
|
||||
func errorIs(err, target error) bool {
|
||||
if errors.Is(err, target) {
|
||||
|
||||
@@ -126,20 +126,31 @@ func (s *Signer) GenerateSignedURL(
|
||||
sizeStr = fmt.Sprintf("%dx%d", req.Width, req.Height)
|
||||
}
|
||||
|
||||
// Build the path. A source query follows the source path after a "?".
|
||||
// The image route reads the path decoded once, so the whole path is
|
||||
// written in Go's own percent-encoding of a URL path, which writes "?"
|
||||
// as %3F and "%" as %25, and leaves "/", ";", ",", "=" and "&" as they
|
||||
// are.
|
||||
sourcePathAndQuery := req.SourcePath
|
||||
// Build the path.
|
||||
// When a source query is present, it is embedded as a path segment
|
||||
// (e.g. /host/path?query/size.fmt) so that the URL parser can extract
|
||||
// it from the last-slash split. The "?" inside a path segment is
|
||||
// percent-encoded by clients but chi delivers it decoded, which is
|
||||
// exactly what the URL parser expects.
|
||||
var path string
|
||||
if req.SourceQuery != "" {
|
||||
sourcePathAndQuery += "?" + req.SourceQuery
|
||||
path = fmt.Sprintf("/v1/image/%s%s%%3F%s/%s.%s",
|
||||
req.SourceHost,
|
||||
req.SourcePath,
|
||||
url.PathEscape(req.SourceQuery),
|
||||
sizeStr,
|
||||
req.Format,
|
||||
)
|
||||
} else {
|
||||
path = fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||
req.SourceHost,
|
||||
req.SourcePath,
|
||||
sizeStr,
|
||||
req.Format,
|
||||
)
|
||||
}
|
||||
|
||||
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%s.%s",
|
||||
req.SourceHost, sourcePathAndQuery, sizeStr, req.Format)
|
||||
|
||||
return (&url.URL{Path: unescapedPath}).EscapedPath(), sig, exp
|
||||
return path, sig, exp
|
||||
}
|
||||
|
||||
// buildSignatureData creates the string to be signed.
|
||||
|
||||
@@ -475,44 +475,6 @@ func TestGenerateSignedURL_WithoutQueryString(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestGenerateSignedURL_ReadmeQueryExample checks the README's example of a
|
||||
// source URL with a query, the image at
|
||||
// https://cdn.example.com/img?url=https://origin.example.com/my%20cat.jpg: its
|
||||
// signature with the README's key, and the path GenerateSignedURL writes for
|
||||
// it, with the "?" written as %3F and the "%" as %25.
|
||||
func TestGenerateSignedURL_ReadmeQueryExample(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signer := signature.New("example-signing-key-for-documentation")
|
||||
|
||||
req := signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: "/img",
|
||||
SourceQuery: "url=https://origin.example.com/my%20cat.jpg",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Unix(1704067200, 0),
|
||||
}
|
||||
|
||||
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
|
||||
|
||||
gotSig := signer.Sign(&req)
|
||||
if gotSig != wantSig {
|
||||
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
|
||||
}
|
||||
|
||||
const wantPath = "/v1/image/cdn.example.com" +
|
||||
"/img%3Furl=https://origin.example.com/my%2520cat.jpg/800x600.webp"
|
||||
|
||||
gotPath, _, _ := signer.GenerateSignedURL(&req, time.Hour)
|
||||
if gotPath != wantPath {
|
||||
t.Errorf("GenerateSignedURL() path = %q, want %q", gotPath, wantPath)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseParams(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
Reference in New Issue
Block a user