Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bd38f29254 | ||
|
|
639a86a4a8 | ||
|
|
7789701831 | ||
|
|
5d1e3ea871 |
@@ -313,9 +313,23 @@ cache at once, they share one upstream fetch (or one read of the cached source)
|
||||
and one transcode: the first request does the work, and the others wait for its
|
||||
image or its error, holding no upstream connection or processing slot of their
|
||||
own. A waiting request stops waiting when its own client goes away. The work
|
||||
goes on for the others even if the first request's client goes away, until that
|
||||
request's `downstream_timeout` ends. The shared fetch sends the first request's
|
||||
ID upstream, and the lines logged for the fetch and the transcode carry that ID.
|
||||
goes on for the others even if the first request's client goes away. The shared
|
||||
fetch sends the first request's ID upstream, and the lines logged for the fetch
|
||||
and the transcode carry that ID.
|
||||
|
||||
A transcode whose request's `downstream_timeout` has ended (for a shared one,
|
||||
the first request's) stops when it gets its processing slot or before it begins
|
||||
encoding, and gives the slot back. libvips cannot stop an encode once it has
|
||||
begun, so an encode keeps its slot until it finishes, which can be after
|
||||
`downstream_timeout`. The limit of 8192 on each side of the source and of the
|
||||
output bounds how long that takes. With one libvips thread, as pixad runs it,
|
||||
encoding 8192x8192 random pixels takes up to about 1 second as JPEG, 15 as PNG,
|
||||
16 as GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the slowest of 8 or 16
|
||||
bits per sample, with or without an alpha channel. WebP, AVIF and JPEG XL are
|
||||
that slow only with an alpha channel: 8-bit RGB takes about 44, 53 and 28
|
||||
seconds. Decoding such a source adds up to about 11 seconds. A WebP, AVIF or
|
||||
JPEG XL encode can therefore hold its slot past the default `downstream_timeout`
|
||||
of 60 seconds.
|
||||
|
||||
The login form (`POST /`) is limited to 5 attempts per minute per client
|
||||
address, counting an IPv6 client by its /64; an attempt over the limit is
|
||||
@@ -419,8 +433,11 @@ HMAC-SHA256(secret, "host:path:query:width:height:format:expiration:quality:fit"
|
||||
Where:
|
||||
|
||||
- `host` — source origin hostname (e.g. `cdn.example.com`)
|
||||
- `path` — source path (e.g. `/photos/cat.jpg`)
|
||||
- `query` — source query string, empty string if none
|
||||
- `path` — source path (e.g. `/photos/cat.jpg`), with each `%` written as `%25`
|
||||
and each `:` as `%3A`, so `/photos/a:b.jpg` is signed as `/photos/a%3Ab.jpg`
|
||||
- `query` — source query string, empty string if none, with `%` and `:` written
|
||||
as in `path`; neither then holds a `:`, so path `/a:b` with no query and path
|
||||
`/a` with query `b:` are signed differently
|
||||
- `width` — requested width in pixels, `0` for original
|
||||
- `height` — requested height in pixels, `0` for original
|
||||
- `format` — output format, one of those listed under Routes, with `original`
|
||||
|
||||
@@ -30,6 +30,29 @@ P2: security: per-origin rate limiting
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-15 a `:` in the source path or query no longer moves where the path
|
||||
ends and the query begins (closes #247): the signed data and the cache key
|
||||
joined host, path and query with `:` and no escaping, so a URL signed for
|
||||
`https://h/a:b` verified for `https://h/a?b:`; the two also shared a cache
|
||||
key, and whichever was fetched first was served for both. In both, the path
|
||||
and the query now have each `%` written as `%25` and each `:` as `%3A`; a path
|
||||
and query holding neither are signed and keyed as before. Signature
|
||||
Specification in `README.md` says so.
|
||||
- 2026-10-15 a transcode stops once its request's `downstream_timeout` has
|
||||
ended, unless it has begun encoding (closes #243): the image processor looked
|
||||
at its context only while it waited for a processing slot, so an image was
|
||||
decoded and encoded, holding the slot, however long after its request's
|
||||
deadline that took. `Process` now returns the context's error, and gives the
|
||||
slot back, when the context has ended by the time it has the slot or before it
|
||||
encodes. libvips cannot stop an encode once it has begun, so that encode keeps
|
||||
its slot until it finishes; the limit of 8192 on each side of the source and
|
||||
of the output bounds how long that takes. With one libvips thread, 8192x8192
|
||||
random pixels take up to about 1 second to encode as JPEG, 15 as PNG, 16 as
|
||||
GIF, 63 as WebP, 110 as AVIF and 96 as JPEG XL, the last three with an alpha
|
||||
channel, plus up to about 11 seconds to decode the source; `README.md` gives
|
||||
these times and no longer says the work stops when `downstream_timeout` ends.
|
||||
The settings of the WebP, AVIF and JPEG XL encoders, which can hold a slot
|
||||
past the default `downstream_timeout` of 60 seconds, are unchanged.
|
||||
- 2026-10-15 the example config allowlists no host (closes #249): it allowlisted
|
||||
`github.com` and `user-images.githubusercontent.com`, where anyone can put an
|
||||
image, so a deployment set up as Getting Started says served those images to
|
||||
@@ -493,11 +516,11 @@ P2: security: per-origin rate limiting
|
||||
cancellation but keeps its deadline, and the others wait for its image or
|
||||
error holding no upstream connection or processing slot, and stop waiting when
|
||||
their own context ends; the request doing the processing waits for it even
|
||||
then, up to its deadline; a request whose context has already ended starts
|
||||
nothing; each request counts one miss, and the processing counts its fetch and
|
||||
transcode once; a panic while processing is reported to Sentry when
|
||||
`sentry_dsn` is set and becomes an error for every waiting request instead of
|
||||
stopping pixad; documented in `README.md`.
|
||||
then, until it ends, which can be after its deadline; a request whose context
|
||||
has already ended starts nothing; each request counts one miss, and the
|
||||
processing counts its fetch and transcode once; a panic while processing is
|
||||
reported to Sentry when `sentry_dsn` is set and becomes an error for every
|
||||
waiting request instead of stopping pixad; documented in `README.md`.
|
||||
- 2026-09-29 only the image routes send CORS headers (closes #98): the CORS
|
||||
middleware, with the `access_control_allow_origin` origin, moved from the
|
||||
router root onto a `/v1` subrouter holding `/v1/image/` and `/v1/e/`, where it
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
@@ -47,3 +48,39 @@ func TestServerBusyAnswers503(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestProcessingTimedOutAnswers503 checks that both image routes answer 503
|
||||
// with the server busy error when the image service reports that processing
|
||||
// stopped at the deadline of the request doing it, wrapped as the service
|
||||
// wraps it. A request that waited for that processing gets this error while
|
||||
// its own deadline may have time left.
|
||||
func TestProcessingTimedOutAnswers503(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h := &Handlers{log: slog.New(slog.DiscardHandler)}
|
||||
req := &imgcache.ImageRequest{SourceHost: "img.example.com", SourcePath: "/a.jpg"}
|
||||
err := fmt.Errorf("%w: failed to encode: %w",
|
||||
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
|
||||
|
||||
plain := httptest.NewRecorder()
|
||||
h.respondImageError(plain, req, err)
|
||||
|
||||
encrypted := httptest.NewRecorder()
|
||||
h.handleImageError(encrypted, err)
|
||||
|
||||
for route, rec := range map[string]*httptest.ResponseRecorder{
|
||||
"/v1/image/": plain, "/v1/e/": encrypted,
|
||||
} {
|
||||
t.Logf("%s: status = %d, body = %q", route, rec.Code, rec.Body.String())
|
||||
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Errorf("%s: status = %d, want %d",
|
||||
route, rec.Code, http.StatusServiceUnavailable)
|
||||
}
|
||||
|
||||
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
|
||||
t.Errorf("%s: body = %q, want the server busy error",
|
||||
route, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package imageprocessor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// readerAfterDeadline is a Process input whose first Read waits until done
|
||||
// is closed, so the deadline passes after Process has taken its slot and
|
||||
// before it encodes.
|
||||
type readerAfterDeadline struct {
|
||||
data *bytes.Reader
|
||||
done <-chan struct{}
|
||||
waited bool
|
||||
}
|
||||
|
||||
func (r *readerAfterDeadline) Read(p []byte) (int, error) {
|
||||
if !r.waited {
|
||||
r.waited = true
|
||||
|
||||
<-r.done
|
||||
}
|
||||
|
||||
return r.data.Read(p)
|
||||
}
|
||||
|
||||
// TestProcessStopsWhenDeadlinePassed checks that Process returns
|
||||
// context.DeadlineExceeded, and holds no slot afterwards, when its context's
|
||||
// deadline has passed before it starts or while it reads its input. libvips
|
||||
// cannot stop an encode once it has begun, so Process must not begin one for
|
||||
// a request that has ended.
|
||||
func TestProcessStopsWhenDeadlinePassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
input := createTestJPEG(t, 50, 50)
|
||||
|
||||
t.Run("before Process starts", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ctx, cancel := context.WithDeadline(t.Context(),
|
||||
time.Now().Add(-time.Second))
|
||||
defer cancel()
|
||||
|
||||
checkProcessStopsAtDeadline(ctx, t, bytes.NewReader(input))
|
||||
})
|
||||
|
||||
t.Run("while Process reads its input", func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
|
||||
defer cancel()
|
||||
|
||||
checkProcessStopsAtDeadline(ctx, t, &readerAfterDeadline{
|
||||
data: bytes.NewReader(input), done: ctx.Done(),
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// checkProcessStopsAtDeadline runs Process on input with ctx, whose deadline
|
||||
// has passed by the time Process could encode, and fails the test unless
|
||||
// Process returns context.DeadlineExceeded and gives its slot back.
|
||||
func checkProcessStopsAtDeadline(
|
||||
ctx context.Context, t *testing.T, input io.Reader,
|
||||
) {
|
||||
t.Helper()
|
||||
|
||||
proc := New(Params{MaxConcurrentProcessing: 1})
|
||||
|
||||
result, err := proc.Process(ctx, input, smallJPEGRequest())
|
||||
if result != nil {
|
||||
t.Logf("Process() made a %dx%d image after its deadline passed",
|
||||
result.Width, result.Height)
|
||||
|
||||
_ = result.Content.Close()
|
||||
}
|
||||
|
||||
t.Logf("Process() error = %v", err)
|
||||
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Errorf("Process() error = %v, want context.DeadlineExceeded", err)
|
||||
}
|
||||
|
||||
if held := len(proc.processingSemaphore); held != 0 {
|
||||
t.Errorf("%d slots still held after Process() returned", held)
|
||||
}
|
||||
}
|
||||
|
||||
// inputNotToBeRead is a Process input that fails the test when it is read.
|
||||
type inputNotToBeRead struct {
|
||||
t *testing.T
|
||||
}
|
||||
|
||||
func (r inputNotToBeRead) Read([]byte) (int, error) {
|
||||
r.t.Error("Process read its input although its deadline had passed")
|
||||
|
||||
return 0, io.EOF
|
||||
}
|
||||
|
||||
// TestProcessReadsNoInputOnceDeadlinePassed checks that Process, called with
|
||||
// a context whose deadline passed before it starts, gives its slot back
|
||||
// without reading its input. With a slot free, the select in acquireSlot
|
||||
// takes the slot or sees the ended context at random, so Process is called
|
||||
// many times.
|
||||
func TestProcessReadsNoInputOnceDeadlinePassed(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
ctx, cancel := context.WithDeadline(t.Context(),
|
||||
time.Now().Add(-time.Second))
|
||||
defer cancel()
|
||||
|
||||
for range 100 {
|
||||
checkProcessStopsAtDeadline(ctx, t, inputNotToBeRead{t: t})
|
||||
}
|
||||
}
|
||||
@@ -208,7 +208,9 @@ func New(params Params) *ImageProcessor {
|
||||
// Process transforms an image according to the request. When
|
||||
// MaxConcurrentProcessing images are already being processed, it waits up
|
||||
// to ProcessingWaitTimeout for one to finish, then fails with
|
||||
// ErrTooManyImages.
|
||||
// ErrTooManyImages. When ctx has ended once Process has a slot, or before it
|
||||
// encodes, it returns ctx's error and gives the slot back. libvips cannot stop
|
||||
// an encode once it has begun, so an encode keeps its slot until it finishes.
|
||||
func (p *ImageProcessor) Process(
|
||||
ctx context.Context,
|
||||
input io.Reader,
|
||||
@@ -290,7 +292,7 @@ func (p *ImageProcessor) Process(
|
||||
}
|
||||
|
||||
// Encode to target format
|
||||
output, err := p.encode(img, outputFormat, req.Quality)
|
||||
output, err := p.encode(ctx, img, outputFormat, req.Quality)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to encode: %w", err)
|
||||
}
|
||||
@@ -417,25 +419,28 @@ func (p *ImageProcessor) WaitForProcessing(ctx context.Context) int {
|
||||
|
||||
// acquireSlot takes a slot in processingSemaphore, waiting at most
|
||||
// processingWaitTimeout for one to free up, and returns the func that gives
|
||||
// it back. A free slot is taken even when ctx has ended; only the wait for
|
||||
// one stops when ctx ends, as the rest of Process does not check ctx.
|
||||
// it back. When ctx has ended by the time it has a slot, it gives the slot
|
||||
// back and returns ctx's error.
|
||||
func (p *ImageProcessor) acquireSlot(ctx context.Context) (func(), error) {
|
||||
release := func() { <-p.processingSemaphore }
|
||||
|
||||
select {
|
||||
case p.processingSemaphore <- struct{}{}:
|
||||
return release, nil
|
||||
default:
|
||||
}
|
||||
|
||||
select {
|
||||
case p.processingSemaphore <- struct{}{}:
|
||||
return release, nil
|
||||
case <-time.After(p.processingWaitTimeout):
|
||||
return nil, ErrTooManyImages
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
|
||||
release := func() { <-p.processingSemaphore }
|
||||
|
||||
// select takes the slot even when ctx has ended at the same moment
|
||||
err := ctx.Err()
|
||||
if err != nil {
|
||||
release()
|
||||
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return release, nil
|
||||
}
|
||||
|
||||
// detectFormat returns the format string for a govips image type.
|
||||
@@ -527,10 +532,17 @@ func (p *ImageProcessor) resize(
|
||||
|
||||
const defaultQuality = 85
|
||||
|
||||
// encode encodes an image to the specified format.
|
||||
// encode encodes an image to the specified format. libvips cannot stop an
|
||||
// encode once it has begun, so when ctx has ended encode returns ctx's error
|
||||
// without beginning one.
|
||||
func (p *ImageProcessor) encode(
|
||||
img *vips.ImageRef, format Format, quality int,
|
||||
ctx context.Context, img *vips.ImageRef, format Format, quality int,
|
||||
) ([]byte, error) {
|
||||
err := ctx.Err()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if quality <= 0 {
|
||||
quality = defaultQuality
|
||||
}
|
||||
@@ -540,7 +552,7 @@ func (p *ImageProcessor) encode(
|
||||
// libvips' built-in profile; govips' own sRGB path variable is set on
|
||||
// first use but read without a lock, so concurrent requests race on it.
|
||||
if img.HasICCProfile() {
|
||||
err := img.TransformICCProfileWithFallback("srgb", "srgb")
|
||||
err = img.TransformICCProfileWithFallback("srgb", "srgb")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to convert to sRGB: %w", err)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
|
||||
"github.com/getsentry/sentry-go"
|
||||
"sneak.berlin/go/pixa/internal/httpfetcher"
|
||||
"sneak.berlin/go/pixa/internal/imageprocessor"
|
||||
"sneak.berlin/go/pixa/internal/magic"
|
||||
)
|
||||
|
||||
@@ -435,6 +436,47 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestService_Get_WaiterGetsProcessingTimedOut has a request with no deadline
|
||||
// join the held fetch of a first request with a short one. With the only
|
||||
// processing slot held, the first request's transcode waits for it until the
|
||||
// first request's deadline ends, and stops then. The waiting request, whose
|
||||
// own context has not ended, must get ErrProcessingTimedOut, which the image
|
||||
// routes answer 503 as when pixa is busy.
|
||||
func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
svc, fixtures, fetcher := setupHeldFetchService(t)
|
||||
svc.processor = imageprocessor.New(
|
||||
imageprocessor.Params{MaxConcurrentProcessing: 1},
|
||||
)
|
||||
|
||||
holdProcessingSlot(t, svc.processor)
|
||||
|
||||
const firstTimeout = 500 * time.Millisecond
|
||||
|
||||
firstCtx, cancelFirst := context.WithTimeout(t.Context(), firstTimeout)
|
||||
defer cancelFirst()
|
||||
|
||||
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, FitCover))
|
||||
|
||||
<-fetcher.started
|
||||
|
||||
waiter := startGet(t.Context(), svc, photoVariant(fixtures, 85, FitCover))
|
||||
|
||||
time.Sleep(arrivalWait)
|
||||
fetcher.releaseFetches()
|
||||
|
||||
t.Logf("first request: Get() error = %v", (<-first).err)
|
||||
|
||||
got := <-waiter
|
||||
t.Logf("waiting request: Get() error = %v", got.err)
|
||||
|
||||
if !errors.Is(got.err, ErrProcessingTimedOut) {
|
||||
t.Errorf("waiting request: Get() error = %v, want %v",
|
||||
got.err, ErrProcessingTimedOut)
|
||||
}
|
||||
}
|
||||
|
||||
// panickingFetcher panics on every fetch.
|
||||
type panickingFetcher struct{}
|
||||
|
||||
|
||||
@@ -128,6 +128,12 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
|
||||
// ErrNegativeCached is returned when a URL is in the negative cache (recently failed).
|
||||
var ErrNegativeCached = errors.New("request is in negative cache (recently failed)")
|
||||
|
||||
// ErrProcessingTimedOut is returned when processing an image stopped because
|
||||
// the downstream_timeout of the request doing it ended, to that request and to
|
||||
// every request that waited for its image.
|
||||
var ErrProcessingTimedOut = errors.New(
|
||||
"image processing stopped at the deadline of the request doing it")
|
||||
|
||||
// Get retrieves a processed image, fetching and processing if necessary.
|
||||
func (s *Service) Get(ctx context.Context, req *ImageRequest) (*ImageResponse, error) {
|
||||
// Propagate AllowHTTP setting to the request
|
||||
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -371,11 +372,13 @@ func HashPath(path string) PathHash {
|
||||
|
||||
// CacheKey generates a unique key for a request variant.
|
||||
// Format: sha256(host:path:query:width:height:format:quality:fit_mode)
|
||||
// The path and query go through escapePercentAndColon, so that a ":" in
|
||||
// either cannot be taken for the one between them.
|
||||
func CacheKey(req *ImageRequest) VariantKey {
|
||||
data := fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%s",
|
||||
req.SourceHost,
|
||||
req.SourcePath,
|
||||
req.SourceQuery,
|
||||
escapePercentAndColon(req.SourcePath),
|
||||
escapePercentAndColon(req.SourceQuery),
|
||||
req.Size.Width,
|
||||
req.Size.Height,
|
||||
req.Format,
|
||||
@@ -387,6 +390,14 @@ func CacheKey(req *ImageRequest) VariantKey {
|
||||
return VariantKey(hex.EncodeToString(h[:]))
|
||||
}
|
||||
|
||||
// escapePercentAndColon returns s with each "%" written as "%25" and each
|
||||
// ":" as "%3A", so that the result holds no ":".
|
||||
func escapePercentAndColon(s string) string {
|
||||
s = strings.ReplaceAll(s, "%", "%25")
|
||||
|
||||
return strings.ReplaceAll(s, ":", "%3A")
|
||||
}
|
||||
|
||||
// VariantStorage handles key-based file storage for processed image variants.
|
||||
// Files are stored at: <basedir>/<ab>/<cd>/<cache_key>
|
||||
// Metadata is stored at: <basedir>/<ab>/<cd>/<cache_key>.meta
|
||||
|
||||
@@ -407,6 +407,62 @@ func TestCacheKey(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheKey_ColonOrPercentInPathOrQuery checks that two different source
|
||||
// URLs whose path or query holds ":" or "%" get different keys.
|
||||
func TestCacheKey_ColonOrPercentInPathOrQuery(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
path1, query1 string
|
||||
path2, query2 string
|
||||
}{
|
||||
{
|
||||
name: "colon moved from path to query",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a", query2: "b:",
|
||||
},
|
||||
{
|
||||
name: "colon in path and its percent-encoding",
|
||||
path1: "/a:b", query1: "",
|
||||
path2: "/a%3Ab", query2: "",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range cases {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
key1 := CacheKey(&ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: tt.path1,
|
||||
SourceQuery: tt.query1,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
})
|
||||
key2 := CacheKey(&ImageRequest{
|
||||
SourceHost: testHostCDN,
|
||||
SourcePath: tt.path2,
|
||||
SourceQuery: tt.query2,
|
||||
Size: Size{Width: 800, Height: 600},
|
||||
Format: FormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: FitCover,
|
||||
})
|
||||
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path1, tt.query1, key1)
|
||||
t.Logf("CacheKey() of path %q query %q = %s", tt.path2, tt.query2, key2)
|
||||
|
||||
if key1 == key2 {
|
||||
t.Errorf("CacheKey() gave path %q query %q and path %q query %q "+
|
||||
"the same key", tt.path1, tt.query1, tt.path2, tt.query2)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestVariantStorage_StoreLogsFailedMetaWrite verifies that a .meta write
|
||||
// that fails is logged at warn and the store still succeeds.
|
||||
func TestVariantStorage_StoreLogsFailedMetaWrite(t *testing.T) {
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
package signature_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"sneak.berlin/go/pixa/internal/signature"
|
||||
)
|
||||
|
||||
// TestSigner_Verify_ColonCannotMovePathQueryBoundary checks that a URL
|
||||
// signed for path "/a:b" with no query does not verify for path "/a" with
|
||||
// query "b:", a different resource on the same host.
|
||||
func TestSigner_Verify_ColonCannotMovePathQueryBoundary(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signer := signature.New(goldenSigningKey)
|
||||
|
||||
signed := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: "/a:b",
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Now().Add(1 * time.Hour),
|
||||
}
|
||||
signed.Signature = signer.Sign(signed)
|
||||
|
||||
replayed := *signed
|
||||
replayed.SourcePath = "/a"
|
||||
replayed.SourceQuery = "b:"
|
||||
|
||||
t.Logf("signature for path %q query %q: %s",
|
||||
signed.SourcePath, signed.SourceQuery, signed.Signature)
|
||||
t.Logf("signature for path %q query %q: %s",
|
||||
replayed.SourcePath, replayed.SourceQuery, signer.Sign(&replayed))
|
||||
|
||||
err := signer.Verify(&replayed)
|
||||
if !errors.Is(err, signature.ErrInvalid) {
|
||||
t.Errorf("Verify() of path %q query %q with the signature for "+
|
||||
"path %q = %v, want %v",
|
||||
replayed.SourcePath, replayed.SourceQuery, signed.SourcePath,
|
||||
err, signature.ErrInvalid)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSigner_EscapedGoldenVectors pins the signature of requests whose path
|
||||
// or query holds ":" or "%", which are signed with "%" written as "%25" and
|
||||
// ":" as "%3A". The expected values were computed once, outside pixa, from
|
||||
// the signed data in each comment.
|
||||
func TestSigner_EscapedGoldenVectors(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
signer := signature.New(goldenSigningKey)
|
||||
|
||||
vectors := []struct {
|
||||
name string
|
||||
path string
|
||||
query string
|
||||
wantSignature string
|
||||
}{
|
||||
{
|
||||
name: "colon in path",
|
||||
path: "/photos/a:b.jpg",
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/a%3Ab.jpg::800:600:webp:1704067200:85:cover"
|
||||
wantSignature: "WnEpZ7bQp1Zvw8WDatlj2vu-4wYKWE3BFBizV8m__6k=",
|
||||
},
|
||||
{
|
||||
name: "colon in query",
|
||||
path: testPath,
|
||||
query: "t=1:2",
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/cat.jpg:t=1%3A2:800:600:webp:1704067200:85:cover"
|
||||
wantSignature: "85kRoN8hhr4cCUkEUjNvUsvVYKjYYfWdeffx5TjJjqA=",
|
||||
},
|
||||
{
|
||||
name: "percent in path and query",
|
||||
path: "/photos/a%20b.jpg",
|
||||
query: "q=50%",
|
||||
// Signed data:
|
||||
// "cdn.example.com:/photos/a%2520b.jpg:q=50%25:800:600:webp:1704067200:85:cover"
|
||||
wantSignature: "MjDiiRdgXMrsjqPMsENtD4aG1hp1e-a6zy67vKTX4CQ=",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range vectors {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
req := &signature.Request{
|
||||
SourceHost: testHost,
|
||||
SourcePath: tt.path,
|
||||
SourceQuery: tt.query,
|
||||
Width: 800,
|
||||
Height: 600,
|
||||
Format: testFormatWebP,
|
||||
Quality: 85,
|
||||
FitMode: testFitCover,
|
||||
Expires: time.Unix(goldenExpiresUnix, 0),
|
||||
}
|
||||
|
||||
gotSignature := signer.Sign(req)
|
||||
t.Logf("Sign() of path %q query %q = %s", tt.path, tt.query, gotSignature)
|
||||
|
||||
if gotSignature != tt.wantSignature {
|
||||
t.Errorf("Sign() = %q, want %q", gotSignature, tt.wantSignature)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@ import (
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -154,15 +155,17 @@ func (s *Signer) GenerateSignedURL(
|
||||
|
||||
// buildSignatureData creates the string to be signed.
|
||||
// Format: "host:path:query:width:height:format:expiration:quality:fit"
|
||||
// All components are used verbatim (exact match). No normalization,
|
||||
// suffix matching, or wildcard expansion is performed. Quality and fit
|
||||
// are the effective transform values, so replaying a signed URL with a
|
||||
// different quality or fit mode fails verification.
|
||||
// The path and query go through escapePercentAndColon, so that a ":" in
|
||||
// either cannot be taken for the one between them; every other component
|
||||
// is used verbatim. No normalization, suffix matching, or wildcard
|
||||
// expansion is performed. Quality and fit are the effective transform
|
||||
// values, so replaying a signed URL with a different quality or fit mode
|
||||
// fails verification.
|
||||
func (s *Signer) buildSignatureData(req *Request) string {
|
||||
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%d:%s",
|
||||
req.SourceHost,
|
||||
req.SourcePath,
|
||||
req.SourceQuery,
|
||||
escapePercentAndColon(req.SourcePath),
|
||||
escapePercentAndColon(req.SourceQuery),
|
||||
req.Width,
|
||||
req.Height,
|
||||
req.Format,
|
||||
@@ -172,6 +175,14 @@ func (s *Signer) buildSignatureData(req *Request) string {
|
||||
)
|
||||
}
|
||||
|
||||
// escapePercentAndColon returns s with each "%" written as "%25" and each
|
||||
// ":" as "%3A", so that the result holds no ":".
|
||||
func escapePercentAndColon(s string) string {
|
||||
s = strings.ReplaceAll(s, "%", "%25")
|
||||
|
||||
return strings.ReplaceAll(s, ":", "%3A")
|
||||
}
|
||||
|
||||
// ParseParams extracts signature and expiration from query parameters.
|
||||
func ParseParams(sig, expStr string) (string, time.Time, error) {
|
||||
if expStr == "" {
|
||||
|
||||
Reference in New Issue
Block a user