Author SHA1 Message Date
clawbot 078bae74a5 Split internal/imgcache into the image proxy and the cache (closes #317)
check / check (push) Waiting to run
The new package internal/imageproxy holds Proxy, which was imgcache.Service,
with its Config, ImageResponse, CacheStatus and errors. internal/imgcache holds
the cache only, and imgcache.CacheConfig is imgcache.Config. The cache reads
and writes each kind it keeps with one pair of methods: Variant and PutVariant,
Source and PutSource, Failure and PutFailure. Source replaces LookupSource
followed by GetSourceContent. imgcache.go and module.go are gone. The proxy's
tests moved with it. They reach the cache through its database, its state
directory and its metrics registry, and run its reconciliation pass with a
second cache, as pixad does at startup.

Model: opus-5-5
2026-10-26 15:56:47 +00:00
clawbot 87cb1ab5f6 Serve both image routes with one function and one error mapping (closes #316)
check / check (push) Waiting to run
HandleImage parses the path and verifies the signature, and HandleImageEnc
decrypts the token. Each then calls serveImage, which chooses the format for
auto, gets the image and serves it. respondImageError answers the image
service's errors on both routes, and handleImageError is gone. /v1/image/ now
answers 504 when the upstream sends no response headers within
upstream_fetch_timeout, as README.md documented for /v1/e/. respondImageError
logs a 500 at error level and the other answers at warn. The image service
alone logs a source refused for its first bytes. A failed write is logged once,
at warn, with the request and the bytes served. README.md lists each image
route's answers as bullets. Tests send the same errors through both routes.

Model: opus-5-5
2026-10-26 15:24:24 +01:00
clawbot 77d55abb7e Derive every key from the signing key in one place (closes #315)
check / check (push) Canceled after 0s
seal.DeriveKeys derives the session hash and block keys, the encrypted URL
key and the CSRF key from the signing key, with their salts declared beside
it, and returns them as one seal.Keys. internal/providers calls it once at
startup and provides the value to the fx graph. The session manager, the
encrypted URL generator and the CSRF protection each take their own key.
session.NewManager, encurl.NewGenerator, newCSRFProtect and handlers.New
return no error, as nothing in them can fail now. Each key is derived as
before, with seal.DeriveKey and the same salt. The signing key itself stays
the HMAC key of signed URLs and the login password. README.md says under
signing_key what it protects.

Model: opus-5-5
2026-10-26 14:10:20 +01:00
clawbot 7b6712538e Take the signature, the expiry and the URL scheme off the image request (closes #314)
check / check (push) Canceled after 0s
The image request now holds only its source and its transform, exactly what
the variant key covers. The /v1/image/ handler checks the allowlist and
verifies the signature itself with Signer.Verify(req, sig, expires). Both
image handlers keep the URL's expiry as a local value for Cache-Control. The
image service holds no signer, allowlist or AllowHTTP, and ValidateRequest is
gone. The fetcher takes the source and builds its URL with Source.URL from its
own AllowHTTP, so no URL string is built and parsed again. chooseAutoFormat
returns the format it chooses instead of writing into the request. The signed
URL format and every route's answers are unchanged.

Model: opus-5-5
2026-10-26 13:49:12 +01:00
80 changed files with 2820 additions and 2418 deletions
+62 -26
View File
@@ -248,32 +248,58 @@ path under `/v1/` answers 200, in maintenance mode too.
- `GET` or `HEAD` `/v1/image/<host>/<path>/<size>.<format>`, or
`/v1/image/<host>/<path>/<size>` with no format — an image, fetched, resized
and converted (below). Needs: a signature, unless the host is allowlisted (see
Source Hosts). Answers: 200; 304 when `If-None-Match` matches the image's
`ETag`; 400 for a URL or parameter that is not valid, or for the format `auto`
an `Accept` header that is not valid; 406 for the format `auto` when `Accept`
allows none of the formats it chooses from; 401 for a missing or wrong
signature, a missing `exp` or an `exp` in the past; 403 when the request's
`Referer` names a host in `referer_blocklist`, checked before the signature,
the cache and the upstream fetch; 403 when the upstream host, or a host it
redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an
address in a blocked network (see `blocked_networks`); 429 over
`image_requests_per_minute` (below); 415 when the upstream answers with a
`Content-Type` that is not one of the source formats (below), with a body
whose first bytes are not those of the format its `Content-Type` names, or
with an image whose first bytes libvips reads as another format; 502 when the
upstream answered with an error status, and for 5 minutes after that, or after
a body whose first bytes are not those of its `Content-Type`, for an image of
the same source URL that pixa can neither serve from its cache nor make from
the source image still in its cache; 503 when pixa is busy or in maintenance
mode; 500 for any other failure.
Source Hosts). Answers:
- 200 with the image.
- 304 when `If-None-Match` matches the image's `ETag`.
- 400 for a URL or parameter that is not valid.
- 400 when the format is `auto` and the `Accept` header is not valid.
- 406 when the format is `auto` and `Accept` allows none of the formats it
chooses from.
- 401 for a missing or wrong signature.
- 401 for a missing `exp`.
- 401 for an `exp` in the past.
- 403 when the request's `Referer` names a host in `referer_blocklist`,
checked before the signature, the cache and the upstream fetch.
- 403 when the upstream host, or a host it redirects to, is `localhost` or
ends in `.localhost` or `.local`.
- 403 when the upstream host, or a host it redirects to, has an address in a
blocked network (see `blocked_networks`).
- 429 over `image_requests_per_minute` (below).
- 415 when the upstream answers with a `Content-Type` that is not one of the
source formats (below).
- 415 when the upstream sends a body whose first bytes are not those of the
format its `Content-Type` names.
- 415 when the upstream sends an image whose first bytes libvips reads as
another format.
- 502 when the upstream answers with an error status.
- 502 for 5 minutes after that for the same source URL, when pixa has
neither the image nor its source image in its cache.
- 502 likewise for 5 minutes after a body whose first bytes are not those of
its `Content-Type`.
- 503 when pixa is busy.
- 503 when pixa is in maintenance mode.
- 504 when the upstream has not sent its response headers within
`upstream_fetch_timeout`.
- 500 when that time runs out while the image itself is still arriving.
- 500 for any other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
`If-None-Match` matches the image's `ETag`; 400 for a token that does not
decrypt, or that asks for a size or fit that is not valid; 410 once it has
expired; 504 when the upstream has not sent its response headers within
`upstream_fetch_timeout`, but 500 when that time runs out while the image
itself is still arriving; 400 for an `Accept` header that is not valid, and
406, 403, 415, 429, 502, 503 and 500, as for `/v1/image/`.
(see Encrypted URLs). Needs: nothing but the URL. Answers:
- 200 with the image.
- 304 when `If-None-Match` matches the image's `ETag`.
- 400 for a token that does not decrypt.
- 400 for a token that asks for a size or fit that is not valid.
- 410 once the token has expired.
- 400 when the token's format is `auto` and the `Accept` header is not
valid.
- These answers, for the same causes as on `/v1/image/`:
- 406
- 403
- 415
- 429
- 502
- 503
- 504
- 500
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
nothing. Answers: 200.
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
@@ -765,7 +791,17 @@ Key settings in more detail:
duration; default `60s`. The upstream fetch counts toward it, and so do the
waits for an upstream connection and for a processing slot (up to 10 seconds
each), so keep it longer than `upstream_fetch_timeout` plus 20 seconds
- `signing_key` — HMAC secret for URL signatures
- `signing_key` — the secret that protects signed URLs, encrypted URLs and
logins. pixa uses it as it is for two things:
- the HMAC key of signed URLs (see Signature Specification)
- the password of the login page
pixa derives every other key it uses from it at startup, with HKDF-SHA256:
- a key that signs the session cookie
- a key that encrypts the session cookie
- a key that encrypts the token of an encrypted URL
- a key that signs the CSRF token cookie of the login and generator pages
- `db_url` — the SQLite database to open; omitted, it is
`file:<state_dir>/state.sqlite3?_pragma=journal_mode(WAL)`, which keeps the
database in WAL mode. `state_dir` is percent-encoded in it, so a `%`, `?` or
+87
View File
@@ -30,6 +30,93 @@ P2: security: per-origin rate limiting
# Completed Steps
- 2026-10-26 `internal/imgcache` holds the cache only (closes #317). The new
package `internal/imageproxy` fetches, transforms, caches and serves images.
It imports `imgcache`, and `imgcache` does not import it.
- `imgcache.Service` is `imageproxy.Proxy`, built by `imageproxy.NewProxy`
from an `imageproxy.Config`.
- `ImageResponse`, `CacheStatus`, `ErrNegativeCached` and
`ErrProcessingTimedOut` moved with it, under their names.
- `imgcache.CacheConfig` is `imgcache.Config`.
- The cache has one read and one write for each kind it keeps. Each takes
the place of an earlier method:
- `Variant`, of `GetVariant`
- `PutVariant`, of `StoreVariant`
- `Source`, of `LookupSource` followed by `GetSourceContent`
- `PutSource`, of `StoreSource`
- `Failure`, of `checkNegativeCache`
- `PutFailure`, of `StoreNegative`
- `Variant` and `Source` each open the file and hand its access time to the
background writer. A cache hit still makes no database call.
- `imgcache.go` and `module.go` are gone.
- A cached source whose row cannot be read is logged as "failed to load
cached source, fetching". It was logged as "source lookup failed".
- The proxy's tests moved to `internal/imageproxy`. They reach the cache
through its database, its state directory and its metrics registry.
- Those tests run the reconciliation pass as pixad does when it starts: with
a second cache over the same database and directory.
- 2026-10-26 both image routes serve an image with one function, `serveImage`,
and answer its errors with one, `respondImageError` (closes #316).
`HandleImage` parses the path and verifies the signature, and `HandleImageEnc`
decrypts the token. Each then calls `serveImage`.
- `handleImageError`, the encrypted route's own mapping, is gone.
- `/v1/image/` answers 504 when the upstream has not sent its response
headers within `upstream_fetch_timeout`, as `/v1/e/` did. It answered 500.
- `respondImageError` logs the image service's errors as "failed to get
image". Each answer gets the same level on `/v1/image/` and on `/v1/e/`:
- 500: error, as pixa's own failure
- 400: warn
- 403: warn
- 415: warn
- 502: warn
- 503: warn
- 504: warn
- Before, `/v1/image/` logged all of these answers at error level. `/v1/e/`
logged only its 500, as "image request failed", without the host and the
path.
- `respondImageError` does not log the 415 for a body refused for its first
bytes, on either route. The image service logs it as "source refused", at
warn level.
- The "image served" line carries the host, the path and `fetched_bytes` on
both routes. The debug line "encrypted image request" is gone.
- The debug line "failed to decrypt URL" is now "invalid encrypted URL". A
token that decrypts to a payload that does not decode also gets it.
- A failed write is logged once, as "failed to write response" at warn
level, since most often the client has gone away. It carries the
`request_id`, the host, the path, `served_bytes` and the error, and is not
also logged as "image served".
- 2026-10-26 every key derived from the signing key is derived in one place
(closes #315): `seal.DeriveKeys` returns them as one `seal.Keys`, with their
salts declared beside it. `internal/providers` calls it once at startup. Each
key is derived as before, with `seal.DeriveKey` and the same salt.
- The session manager, the encrypted URL generator and the CSRF protection
each take their own key instead of the signing key.
- `session.NewManager`, `encurl.NewGenerator`, `newCSRFProtect` and
`handlers.New` return no error, as nothing in them can fail now.
- The signing key itself stays the HMAC key of signed URLs and the login
password.
- `README.md` says under `signing_key` what it protects.
- 2026-10-26 the image request holds only what names the cached object (closes
#314): its source and its transform, which the variant key covers. The signed
URL format, the signing algorithm and every route's answers are unchanged.
- The `/v1/image/` handler checks `allowlist_hosts` and verifies the
signature itself, with `Signer.Verify(req, sig, expires)`.
- Each image handler keeps its URL's expiry as a value of its own, for
`Cache-Control`. `encurl`'s `Generate` takes the expiry and its `Parse`
returns it.
- The fetcher takes the source and builds its URL with `Source.URL`, over
http only when the fetcher's own `AllowHTTP` is set. `validateURL` takes
that URL, so no URL string is built and parsed again.
- `chooseAutoFormat` returns the format it chooses, and the handler puts it
into the request it parsed.
- The image service serves images through `Get`, and has `WaitForProcessing`
for shutdown.
- Removed:
- `Signature`, `Expires`, `AllowHTTP` and `SourceURL` of the image
request
- `Service.ValidateRequest`, and the service's signer, allowlist and
`AllowHTTP`
- `extractHost` of the fetcher and `urlToFSPath` of the mock fetcher
- 2026-10-26 pixad's fx graph builds the image pipeline (closes #313).
`database.New` now opens the database, so the parts below can read it while
they are built; its stop hook still runs after the cache's. The cache's
+21 -28
View File
@@ -16,9 +16,6 @@ const (
DefaultQuality = 85
DefaultFormat = imagerequest.FormatJXL
DefaultFitMode = imagerequest.FitCover
// HKDF salt for URL encryption key derivation
urlKeySalt = "pixa-urlenc-v1"
)
// Errors returned by encurl operations.
@@ -48,23 +45,20 @@ type Generator struct {
key [seal.KeySize]byte
}
// NewGenerator creates an encrypted URL generator with a key derived
// from the signing key.
func NewGenerator(signingKey string) (*Generator, error) {
key, err := seal.DeriveKey([]byte(signingKey), urlKeySalt)
if err != nil {
return nil, err
}
return &Generator{key: key}, nil
// NewGenerator creates an encrypted URL generator that encrypts its tokens
// with key.
func NewGenerator(key [seal.KeySize]byte) *Generator {
return &Generator{key: key}
}
// Generate creates an encrypted URL token from the request's source,
// transform and expiry, each as the request holds it; an empty format, a
// quality of 0 or an empty fit mode is left out of the token, and a zero
// Expires makes a token that never expires. The token is CBOR-encoded,
// Generate creates an encrypted URL token from the request's source and
// transform, each as the request holds it, and from expires; an empty format,
// a quality of 0 or an empty fit mode is left out of the token, and a zero
// expires makes a token that never expires. The token is CBOR-encoded,
// encrypted with NaCl secretbox, and base64url-encoded.
func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
func (g *Generator) Generate(
req *imagerequest.Request, expires time.Time,
) (string, error) {
p := &payload{
SourceHost: req.Source.Host,
SourcePath: req.Source.Path,
@@ -76,8 +70,8 @@ func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
FitMode: req.Transform.FitMode,
}
if !req.Expires.IsZero() {
p.ExpiresAt = req.Expires.Unix()
if !expires.IsZero() {
p.ExpiresAt = expires.Unix()
}
// CBOR encode the payload
@@ -91,19 +85,19 @@ func (g *Generator) Generate(req *imagerequest.Request) (string, error) {
}
// Parse decrypts and validates an encrypted URL token and returns the
// request it holds, with default values for omitted optional fields. A token
// that never expires gives the zero Expires.
// request it holds, with default values for omitted optional fields, and its
// expiry. A token that never expires gives the zero time.
// Returns ErrExpired if the token has expired, or other errors for invalid tokens.
func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
func (g *Generator) Parse(token string) (*imagerequest.Request, time.Time, error) {
// Decrypt
data, err := seal.Decrypt(g.key, token)
if err != nil {
if errors.Is(err, seal.ErrDecryptionFailed) ||
errors.Is(err, seal.ErrInvalidPayload) {
return nil, ErrDecryptFailed
return nil, time.Time{}, ErrDecryptFailed
}
return nil, err
return nil, time.Time{}, err
}
// CBOR decode
@@ -111,12 +105,12 @@ func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
err = cbor.Unmarshal(data, &p)
if err != nil {
return nil, ErrInvalidFormat
return nil, time.Time{}, ErrInvalidFormat
}
// Check expiration (0 = never expires)
if p.ExpiresAt != 0 && time.Now().Unix() > p.ExpiresAt {
return nil, ErrExpired
return nil, time.Time{}, ErrExpired
}
format := p.Format
@@ -151,6 +145,5 @@ func (g *Generator) Parse(token string) (*imagerequest.Request, error) {
Quality: quality,
FitMode: fitMode,
},
Expires: expires,
}, nil
}, expires, nil
}
+34 -29
View File
@@ -7,6 +7,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// Shared test fixture strings.
@@ -19,11 +20,13 @@ const (
func TestGenerator_GenerateAndParse(t *testing.T) {
t.Parallel()
gen, err := encurl.NewGenerator("test-signing-key-12345")
keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil {
t.Fatalf("NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
@@ -36,10 +39,10 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
Quality: 90,
FitMode: imagerequest.FitContain,
},
Expires: time.Now().Add(time.Hour),
}
expires := time.Now().Add(time.Hour)
token, err := gen.Generate(req)
token, err := gen.Generate(req, expires)
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -48,7 +51,7 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
t.Fatal("Generate() returned empty token")
}
parsed, err := gen.Parse(token)
parsed, parsedExpires, err := gen.Parse(token)
if err != nil {
t.Fatalf("Parse() error = %v", err)
}
@@ -91,30 +94,30 @@ func TestGenerator_GenerateAndParse(t *testing.T) {
parsed.Transform.FitMode, req.Transform.FitMode)
}
if parsed.Expires.Unix() != req.Expires.Unix() {
t.Errorf("ExpiresAt = %d, want %d", parsed.Expires.Unix(), req.Expires.Unix())
if parsedExpires.Unix() != expires.Unix() {
t.Errorf("ExpiresAt = %d, want %d", parsedExpires.Unix(), expires.Unix())
}
}
func TestGenerator_Parse_Expired(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(-time.Hour), // Already expired
}
token, err := gen.Generate(req)
token, err := gen.Generate(req, time.Now().Add(-time.Hour)) // Already expired
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
_, err = gen.Parse(token)
_, _, err = gen.Parse(token)
if err == nil {
t.Error("Parse() should fail for expired token")
}
@@ -127,9 +130,10 @@ func TestGenerator_Parse_Expired(t *testing.T) {
func TestGenerator_Parse_InvalidToken(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
_, err := gen.Parse("not-a-valid-token")
_, _, err := gen.Parse("not-a-valid-token")
if err == nil {
t.Error("Parse() should fail for invalid token")
}
@@ -138,17 +142,17 @@ func TestGenerator_Parse_InvalidToken(t *testing.T) {
func TestGenerator_Parse_TamperedToken(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen.Generate(req)
token, _ := gen.Generate(req, time.Now().Add(time.Hour))
// Tamper with the token
tampered := []byte(token)
@@ -156,7 +160,7 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
tampered[10] ^= 0x01
}
_, err := gen.Parse(string(tampered))
_, _, err := gen.Parse(string(tampered))
if err == nil {
t.Error("Parse() should fail for tampered token")
}
@@ -165,20 +169,21 @@ func TestGenerator_Parse_TamperedToken(t *testing.T) {
func TestGenerator_Parse_WrongKey(t *testing.T) {
t.Parallel()
gen1, _ := encurl.NewGenerator("signing-key-1")
gen2, _ := encurl.NewGenerator("signing-key-2")
keys1, _ := seal.DeriveKeys("signing-key-1")
keys2, _ := seal.DeriveKeys("signing-key-2")
gen1 := encurl.NewGenerator(keys1.EncryptedURLKey)
gen2 := encurl.NewGenerator(keys2.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen1.Generate(req)
token, _ := gen1.Generate(req, time.Now().Add(time.Hour))
_, err := gen2.Parse(token)
_, _, err := gen2.Parse(token)
if err == nil {
t.Error("Parse() should fail with different signing key")
}
@@ -187,7 +192,8 @@ func TestGenerator_Parse_WrongKey(t *testing.T) {
func TestGenerator_Parse_Defaults(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
// Request with only required fields - should get defaults
token, err := gen.Generate(&imagerequest.Request{
@@ -195,13 +201,12 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
})
}, time.Now().Add(time.Hour))
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
req, err := gen.Parse(token)
req, _, err := gen.Parse(token)
if err != nil {
t.Fatalf("Parse() error = %v", err)
}
@@ -225,17 +230,17 @@ func TestGenerator_Parse_Defaults(t *testing.T) {
func TestGenerator_TokenIsURLSafe(t *testing.T) {
t.Parallel()
gen, _ := encurl.NewGenerator("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
gen := encurl.NewGenerator(keys.EncryptedURLKey)
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testSourceHost,
Path: testSourcePath,
},
Expires: time.Now().Add(time.Hour),
}
token, _ := gen.Generate(req)
token, _ := gen.Generate(req, time.Now().Add(time.Hour))
// Check that token only contains URL-safe characters
for _, c := range token {
+14 -17
View File
@@ -153,20 +153,20 @@ func (s *Handlers) HandleGenerateURL() http.HandlerFunc {
return
}
// The image URL must fetch the source URL as it was written
req.AllowHTTP = s.config.AllowHTTP
fetchURL := req.SourceURL()
// The image URL must fetch the source URL as it was written. The
// fetcher fetches over http only when allow_http is set.
fetchURL := req.Source.URL(s.config.AllowHTTP)
if !fetchedAsWritten(parsed, fetchURL) {
s.renderGeneratorWithForm(w, r, "Source URL cannot be fetched as "+
"written: pixa would fetch "+fetchURL,
"written: pixa would fetch "+fetchURL.String(),
r.Form, http.StatusBadRequest)
return
}
// Generate encrypted token
token, err := s.encGen.Generate(req)
token, err := s.encGen.Generate(req, expiresAt)
if err != nil {
s.log.Error("failed to generate encrypted URL", "error", err)
s.renderGeneratorWithForm(w, r, "Failed to generate URL", r.Form,
@@ -226,25 +226,23 @@ func parseSourceURL(sourceURL string) (*url.URL, error) {
// in RawPath when it is not the decoded path percent-encoded again.
// EscapedPath ignores RawPath when it also holds a character net/url encodes,
// such as a space, so RawPath is compared itself.
func fetchedAsWritten(source *url.URL, fetchURL string) bool {
func fetchedAsWritten(source, fetchURL *url.URL) bool {
writtenPath := source.RawPath
if writtenPath == "" {
writtenPath = source.EscapedPath()
}
fetched, err := url.Parse(fetchURL)
return err == nil && source.User == nil && !source.ForceQuery &&
fetched.EscapedPath() == writtenPath &&
fetched.RawQuery == source.RawQuery
return source.User == nil && !source.ForceQuery &&
fetchURL.EscapedPath() == writtenPath &&
fetchURL.RawQuery == source.RawQuery
}
// buildGenerateRequest parses the numeric form fields and assembles the
// image request the encrypted URL's token is made from. ttl=0 means never
// expires (Expires stays the zero time). A non-numeric or out-of-range
// width, height, quality or ttl, or an unrecognized fit mode, is a client
// error naming the offending field. The format field is passed through
// unchecked.
// image request the encrypted URL's token is made from, and its expiry.
// ttl=0 means never expires (the expiry is the zero time). A non-numeric or
// out-of-range width, height, quality or ttl, or an unrecognized fit mode, is
// a client error naming the offending field. The format field is passed
// through unchecked.
func buildGenerateRequest(
parsed *url.URL, form url.Values,
) (*imagerequest.Request, time.Time, int, error) {
@@ -294,7 +292,6 @@ func buildGenerateRequest(
Quality: quality,
FitMode: fitMode,
},
Expires: expiresAt,
}
return req, expiresAt, ttl, nil
+6 -15
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -44,27 +45,17 @@ func newCSRFTestRouter(t *testing.T) (*Handlers, http.Handler) {
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("session.NewManager() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
config: cfg,
sessMgr: sessMgr,
encGen: encGen,
csrfProtect: protect,
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
}
r := chi.NewRouter()
@@ -10,6 +10,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -22,15 +23,15 @@ func TestLoginLogLeavesOutSubmittedKey(t *testing.T) {
var buf bytes.Buffer
sessMgr, err := session.NewManager(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("session.NewManager() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.NewJSONHandler(&buf, nil)),
config: &config.Config{SigningKey: testSigningKey},
sessMgr: sessMgr,
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
}
submittedKeys := []string{wrongKey, testSigningKey}
@@ -11,6 +11,7 @@ import (
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -232,8 +233,13 @@ func TestHandleLogout_SaveFails_Answers500(t *testing.T) {
db := setupTestDB(t)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
sessMgr, err := session.NewManagerWithDatabase(
t.Context(), testSigningKey, db)
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil {
t.Fatalf("session.NewManagerWithDatabase() error = %v", err)
}
@@ -12,11 +12,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 400 naming the AVIF pixel limit when the image processor refuses an AVIF
// output above it, wrapped as the service wraps it. The request's source only
// goes into a log line, so it is left empty.
// TestAVIFOutputTooLargeAnswers400 checks that respondImageError, which
// answers the errors of both image routes, answers 400 naming the AVIF pixel
// limit when the image processor refuses an AVIF output above it, wrapped as
// the service wraps it. The request's source only goes into a log line, so it
// is left empty.
func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
t.Parallel()
@@ -25,25 +25,15 @@ func TestAVIFOutputTooLargeAnswers400(t *testing.T) {
imageprocessor.ErrAVIFOutputTooLarge)
wantBody := fmt.Sprintf("above %d pixels", imageprocessor.MaxAVIFOutputPixels)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if rec.Code != http.StatusBadRequest {
t.Errorf("status = %d, want %d", rec.Code, http.StatusBadRequest)
}
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusBadRequest {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusBadRequest)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("%s: body = %q, want it to say %q",
name, rec.Body.String(), wantBody)
}
if !strings.Contains(rec.Body.String(), wantBody) {
t.Errorf("body = %q, want it to say %q", rec.Body.String(), wantBody)
}
}
+6 -15
View File
@@ -11,6 +11,7 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -32,27 +33,17 @@ func newBodyLimitTestRouter(
cfg := &config.Config{SigningKey: testSigningKey, Debug: true}
sessMgr, err := session.NewManager(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("session.NewManager() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
protect, err := newCSRFProtect(testSigningKey, cfg.Debug)
if err != nil {
t.Fatalf("newCSRFProtect() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
config: cfg,
sessMgr: sessMgr,
encGen: encGen,
csrfProtect: protect,
sessMgr: session.NewManager(keys.SessionHashKey, keys.SessionBlockKey),
encGen: encurl.NewGenerator(keys.EncryptedURLKey),
csrfProtect: newCSRFProtect(keys.CSRFKey, cfg.Debug),
}
r := chi.NewRouter()
+21 -15
View File
@@ -19,11 +19,15 @@ import (
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// countingHost is the allowlisted host whose photoPath the server of
@@ -119,7 +123,7 @@ func newCountingDatabaseServer(
tb.Fatalf("encoding the photo: %v", err)
}
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tb.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -128,7 +132,7 @@ func newCountingDatabaseServer(
tb.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{
countingHost + photoPath: &fstest.MapFile{Data: photo.Bytes()},
@@ -137,20 +141,20 @@ func newCountingDatabaseServer(
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: testSigningKey,
Allowlist: []string{countingHost},
MaxResponseSize: 50 << 20,
Logger: slog.New(slog.DiscardHandler),
})
if err != nil {
tb.Fatalf("imgcache.NewService() error = %v", err)
tb.Fatalf("imageproxy.NewProxy() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
tb.Fatalf("encurl.NewGenerator() error = %v", err)
tb.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := encGen.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: countingHost, Path: photoPath},
Transform: imagerequest.Transform{
@@ -159,15 +163,17 @@ func newCountingDatabaseServer(
Quality: 85,
FitMode: imagerequest.FitCover,
},
})
}, time.Time{})
if err != nil {
tb.Fatalf("encGen.Generate() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{countingHost}),
}
r := chi.NewRouter()
@@ -223,9 +229,9 @@ func TestImageRoutes_CacheHitsMakeNoDatabaseCall(t *testing.T) {
rec := serveImage(t, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) {
if cacheStatus != string(imageproxy.CacheHit) {
t.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imgcache.CacheHit)
target, cacheStatus, imageproxy.CacheHit)
}
}
@@ -255,9 +261,9 @@ func BenchmarkHandleImage_CacheHits(b *testing.B) {
rec := serveImage(b, srv, target)
cacheStatus := rec.Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) {
if cacheStatus != string(imageproxy.CacheHit) {
b.Errorf("GET %s: X-Pixa-Cache = %q, want %s",
target, cacheStatus, imgcache.CacheHit)
target, cacheStatus, imageproxy.CacheHit)
return
}
+5 -15
View File
@@ -9,15 +9,10 @@ import (
"sneak.berlin/go/pixa/internal/seal"
)
// csrfKeySalt provides domain separation for the CSRF authentication key,
// derived from the signing key so tokens survive restarts without extra
// configuration and never reuse the session or encrypted-URL key material.
const csrfKeySalt = "pixa-csrf-v1"
// newCSRFProtect builds the CSRF-protection middleware for the
// state-mutating HTML form routes. The token lives in its own cookie,
// independent of the session cookie, so it also protects the login POST
// where no session exists yet (login CSRF).
// where no session exists yet (login CSRF). key signs that cookie.
//
// When plaintext is true (local HTTP development), requests are marked
// plaintext so the library neither demands an https Referer nor sets a
@@ -25,13 +20,8 @@ const csrfKeySalt = "pixa-csrf-v1"
// service runs behind a TLS-terminating proxy, so plaintext is false and
// the library enforces its https Referer origin check.
func newCSRFProtect(
signingKey string, plaintext bool,
) (func(http.Handler) http.Handler, error) {
key, err := seal.DeriveKey([]byte(signingKey), csrfKeySalt)
if err != nil {
return nil, err
}
key [seal.KeySize]byte, plaintext bool,
) func(http.Handler) http.Handler {
protect := csrf.Protect(
key[:],
csrf.Path("/"),
@@ -40,7 +30,7 @@ func newCSRFProtect(
)
if !plaintext {
return protect, nil
return protect
}
return func(next http.Handler) http.Handler {
@@ -49,7 +39,7 @@ func newCSRFProtect(
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
protected.ServeHTTP(w, csrf.PlaintextHTTPRequest(r))
})
}, nil
}
}
// CSRF returns the CSRF-protection middleware for the login and generator
+22 -11
View File
@@ -13,11 +13,15 @@ import (
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// mismatchedSourcePath is the path of the source that
@@ -27,14 +31,15 @@ import (
const mismatchedSourcePath = "/images/source.png"
// newRoutesFetchingThrough returns both image routes, and the Handlers behind
// them, whose image service fetches through fetcher and has allowlistedHost on
// its allowlist. The handlers and the image service log to log.
// them, whose image service fetches through fetcher, and which have
// allowlistedHost on their allowlist. The handlers and the image service log
// to log.
func newRoutesFetchingThrough(
t *testing.T, fetcher httpfetcher.Fetcher, log *slog.Logger,
) (*Handlers, http.Handler) {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -43,28 +48,34 @@ func newRoutesFetchingThrough(
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: testSigningKey,
Allowlist: []string{allowlistedHost},
MaxResponseSize: 50 << 20,
Logger: log,
})
if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err)
t.Fatalf("imageproxy.NewProxy() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
h := &Handlers{log: log, imgSvc: svc, encGen: encGen}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: log,
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
@@ -93,7 +104,7 @@ func mismatchedSourceEncURL(t *testing.T, h *Handlers) string {
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
})
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
+11 -12
View File
@@ -18,16 +18,17 @@ var (
"image/jxl, image/avif, image/webp or image/jpeg")
)
// chooseAutoFormat replaces the format auto in req with the format
// formatForAccept chooses from r's Accept header, and adds Vary: Accept to the
// chooseAutoFormat returns the format to serve for the requested format: for
// the format auto, the format formatForAccept chooses from r's Accept header,
// and any other format as it is. For auto it adds Vary: Accept to the
// response, which then depends on that header. It answers 400 for an Accept
// header that is not valid and 406 for one that allows none of the formats,
// and reports whether req can be served. Any other format is left as it is.
// and reports whether the image can be served.
func (s *Handlers) chooseAutoFormat(
w http.ResponseWriter, r *http.Request, req *imagerequest.Request,
) bool {
if req.Transform.Format != imagerequest.FormatAuto {
return true
w http.ResponseWriter, r *http.Request, requested imagerequest.Format,
) (imagerequest.Format, bool) {
if requested != imagerequest.FormatAuto {
return requested, true
}
w.Header().Add("Vary", "Accept")
@@ -36,18 +37,16 @@ func (s *Handlers) chooseAutoFormat(
if errors.Is(err, errNotAcceptable) {
s.respondError(w, err.Error(), http.StatusNotAcceptable)
return false
return "", false
}
if err != nil {
s.respondError(w, err.Error(), http.StatusBadRequest)
return false
return "", false
}
req.Transform.Format = format
return true
return format, true
}
// formatForAccept returns the format an auto URL is served in for the Accept
@@ -75,9 +75,9 @@ func TestFormatForAccept(t *testing.T) {
// autoPhotoURLs returns a signed /v1/image/ URL and an encrypted /v1/e/ URL,
// both valid for a minute, for the JPEG at photoPath on signedHost at 50x50 in
// the format auto. The first is signed with testSigningKey, the key of the
// image service newSignedHostServer builds, and the second is made with h's
// generator.
// the format auto. The first is signed with testSigningKey, the key the
// handlers that newSignedHostServer builds check signatures with. The second
// is made with h's generator.
func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
t.Helper()
@@ -101,8 +101,7 @@ func autoPhotoURLs(t *testing.T, h *Handlers) (string, string) {
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatAuto,
},
Expires: time.Now().Add(time.Minute),
})
}, time.Now().Add(time.Minute))
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
+16 -11
View File
@@ -13,9 +13,11 @@ import (
"sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/healthcheck"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
)
// Params defines dependencies for Handlers.
@@ -25,8 +27,9 @@ type Params struct {
Logger *logger.Logger
Healthcheck *healthcheck.Healthcheck
Config *config.Config
Keys seal.Keys
ImageService *imgcache.Service
ImageService *imageproxy.Proxy
SessionManager *session.Manager
EncryptedURLGenerator *encurl.Generator
}
@@ -36,23 +39,23 @@ type Handlers struct {
log *slog.Logger
hc *healthcheck.Healthcheck
config *config.Config
imgSvc *imgcache.Service
imgSvc *imageproxy.Proxy
sessMgr *session.Manager
encGen *encurl.Generator
csrfProtect func(http.Handler) http.Handler
// signer checks the signature of a /v1/image/ URL with signing_key.
signer *signature.Signer
// allowlist matches the hosts of allowlist_hosts, which a /v1/image/
// URL may name without a signature.
allowlist *allowlist.HostAllowList
// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
// reports whether a URL's host is on that list.
refererBlocklist *allowlist.HostAllowList
}
// New creates a new Handlers instance.
func New(_ fx.Lifecycle, params Params) (*Handlers, error) {
csrfProtect, err := newCSRFProtect(params.Config.SigningKey, params.Config.Debug)
if err != nil {
return nil, err
}
func New(_ fx.Lifecycle, params Params) *Handlers {
return &Handlers{
log: params.Logger.Get(),
hc: params.Healthcheck,
@@ -60,9 +63,11 @@ func New(_ fx.Lifecycle, params Params) (*Handlers, error) {
imgSvc: params.ImageService,
sessMgr: params.SessionManager,
encGen: params.EncryptedURLGenerator,
csrfProtect: csrfProtect,
csrfProtect: newCSRFProtect(params.Keys.CSRFKey, params.Config.Debug),
signer: signature.New(params.Config.SigningKey),
allowlist: allowlist.New(params.Config.AllowlistHosts),
refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
}, nil
}
}
// WaitForProcessing waits until no image is being processed, or until ctx
+13 -10
View File
@@ -19,10 +19,14 @@ import (
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
)
// testFixtures contains test data.
@@ -46,7 +50,7 @@ func setupTestHandler(t *testing.T) *testFixtures {
tmpDir := t.TempDir()
db := setupTestDB(t)
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -55,24 +59,24 @@ func setupTestHandler(t *testing.T) *testFixtures {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: "test-signing-key-must-be-32-chars",
Allowlist: []string{goodHost},
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("failed to create service: %v", err)
t.Fatalf("failed to create proxy: %v", err)
}
h := &Handlers{
imgSvc: svc,
log: slog.Default(),
imgSvc: svc,
log: slog.Default(),
signer: signature.New("test-signing-key-must-be-32-chars"),
allowlist: allowlist.New([]string{goodHost}),
}
return &testFixtures{
@@ -127,10 +131,9 @@ func newMockFetcher(fs fs.FS) *mockFetcher {
}
func (f *mockFetcher) Fetch(
_ context.Context, url string,
_ context.Context, source imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
// Remove https:// prefix
path := url[8:] // Remove "https://"
path := source.Host + source.Path
data, err := fs.ReadFile(f.fs, path)
if err != nil {
+132 -99
View File
@@ -7,14 +7,15 @@ import (
"net/http"
"net/url"
"strconv"
"strings"
"time"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
@@ -27,13 +28,13 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return
}
req, ok := s.parseImageRequest(w, r)
req, sig, expires, ok := s.parseImageRequest(w, r)
if !ok {
return
}
// Validate signature if required
err := s.imgSvc.ValidateRequest(req)
err := s.validateSignature(req, sig, expires)
if err != nil {
s.log.Warn("signature validation failed",
"host", req.Source.Host,
@@ -45,27 +46,7 @@ func (s *Handlers) HandleImage() http.HandlerFunc {
return
}
// The signature covers the format auto, not the format chosen
if !s.chooseAutoFormat(w, r, req) {
return
}
// Get cache key for logging
cacheKey := imagerequest.CacheKey(req)
// Get the image (from cache or fetch/process)
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
s.writeImageResponse(w, r, req, resp, cacheKey, startTime)
s.serveImage(w, r, req, expires)
}
}
@@ -82,11 +63,11 @@ func (s *Handlers) HandleRobotsTxt() http.HandlerFunc {
}
// parseImageRequest parses the image URL's path and query parameters into
// an image request. On invalid input it writes an error response and
// returns false.
// an image request, and returns it with the URL's signature and expiry. On
// invalid input it writes an error response and returns false.
func (s *Handlers) parseImageRequest(
w http.ResponseWriter, r *http.Request,
) (*imagerequest.Request, bool) {
) (*imagerequest.Request, string, time.Time, bool) {
// The path is parsed decoded once, as r.URL.Path holds it, and not from
// chi's wildcard: chi routes on the path as the client wrote it whenever
// that is not Go's own encoding of it, so its wildcard would still hold
@@ -100,7 +81,7 @@ func (s *Handlers) parseImageRequest(
)
s.respondError(w, "invalid image URL: "+err.Error(), http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
// Parse signature params from query string. r.URL.Query() would silently
@@ -113,7 +94,7 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, fmt.Sprintf("invalid query string %q: %v",
r.URL.RawQuery, err), http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
for name, values := range query {
@@ -121,17 +102,15 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, fmt.Sprintf("invalid %s: given more than once",
name), http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
}
req.Signature = query.Get("sig")
req.Expires, err = parseExpires(query)
expires, err := parseExpires(query)
if err != nil {
s.respondError(w, err.Error(), http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
// Parse optional quality and fit params. Only a q missing from the URL is
@@ -143,7 +122,7 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, `invalid q: not a number, got ""`,
http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
req.Transform.Quality, err = parseFormInt(query, "q",
@@ -152,7 +131,7 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, fmt.Sprintf("%v, got %q", err, qStr),
http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
// Only a fit missing from the URL is cover. A fit in the URL that is not a
@@ -162,7 +141,7 @@ func (s *Handlers) parseImageRequest(
if query.Has("fit") && req.Transform.FitMode == "" {
s.respondError(w, `invalid fit: not a fit mode, got ""`, http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
// Default fit mode if not set
@@ -179,15 +158,40 @@ func (s *Handlers) parseImageRequest(
s.respondError(w, "invalid image request: "+err.Error(),
http.StatusBadRequest)
return nil, false
return nil, "", time.Time{}, false
}
return req, true
return req, query.Get("sig"), expires, true
}
// validateSignature returns nil when the source host of req is on
// allowlist_hosts and names no port. Otherwise it verifies sig, the URL's
// signature of req, and expires, the URL's expiry.
func (s *Handlers) validateSignature(
req *imagerequest.Request, sig string, expires time.Time,
) error {
// An allowlist entry names no port, so a host followed by ":", with or
// without a port, needs a signature. Go's URL type reads the port as the
// fetcher does: in an IPv6 address written without brackets, such as
// 2001:db8::1:8443, it is what follows the last ":". This is checked here
// and not in allowlist.IsAllowed, which also matches referer_blocklist,
// where a Referer must be blocked whatever its port. Port and IsAllowed
// read only the host of a URL.
sourceURL := &url.URL{Host: req.Source.Host}
hostHasPort := sourceURL.Port() != "" ||
strings.HasSuffix(sourceURL.Host, ":")
if !hostHasPort && s.allowlist.IsAllowed(sourceURL) {
return nil
}
// Signature required for non-allowed hosts
return s.signer.Verify(req, sig, expires)
}
// parseExpires reads the exp query parameter, a Unix time in seconds. An exp
// missing from the URL gives the zero time, which the signature check takes
// as no expiration. An exp in the URL that is not a whole number, an empty
// missing from the URL gives the zero time, which the signature check refuses
// as a missing exp. An exp in the URL that is not a whole number, an empty
// one included, is an error naming exp and the value.
func parseExpires(query url.Values) (time.Time, error) {
if !query.Has("exp") {
@@ -215,7 +219,11 @@ func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
}
}
// respondImageError maps image retrieval errors to HTTP responses.
// respondImageError answers err, the image service's error for req, on both
// image routes. It logs a 500 at error level, as pixa's own failure. It logs
// the other answers at warn level, since the client, the upstream or a full
// limit caused them. The 415 for a source refused for its first bytes is not
// logged here, because the image service logs it.
func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imagerequest.Request, err error,
) {
@@ -231,59 +239,53 @@ func (s *Handlers) respondImageError(
return
}
s.log.Error("failed to get image",
var (
status int
message string
)
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
status, message = http.StatusForbidden, "forbidden"
case errors.Is(err, httpfetcher.ErrUpstreamError):
status, message = http.StatusBadGateway, "upstream error"
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
status, message = http.StatusGatewayTimeout, "upstream timeout"
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imageproxy.ErrProcessingTimedOut):
status = http.StatusServiceUnavailable
message = "server busy, try again later"
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat):
status = http.StatusUnsupportedMediaType
message = "unsupported source format"
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
status = http.StatusBadRequest
message = fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels)
default:
s.log.Error("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
s.respondError(w, "internal error", http.StatusInternalServerError)
return
}
s.log.Warn("failed to get image",
"host", req.Source.Host,
"path", req.Source.Path,
"error", err,
)
// Check for specific error types
if errors.Is(err, httpfetcher.ErrSSRFBlocked) {
s.respondError(w, "forbidden", http.StatusForbidden)
return
}
if errors.Is(err, httpfetcher.ErrUpstreamError) {
s.respondError(w, "upstream error", http.StatusBadGateway)
return
}
if errors.Is(err, httpfetcher.ErrTooManyConnections) ||
errors.Is(err, imageprocessor.ErrTooManyImages) ||
errors.Is(err, imgcache.ErrProcessingTimedOut) {
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
return
}
if errors.Is(err, httpfetcher.ErrInvalidContentType) ||
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat) {
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
return
}
if errors.Is(err, imageprocessor.ErrOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
return
}
if errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge) {
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
return
}
s.respondError(w, "internal error", http.StatusInternalServerError)
s.respondError(w, message, status)
}
// cacheControl returns the Cache-Control header for an image served through a
@@ -340,13 +342,34 @@ func notModified(w http.ResponseWriter, r *http.Request, etag string) bool {
return true
}
// writeImageResponse writes headers and streams the image content,
// handling conditional and HEAD requests.
func (s *Handlers) writeImageResponse(
// serveImage serves the image that req asks for. Both image routes call it
// once they have the request and its URL's expiry, which ends the
// Cache-Control max-age. It chooses the format for the format auto, answers
// If-None-Match with 304 and HEAD with the headers only, and answers an error
// of the image service with respondImageError.
func (s *Handlers) serveImage(
w http.ResponseWriter, r *http.Request,
req *imagerequest.Request, resp *imgcache.ImageResponse,
cacheKey imagerequest.VariantKey, startTime time.Time,
req *imagerequest.Request, expires time.Time,
) {
// The signature or the token covers the format auto, not the format chosen
format, ok := s.chooseAutoFormat(w, r, req.Transform.Format)
if !ok {
return
}
req.Transform.Format = format
startTime := time.Now()
resp, err := s.imgSvc.Get(r.Context(), req)
if err != nil {
s.respondImageError(w, req, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
@@ -355,7 +378,7 @@ func (s *Handlers) writeImageResponse(
}
// Cache control headers
w.Header().Set("Cache-Control", cacheControl(req.Expires))
w.Header().Set("Cache-Control", cacheControl(expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) {
@@ -372,18 +395,28 @@ func (s *Handlers) writeImageResponse(
// Stream the response
w.WriteHeader(http.StatusOK)
// A failed write is most often a client that has gone away, not pixa's
// failure, so it is logged at warn level
servedBytes, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response",
s.log.Warn("failed to write response",
"request_id", middleware.GetReqID(r.Context()),
"host", req.Source.Host,
"path", req.Source.Path,
"served_bytes", servedBytes,
"error", err,
)
return
}
// Log cache status and timing after serving
duration := time.Since(startTime)
s.log.Info("image served",
"request_id", middleware.GetReqID(r.Context()),
"cache_key", cacheKey,
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"cache_status", resp.CacheStatus,
"duration_ms", duration.Milliseconds(),
"format", req.Transform.Format,
@@ -14,10 +14,14 @@ import (
"github.com/go-chi/chi/v5"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// photoPath is the path of the JPEG that newSignedHostServer serves.
@@ -32,7 +36,7 @@ func newSignedHostServer(
) (*Handlers, http.Handler) {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -43,7 +47,7 @@ func newSignedHostServer(
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: newMockFetcher(fstest.MapFS{
signedHost + photoPath: &fstest.MapFile{Data: jpegData},
@@ -52,23 +56,26 @@ func newSignedHostServer(
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: testSigningKey,
MaxResponseSize: 50 << 20,
Logger: log,
})
if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err)
t.Fatalf("imageproxy.NewProxy() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: log,
imgSvc: svc,
encGen: encGen,
log: log,
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New(nil),
}
r := chi.NewRouter()
@@ -202,8 +209,7 @@ func TestHandleImageEnc_MaxAge(t *testing.T) {
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
Expires: tt.expires,
})
}, tt.expires)
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
+201 -24
View File
@@ -1,21 +1,28 @@
package handlers
import (
"bytes"
"context"
"encoding/json"
"fmt"
"image/color"
"io"
"log/slog"
"net/http"
"net/http/httptest"
"slices"
"strings"
"testing"
"testing/fstest"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/signature"
@@ -26,11 +33,11 @@ import (
const allowlistedHost = "allowed.example.com"
// newImageRoute returns the image route of a Handlers whose service fetches
// with fetcher and checks signatures with testSigningKey.
// with fetcher, and which checks signatures with testSigningKey.
func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Helper()
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -39,22 +46,25 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: testSigningKey,
Allowlist: []string{allowlistedHost},
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("failed to create service: %v", err)
t.Fatalf("failed to create proxy: %v", err)
}
h := &Handlers{imgSvc: svc, log: slog.New(slog.DiscardHandler)}
h := &Handlers{
imgSvc: svc,
log: slog.New(slog.DiscardHandler),
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
@@ -63,7 +73,7 @@ func newImageRoute(t *testing.T, fetcher httpfetcher.Fetcher) http.Handler {
}
// newPhotoFetcher returns a mock fetcher that serves a JPEG at photoPath on
// each of hosts, and answers any other URL with an upstream error.
// each of hosts, and answers any other source with an upstream error.
func newPhotoFetcher(t *testing.T, hosts ...string) *httpfetcher.MockFetcher {
t.Helper()
@@ -90,10 +100,10 @@ func photoURLWithSig(host, sig string, expires time.Time) string {
return fmt.Sprintf("%s?sig=%s&exp=%d", photoURL(host), sig, expires.Unix())
}
// photoSignature returns the signature of photoURL(host) at the default
// quality and fit, made with key and expiring at expires.
func photoSignature(key, host string, expires time.Time) string {
return signature.New(key).Sign(&imagerequest.Request{
// photoRequest returns the image request of photoURL(host) at the default
// quality and fit.
func photoRequest(host string) *imagerequest.Request {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: photoPath,
@@ -104,8 +114,13 @@ func photoSignature(key, host string, expires time.Time) string {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}
}
// photoSignature returns the signature of photoURL(host) at the default
// quality and fit, made with key and expiring at expires.
func photoSignature(key, host string, expires time.Time) string {
return signature.New(key).Sign(photoRequest(host), expires)
}
// sendGet sends a GET for target to route and returns the response.
@@ -286,10 +301,9 @@ func TestHandleImage_AllowlistOrSignature(t *testing.T) {
// host can be, and checks that each is refused with 400 naming the host as
// decoded: allowlistedHost with a character after it that a URL host cannot
// hold, percent-encoded, an IPv6 address without brackets, and brackets
// around something that is not an IPv6 address. A "?" or "#" after
// allowlistedHost would otherwise end the host when the source URL is read,
// so the request would pass the allowlist and be fetched and cached under a
// host that is not the one fetched.
// around something that is not an IPv6 address. Later steps put the host
// unparsed into the allowlist check, the cache key and the upstream URL, so
// such a host is refused first.
func TestHandleImage_InvalidHost_Returns400(t *testing.T) {
t.Parallel()
@@ -360,10 +374,9 @@ func TestHandleImage_HostForms_Served(t *testing.T) {
// TestHandleImage_HashInSourcePathOrQuery_Returns400 sends image URLs whose
// source path or source query holds a "#", written as %23, and checks that
// each is refused with 400, the one signed for that source path included. The
// source URL would otherwise end at the "#" when it is fetched, so each text
// after it would fetch the image at photoPath and cache it under keys of its
// own.
// each is refused with 400, the one signed for that source path included.
// A "#" starts a URL's fragment, so neither the source path nor the source
// query may hold one.
func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
t.Parallel()
@@ -381,8 +394,7 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: exp,
})
}, exp)
tests := []struct {
name string
@@ -408,3 +420,168 @@ func TestHandleImage_HashInSourcePathOrQuery_Returns400(t *testing.T) {
})
}
}
// failingFetcher answers every fetch with its error.
type failingFetcher struct {
err error
}
// Fetch returns f's error.
func (f failingFetcher) Fetch(
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
return nil, f.err
}
// photoURLsByRoute returns, for each image route, the function that makes the
// URL of the JPEG at photoPath on signedHost on that route.
func photoURLsByRoute() map[string]func(*testing.T, *Handlers) string {
return map[string]func(*testing.T, *Handlers) string{
"/v1/image/": signedPhotoURL,
"/v1/e/": encPhotoURL,
}
}
// TestImageRoutes_SameErrorAnswers checks that both image routes answer each
// upstream fetch error below with the same status. They log it as "failed to
// get image" at the same level: error for the 500 and warn for the others.
// Each request gets routes of its own, so no request finds another's failed
// fetch in the negative cache.
func TestImageRoutes_SameErrorAnswers(t *testing.T) {
t.Parallel()
tests := []struct {
name string
err error
wantStatus int
wantLevel slog.Level
}{
{"blocked upstream address", httpfetcher.ErrSSRFBlocked,
http.StatusForbidden, slog.LevelWarn},
{"upstream error status",
fmt.Errorf("%w: status 404", httpfetcher.ErrUpstreamError),
http.StatusBadGateway, slog.LevelWarn},
{"no response headers within the fetch timeout",
httpfetcher.ErrUpstreamTimeout, http.StatusGatewayTimeout,
slog.LevelWarn},
{"no free upstream connection", httpfetcher.ErrTooManyConnections,
http.StatusServiceUnavailable, slog.LevelWarn},
{"Content-Type not a source format",
fmt.Errorf("%w: image/svg+xml", httpfetcher.ErrInvalidContentType),
http.StatusUnsupportedMediaType, slog.LevelWarn},
{"any other failure", io.ErrUnexpectedEOF,
http.StatusInternalServerError, slog.LevelError},
}
for _, tt := range tests {
for route, imageURL := range photoURLsByRoute() {
t.Run(route+" "+tt.name, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, routes := newRoutesFetchingThrough(t,
failingFetcher{err: tt.err},
slog.New(slog.NewJSONHandler(&logs, nil)))
rec := sendGet(t, routes, imageURL(t, h))
if rec.Code != tt.wantStatus {
t.Errorf("status = %d, want %d: %s",
rec.Code, tt.wantStatus, rec.Body)
}
levels := loggedLevels(t, &logs)
if !slices.Contains(levels[tt.wantLevel.String()],
"failed to get image") {
t.Errorf(`no "failed to get image" line logged at level %s`,
tt.wantLevel)
}
if tt.wantLevel != slog.LevelError &&
len(levels[slog.LevelError.String()]) != 0 {
t.Errorf("logged at error level: %q",
levels[slog.LevelError.String()])
}
})
}
}
}
// failingResponseWriter is a response writer whose Write always fails, as when
// the client has gone away.
type failingResponseWriter struct {
*httptest.ResponseRecorder
}
// Write writes nothing and returns io.ErrClosedPipe.
func (failingResponseWriter) Write([]byte) (int, error) {
return 0, io.ErrClosedPipe
}
// TestImageRoutes_FailedWriteLoggedOnce checks that both image routes log a
// failed write of the image once, at warn level, with the request's ID, the
// source host and path, the bytes served and the error. Neither may also log
// the image as served.
func TestImageRoutes_FailedWriteLoggedOnce(t *testing.T) {
t.Parallel()
const requestID = "test-request-id"
for route, imageURL := range photoURLsByRoute() {
t.Run(route, func(t *testing.T) {
t.Parallel()
var logs bytes.Buffer
h, srv := newSignedHostServer(t,
slog.New(slog.NewJSONHandler(&logs, nil)))
ctx := context.WithValue(t.Context(),
middleware.RequestIDKey, requestID)
srv.ServeHTTP(failingResponseWriter{httptest.NewRecorder()},
httptest.NewRequestWithContext(
ctx, http.MethodGet, imageURL(t, h), nil))
t.Logf("logged:\n%s", logs.String())
var failedWrites []map[string]any
dec := json.NewDecoder(&logs)
for dec.More() {
var line map[string]any
err := dec.Decode(&line)
if err != nil {
t.Fatalf("decoding log line: %v", err)
}
switch line["msg"] {
case "image served":
t.Error(`"image served" logged after a failed write`)
case "failed to write response":
failedWrites = append(failedWrites, line)
}
}
if len(failedWrites) != 1 {
t.Fatalf(`"failed to write response" logged %d times, want 1`,
len(failedWrites))
}
want := map[string]any{
"level": slog.LevelWarn.String(),
"request_id": requestID,
"host": signedHost,
"path": photoPath,
"served_bytes": float64(0),
"error": io.ErrClosedPipe.Error(),
}
for key, wantValue := range want {
if got := failedWrites[0][key]; got != wantValue {
t.Errorf("%s = %v, want %v", key, got, wantValue)
}
}
})
}
}
@@ -38,40 +38,44 @@ func getImage(t *testing.T, fix *testFixtures, target string) int {
}
// generateSignedURL returns the /v1/image/ URL of req, signed with key and
// expiring after ttl, and sets req's expiry. An unset quality or fit of req is
// first set to 85 or cover, the values the route uses when a URL has no q or
// fit. The URL carries q and fit next to sig and exp, so the route verifies it
// for the quality and fit it was signed with.
// expiring after ttl. It signs a copy of req, so req itself is not changed.
// An unset quality or fit in the copy is set to 85 or cover, the values the
// route uses when a URL has no q or fit. The URL carries q and fit next to sig
// and exp, so the route verifies it for the quality and fit it was signed
// with.
func generateSignedURL(
key string, req *imagerequest.Request, ttl time.Duration,
) string {
if req.Transform.Quality == 0 {
req.Transform.Quality = 85
signed := *req
if signed.Transform.Quality == 0 {
signed.Transform.Quality = 85
}
if req.Transform.FitMode == "" {
req.Transform.FitMode = imagerequest.FitCover
if signed.Transform.FitMode == "" {
signed.Transform.FitMode = imagerequest.FitCover
}
req.Expires = time.Now().Add(ttl)
sig := signature.New(key).Sign(req)
expires := time.Now().Add(ttl)
sig := signature.New(key).Sign(&signed, expires)
// A source query follows the source path after a "?". The image route
// reads the path decoded once, so the whole path is written in Go's own
// percent-encoding of a URL path, which writes "?" as %3F and "%" as %25,
// and leaves "/", ";", ",", "=" and "&" as they are.
sourcePathAndQuery := req.Source.Path
if req.Source.Query != "" {
sourcePathAndQuery += "?" + req.Source.Query
sourcePathAndQuery := signed.Source.Path
if signed.Source.Query != "" {
sourcePathAndQuery += "?" + signed.Source.Query
}
unescapedPath := fmt.Sprintf("/v1/image/%s%s/%dx%d.%s",
req.Source.Host, sourcePathAndQuery,
req.Transform.Size.Width, req.Transform.Size.Height, req.Transform.Format)
signed.Source.Host, sourcePathAndQuery,
signed.Transform.Size.Width, signed.Transform.Size.Height,
signed.Transform.Format)
return fmt.Sprintf("%s?sig=%s&exp=%d&q=%d&fit=%s",
(&url.URL{Path: unescapedPath}).EscapedPath(), sig, req.Expires.Unix(),
req.Transform.Quality, req.Transform.FitMode)
(&url.URL{Path: unescapedPath}).EscapedPath(), sig, expires.Unix(),
signed.Transform.Quality, signed.Transform.FitMode)
}
// TestHandleImage_SignatureCoversQualityAndFit signs a URL for quality 85
@@ -93,8 +97,7 @@ func TestHandleImage_SignatureCoversQualityAndFit(t *testing.T) {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}, expires)
signedURL := fmt.Sprintf("/v1/image/%s/images/photo.jpg/50x50.jpeg?sig=%s&exp=%d",
signedHost, sig, expires.Unix())
@@ -277,8 +280,7 @@ func TestHandleImage_SourceQueryEncodedAnyWay(t *testing.T) {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}, expires)
tests := []struct {
name string
@@ -344,8 +346,7 @@ func TestHandleImage_SourceURLHoldingPercent25Verifies(t *testing.T) {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}, expires)
target := fmt.Sprintf("/v1/image/%s%s/50x50.jpeg?sig=%s&exp=%d",
signedHost, tt.writtenPathAndQuery, sig, expires.Unix())
+17 -123
View File
@@ -2,21 +2,13 @@ package handlers
import (
"errors"
"fmt"
"io"
"net/http"
"strconv"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
)
// HandleImageEnc handles requests to /v1/e/{token}/* for encrypted
@@ -28,157 +20,59 @@ func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return
}
ctx := r.Context()
start := time.Now()
req, ok := s.parseImageEncRequest(w, r)
if !ok || !s.chooseAutoFormat(w, r, req) {
req, expires, ok := s.parseImageEncRequest(w, r)
if !ok {
return
}
// Log the request
s.log.Debug("encrypted image request",
"host", req.Source.Host,
"path", req.Source.Path,
"dimensions", fmt.Sprintf("%dx%d",
req.Transform.Size.Width, req.Transform.Size.Height),
"format", req.Transform.Format,
)
// Fetch and process the image (no signature validation
// needed - encrypted URL is trusted)
resp, err := s.imgSvc.Get(ctx, req)
if err != nil {
s.handleImageError(w, err)
return
}
defer func() { _ = resp.Content.Close() }()
// Set response headers
w.Header().Set("Content-Type", resp.ContentType)
if resp.ContentLength > 0 {
w.Header().Set("Content-Length", strconv.FormatInt(resp.ContentLength, 10))
}
// Cache headers: max-age ends at the URL's expiry
w.Header().Set("Cache-Control", cacheControl(req.Expires))
w.Header().Set("X-Pixa-Cache", string(resp.CacheStatus))
if notModified(w, r, resp.ETag) {
return
}
// A HEAD request gets the headers only
if r.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}
// Stream the response
written, err := io.Copy(w, resp.Content)
if err != nil {
s.log.Error("failed to write response", "error", err)
return
}
// Log completion
duration := time.Since(start)
s.log.Info("image served",
"request_id", middleware.GetReqID(ctx),
"cache_key", imagerequest.CacheKey(req),
"host", req.Source.Host,
"path", req.Source.Path,
"format", req.Transform.Format,
"cache_status", resp.CacheStatus,
"served_bytes", written,
"duration_ms", duration.Milliseconds(),
)
s.serveImage(w, r, req, expires)
}
}
// parseImageEncRequest decrypts the token of an encrypted image URL into an
// image request and checks it. On a token that is missing, does not decrypt,
// has expired or asks for something not valid, it writes an error response
// and returns false.
// image request and checks it, and returns it with the token's expiry. On a
// token that is missing, does not decrypt, has expired or asks for something
// not valid, it writes an error response and returns false.
func (s *Handlers) parseImageEncRequest(
w http.ResponseWriter, r *http.Request,
) (*imagerequest.Request, bool) {
) (*imagerequest.Request, time.Time, bool) {
// Extract token from URL
token := chi.URLParam(r, "token")
if token == "" {
s.respondError(w, "missing token", http.StatusBadRequest)
return nil, false
return nil, time.Time{}, false
}
// Decrypt and validate the payload
req, err := s.encGen.Parse(token)
req, expires, err := s.encGen.Parse(token)
if err != nil {
if errors.Is(err, encurl.ErrExpired) {
s.log.Debug("encrypted URL expired", "error", err)
s.respondError(w, "URL has expired", http.StatusGone)
return nil, false
return nil, time.Time{}, false
}
s.log.Debug("failed to decrypt URL", "error", err)
s.log.Debug("invalid encrypted URL", "error", err)
s.respondError(w, "invalid encrypted URL", http.StatusBadRequest)
return nil, false
return nil, time.Time{}, false
}
// Apply the same dimension and fit-mode bounds as the plain image
// route: a sealed payload is trusted for its origin, not for staying
// within limits, so an over-limit size or unknown fit mode is a 400
// here rather than an out-of-memory or a 500 from the processor.
// within limits. This refuses an over-limit size before the source is
// fetched. An unknown fit mode is a 400 here rather than a 500 from the
// processor.
err = imagerequest.ValidateImageRequest(req)
if err != nil {
s.log.Debug("encrypted URL failed validation", "error", err)
s.respondError(w, "invalid encrypted URL: "+err.Error(),
http.StatusBadRequest)
return nil, false
return nil, time.Time{}, false
}
return req, true
}
// handleImageError converts image service errors to HTTP responses.
func (s *Handlers) handleImageError(w http.ResponseWriter, err error) {
switch {
case errors.Is(err, httpfetcher.ErrSSRFBlocked):
s.respondError(w, "forbidden", http.StatusForbidden)
case errors.Is(err, httpfetcher.ErrUpstreamError):
s.respondError(w, "upstream error", http.StatusBadGateway)
case errors.Is(err, httpfetcher.ErrUpstreamTimeout):
s.respondError(w, "upstream timeout", http.StatusGatewayTimeout)
case errors.Is(err, httpfetcher.ErrTooManyConnections),
errors.Is(err, imageprocessor.ErrTooManyImages),
errors.Is(err, imgcache.ErrProcessingTimedOut):
s.respondError(w, "server busy, try again later",
http.StatusServiceUnavailable)
case errors.Is(err, httpfetcher.ErrInvalidContentType),
errors.Is(err, imageprocessor.ErrUnsupportedInputFormat),
errors.Is(err, magic.ErrMagicByteMismatch),
errors.Is(err, magic.ErrUnknownFormat),
errors.Is(err, magic.ErrNotEnoughData):
s.respondError(w, "unsupported source format",
http.StatusUnsupportedMediaType)
case errors.Is(err, imageprocessor.ErrOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for the source: the side "+
"computed from its proportions is above %d",
imageprocessor.MaxOutputDimension), http.StatusBadRequest)
case errors.Is(err, imageprocessor.ErrAVIFOutputTooLarge):
s.respondError(w, fmt.Sprintf("size too large for AVIF: the output "+
"would be above %d pixels", imageprocessor.MaxAVIFOutputPixels),
http.StatusBadRequest)
default:
s.log.Error("image request failed", "error", err)
s.respondError(w, "internal error", http.StatusInternalServerError)
}
return req, expires, true
}
+8 -4
View File
@@ -6,11 +6,13 @@ import (
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// newEncTestServer builds a router serving the encrypted-URL route with a
@@ -20,11 +22,13 @@ import (
func newEncTestServer(t *testing.T) (*encurl.Generator, http.Handler) {
t.Helper()
encGen, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: slog.New(slog.DiscardHandler),
encGen: encGen,
@@ -62,7 +66,7 @@ func TestHandleImageEnc_OverLimitDimension_Returns400(t *testing.T) {
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 100000, Height: 100000},
},
})
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -91,7 +95,7 @@ func TestHandleImageEnc_InvalidFitMode_Returns400(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
FitMode: imagerequest.Fit("bogus"),
},
})
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -10,6 +10,7 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// requireServedPhoto requires that rec answers 200 with the JPEG at photoPath
@@ -70,25 +71,24 @@ func TestHandleImageEnc_RejectedToken(t *testing.T) {
},
}
valid, err := h.encGen.Generate(&photo)
valid, err := h.encGen.Generate(&photo, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
expiredPhoto := photo
expiredPhoto.Expires = time.Now().Add(-time.Minute)
expired, err := h.encGen.Generate(&expiredPhoto)
expired, err := h.encGen.Generate(&photo, time.Now().Add(-time.Minute))
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
otherGen, err := encurl.NewGenerator("another-signing-key-fedcba9876543210")
keys, err := seal.DeriveKeys("another-signing-key-fedcba9876543210")
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
otherKey, err := otherGen.Generate(&photo)
otherGen := encurl.NewGenerator(keys.EncryptedURLKey)
otherKey, err := otherGen.Generate(&photo, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -69,8 +69,7 @@ func TestImageWithoutFormat_SignedAsJXL(t *testing.T) {
Quality: encurl.DefaultQuality,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}, expires)
query := fmt.Sprintf("?sig=%s&exp=%d", sig, expires.Unix())
for _, size := range []string{"50x50.jxl", "50x50"} {
@@ -94,7 +93,7 @@ func TestImageEncWithoutFormat_ServesJPEGXL(t *testing.T) {
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
},
})
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
@@ -15,7 +15,11 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/signature"
)
// blockedReferer is a page on leech.example, which newRefererRoutes puts on
@@ -31,11 +35,11 @@ type countingFetcher struct {
// Fetch counts the fetch and passes it on.
func (f *countingFetcher) Fetch(
ctx context.Context, url string,
ctx context.Context, source imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1)
return f.Fetcher.Fetch(ctx, url)
return f.Fetcher.Fetch(ctx, source)
}
// newRefererRoutes returns both image routes of a Handlers whose
@@ -49,7 +53,7 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
}
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.Config{
StateDir: t.TempDir(),
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -58,30 +62,32 @@ func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher)
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
svc, err := imageproxy.NewProxy(&imageproxy.Config{
Cache: cache,
Fetcher: fetcher,
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: testSigningKey,
Allowlist: []string{allowlistedHost},
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err)
t.Fatalf("imageproxy.NewProxy() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
encGen := encurl.NewGenerator(keys.EncryptedURLKey)
h := &Handlers{
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
signer: signature.New(testSigningKey),
allowlist: allowlist.New([]string{allowlistedHost}),
refererBlocklist: allowlist.New(
[]string{"leech.example", ".hotlinker.example"}),
}
@@ -18,8 +18,8 @@ import (
// signedPhotoURL returns a signed /v1/image/ URL, valid for a minute, for the
// JPEG at photoPath on signedHost at 50x50. It is signed with testSigningKey,
// the key of the image service newSignedHostServer builds. It takes a Handlers
// it does not use, so that it has the type of encPhotoURL.
// the key the handlers that newSignedHostServer builds check signatures with.
// It takes a Handlers it does not use, so that it has the type of encPhotoURL.
func signedPhotoURL(t *testing.T, _ *Handlers) string {
t.Helper()
@@ -49,7 +49,7 @@ func encPhotoURL(t *testing.T, h *Handlers) string {
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
},
})
}, time.Time{})
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
+30 -46
View File
@@ -11,13 +11,14 @@ import (
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestServerBusyAnswers503 checks that both image routes answer 503 with
// a clear error when the image service gives up waiting for a free upstream
// connection or processing slot, wrapped as the service wraps them.
// TestServerBusyAnswers503 checks that respondImageError, which answers the
// errors of both image routes, answers 503 with a clear error when the image
// service gives up waiting for a free upstream connection or processing slot,
// wrapped as the service wraps them.
func TestServerBusyAnswers503(t *testing.T) {
t.Parallel()
@@ -30,61 +31,44 @@ func TestServerBusyAnswers503(t *testing.T) {
fmt.Errorf("upstream fetch failed: %w", httpfetcher.ErrTooManyConnections),
fmt.Errorf("image processing failed: %w", imageprocessor.ErrTooManyImages),
} {
plain := httptest.NewRecorder()
h.respondImageError(plain, req, err)
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusServiceUnavailable)
}
for route, rec := range map[string]*httptest.ResponseRecorder{
"/v1/image/": plain, "/v1/e/": encrypted,
} {
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s for %v: body = %q, want the server busy error",
route, err, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("for %v: body = %q, want the server busy error",
err, rec.Body.String())
}
}
}
// TestProcessingTimedOutAnswers503 checks that respondImageError and
// handleImageError, which answer the errors of the two image routes, answer
// 503 with the server busy error when the image service reports that
// processing stopped at the deadline of the request doing it, wrapped as the
// service wraps it. A request that waited for that processing gets this error
// while its own deadline may have time left. The request's source only goes
// into a log line, so it is left empty.
// TestProcessingTimedOutAnswers503 checks that respondImageError, which
// answers the errors of both image routes, answers 503 with the server busy
// error when the image service reports that processing stopped at the
// deadline of the request doing it, wrapped as the service wraps it. A request
// that waited for that processing gets this error while its own deadline may
// have time left. The request's source only goes into a log line, so it is
// left empty.
func TestProcessingTimedOutAnswers503(t *testing.T) {
t.Parallel()
h := &Handlers{log: slog.New(slog.DiscardHandler)}
err := fmt.Errorf("%w: failed to encode: %w",
imgcache.ErrProcessingTimedOut, context.DeadlineExceeded)
imageproxy.ErrProcessingTimedOut, context.DeadlineExceeded)
plain := httptest.NewRecorder()
h.respondImageError(plain, &imagerequest.Request{}, err)
rec := httptest.NewRecorder()
h.respondImageError(rec, &imagerequest.Request{}, err)
t.Logf("status = %d, body = %q", rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable)
}
for name, rec := range map[string]*httptest.ResponseRecorder{
"respondImageError": plain, "handleImageError": encrypted,
} {
t.Logf("%s: status = %d, body = %q", name, rec.Code, rec.Body.String())
if rec.Code != http.StatusServiceUnavailable {
t.Errorf("%s: status = %d, want %d",
name, rec.Code, http.StatusServiceUnavailable)
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("%s: body = %q, want the server busy error",
name, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "server busy, try again later") {
t.Errorf("body = %q, want the server busy error", rec.Body.String())
}
}
@@ -13,10 +13,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestUnsupportedSourceFormatAnswers415 checks that both image routes answer
// 415 with a clear error when the upstream answers with a content type pixa
// does not accept, such as SVG, and when orig is asked of a source in a
// format the processor cannot name, wrapped as the service wraps them.
// TestUnsupportedSourceFormatAnswers415 checks that respondImageError, which
// answers the errors of both image routes, answers 415 with a clear error when
// the upstream answers with a content type pixa does not accept, such as SVG,
// and when orig is asked of a source in a format the processor cannot name,
// wrapped as the service wraps them.
func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
t.Parallel()
@@ -31,26 +32,18 @@ func TestUnsupportedSourceFormatAnswers415(t *testing.T) {
fmt.Errorf("image processing failed: %w: unknown",
imageprocessor.ErrUnsupportedInputFormat),
} {
plain := httptest.NewRecorder()
h.respondImageError(plain, req, err)
rec := httptest.NewRecorder()
h.respondImageError(rec, req, err)
t.Logf("for %v: %d %s", err, rec.Code, rec.Body.String())
encrypted := httptest.NewRecorder()
h.handleImageError(encrypted, err)
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("for %v: status = %d, want %d",
err, rec.Code, http.StatusUnsupportedMediaType)
}
for route, rec := range map[string]*httptest.ResponseRecorder{
"HandleImage": plain, "HandleImageEnc": encrypted,
} {
t.Logf("%s for %v: %d %s", route, err, rec.Code, rec.Body.String())
if rec.Code != http.StatusUnsupportedMediaType {
t.Errorf("%s for %v: status = %d, want %d",
route, err, rec.Code, http.StatusUnsupportedMediaType)
}
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("%s for %v: body = %q, want the unsupported source format error",
route, err, rec.Body.String())
}
if !strings.Contains(rec.Body.String(), "unsupported source format") {
t.Errorf("for %v: body = %q, want the unsupported source format error",
err, rec.Body.String())
}
}
}
@@ -0,0 +1,211 @@
package handlers
import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/signature"
)
// Addresses the port tests below put on the allowlist.
const (
allowlistedIPv4 = "203.0.113.5"
allowlistedIPv6 = "2001:db8::1"
)
// newSignatureChecker returns Handlers that check signatures with
// testSigningKey and have allowlistHosts on their allowlist, as New builds
// them from signing_key and allowlist_hosts.
func newSignatureChecker(allowlistHosts ...string) *Handlers {
return &Handlers{
signer: signature.New(testSigningKey),
allowlist: allowlist.New(allowlistHosts),
}
}
func TestValidateSignature_NonAllowlistedHost_NoSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
// Should fail validation - not allowlisted and no signature
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
if err == nil {
t.Error("validateSignature() expected error for non-allowlisted " +
"host without signature")
}
}
func TestValidateSignature_NonAllowlistedHost_ValidSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
req := photoRequest(signedHost)
// Generate a valid signature
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(req, expires)
// Should pass validation
err := h.validateSignature(req, sig, expires)
if err != nil {
t.Errorf("validateSignature() error = %v", err)
}
}
func TestValidateSignature_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
req := photoRequest(signedHost)
// Generate an expired signature
expires := time.Now().Add(-time.Hour)
sig := signature.New(testSigningKey).Sign(req, expires)
// Should fail validation
err := h.validateSignature(req, sig, expires)
if err == nil {
t.Error("validateSignature() expected error for expired signature")
}
}
func TestValidateSignature_NonAllowlistedHost_InvalidSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost)
// Should fail validation
err := h.validateSignature(photoRequest(signedHost), "invalid-signature",
time.Now().Add(time.Hour))
if err == nil {
t.Error("validateSignature() expected error for invalid signature")
}
}
// TestValidateSignature_NoAllowlist checks that with no host on the
// allowlist, a request without a signature fails validation.
func TestValidateSignature_NoAllowlist(t *testing.T) {
t.Parallel()
h := newSignatureChecker()
err := h.validateSignature(photoRequest(signedHost), "", time.Time{})
if err == nil {
t.Error("validateSignature() expected error when no host is allowlisted")
}
}
// TestValidateSignature_SignatureExactHostMatch verifies that
// validateSignature enforces exact host matching for signatures. A
// signature for one host must not verify for a different host, even
// if they share a domain suffix.
func TestValidateSignature_SignatureExactHostMatch(t *testing.T) {
t.Parallel()
h := newSignatureChecker()
// Sign a request for signedHost
signedReq := photoRequest(signedHost)
expires := time.Now().Add(time.Hour)
sig := signature.New(testSigningKey).Sign(signedReq, expires)
// The original request should pass validation
t.Run("exact host passes", func(t *testing.T) {
t.Parallel()
err := h.validateSignature(signedReq, sig, expires)
if err != nil {
t.Errorf("validateSignature() exact host failed: %v", err)
}
})
// Try to reuse the signature with different hosts
tests := []struct {
name string
host string
}{
{"parent domain", "example.com"},
{"sibling subdomain", "images.example.com"},
{"deeper subdomain", "a." + signedHost},
{"evil suffix domain", signedHost + ".evil.com"},
{"prefixed host", "evil" + signedHost},
}
for _, tt := range tests {
t.Run(tt.name+" rejected", func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(tt.host), sig, expires)
if err == nil {
t.Errorf("validateSignature() should reject signature for "+
"host %q (signed for %q)", tt.host, signedHost)
}
})
}
}
// TestValidateSignature_HostWithPortNeedsSignature verifies that a host
// followed by a port, or by a colon with no port, matches no allowlist entry,
// so that a request for it without a signature is refused, while the same
// host with no port is allowlisted.
func TestValidateSignature_HostWithPortNeedsSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedHost, allowlistedIPv4, allowlistedIPv6)
tests := []struct {
host string
allowlisted bool
}{
{allowlistedHost, true},
{allowlistedHost + ":8443", false},
{allowlistedHost + ":443", false},
{allowlistedHost + ":", false},
{allowlistedIPv4, true},
{allowlistedIPv4 + ":8080", false},
{"[" + allowlistedIPv6 + "]", true},
{"[" + allowlistedIPv6 + "]:8443", false},
}
for _, tt := range tests {
t.Run(tt.host, func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(tt.host), "", time.Time{})
t.Logf("validateSignature() for host %q with no signature: %v",
tt.host, err)
if allowlisted := err == nil; allowlisted != tt.allowlisted {
t.Errorf("host %q served without a signature = %v, want %v",
tt.host, allowlisted, tt.allowlisted)
}
})
}
}
// TestValidateSignature_UnbracketedIPv6PortNeedsSignature verifies that an
// allowlisted IPv6 address written without brackets and followed by a port,
// or by a colon with no port, needs a signature. Go's URL type reads
// 2001:db8::1:8443 as the host 2001:db8::1 with the port 8443.
func TestValidateSignature_UnbracketedIPv6PortNeedsSignature(t *testing.T) {
t.Parallel()
h := newSignatureChecker(allowlistedIPv6)
for _, host := range []string{allowlistedIPv6 + ":8443", allowlistedIPv6 + ":"} {
t.Run(host, func(t *testing.T) {
t.Parallel()
err := h.validateSignature(photoRequest(host), "", time.Time{})
t.Logf("validateSignature() for host %q with no signature: %v",
host, err)
if err == nil {
t.Errorf("host %q served without a signature, want refused",
host)
}
})
}
}
@@ -51,12 +51,12 @@ func TestDialContextReplacesOnlyTheDialer(t *testing.T) {
t.Errorf("body = %q, want %q", body, imagePayload)
}
_, err := f.Fetch(testContext(t), "http://127.0.0.1/image")
_, err := f.Fetch(testContext(t), imageSourceOn("127.0.0.1"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(loopback URL) error = %v, want ErrSSRFBlocked", err)
}
_, err = f.Fetch(testContext(t), upstreamURL("/redirect/private"))
_, err = f.Fetch(testContext(t), upstreamSource("/redirect/private"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("Fetch(/redirect/private) error = %v, want ErrSSRFBlocked", err)
}
+58 -6
View File
@@ -14,6 +14,8 @@ import (
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// testPublicHost is a TEST-NET-1 (RFC 5737) literal. isPrivateIP treats it as
@@ -29,9 +31,9 @@ const imagePayload = "fake-jpeg-bytes"
// would mean SSRF protection let a forbidden target reach the transport.
var errUnexpectedDial = errors.New("unexpected dial target")
// upstreamURL builds a fetch URL on the fake public host for the given path.
func upstreamURL(path string) string {
return "http://" + testPublicHost + path
// upstreamSource returns the source at path on the fake public host.
func upstreamSource(path string) imagerequest.Source {
return imagerequest.Source{Host: testPublicHost, Path: path}
}
// recordingDialer records every address the transport asks it to dial and
@@ -158,7 +160,7 @@ func testContext(t *testing.T) context.Context {
func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
t.Helper()
res, err := f.Fetch(testContext(t), upstreamURL(path))
res, err := f.Fetch(testContext(t), upstreamSource(path))
if err != nil {
t.Fatalf("Fetch(%s) error = %v", path, err)
}
@@ -170,7 +172,7 @@ func fetchImage(t *testing.T, f *HTTPFetcher, path string) *FetchResult {
func fetchExpectError(t *testing.T, f *HTTPFetcher, path string) error {
t.Helper()
res, err := f.Fetch(testContext(t), upstreamURL(path))
res, err := f.Fetch(testContext(t), upstreamSource(path))
if err == nil {
_ = res.Content.Close()
@@ -222,7 +224,7 @@ func TestFetchRedirectToPrivateIPBlocked(t *testing.T) {
srv := startUpstream(t)
f, dialer := newServerFetcher(t, srv, nil)
_, err := f.Fetch(testContext(t), upstreamURL("/redirect/private"))
_, err := f.Fetch(testContext(t), upstreamSource("/redirect/private"))
if !errors.Is(err, ErrSSRFBlocked) {
t.Fatalf("Fetch() error = %v, want ErrSSRFBlocked", err)
}
@@ -590,3 +592,53 @@ func TestSSRFSafeDialerAllowsPublicTarget(t *testing.T) {
t.Errorf("public target was SSRF-blocked: %v", err)
}
}
// TestFetchSchemeFollowsAllowHTTP checks that the fetcher fetches a source
// over https, on port 443, unless AllowHTTP is set, and then over http, on
// port 80. The test server speaks plain HTTP, so only the fetch over http
// succeeds.
func TestFetchSchemeFollowsAllowHTTP(t *testing.T) {
t.Parallel()
tests := []struct {
allowHTTP bool
wantAddr string
}{
{false, net.JoinHostPort(testPublicHost, "443")},
{true, net.JoinHostPort(testPublicHost, "80")},
}
for _, tt := range tests {
t.Run(fmt.Sprintf("AllowHTTP %v", tt.allowHTTP), func(t *testing.T) {
t.Parallel()
srv := startUpstream(t)
dialer := &recordingDialer{target: srv.Listener.Addr().String()}
f := New(&Config{
Timeout: 30 * time.Second,
MaxResponseSize: 50 << 20,
AllowHTTP: tt.allowHTTP,
MaxConnectionsPerHost: 20,
MaxConnections: 64,
DialContext: dialer.dialContext,
})
res, err := f.Fetch(testContext(t), upstreamSource("/image"))
if err == nil {
_ = res.Content.Close()
}
dialed := dialer.dialedAddrs()
t.Logf("Fetch(): error %v, dialed %v", err, dialed)
if served := err == nil; served != tt.allowHTTP {
t.Errorf("served = %v, want %v", served, tt.allowHTTP)
}
if len(dialed) != 1 || dialed[0] != tt.wantAddr {
t.Errorf("dialed %v, want only %s", dialed, tt.wantAddr)
}
})
}
}
+21 -44
View File
@@ -20,6 +20,7 @@ import (
"time"
"github.com/go-chi/chi/v5/middleware"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// Fetcher configuration constants.
@@ -91,8 +92,8 @@ var (
// Fetcher retrieves content from upstream origins.
type Fetcher interface {
// Fetch retrieves content from the given URL.
Fetch(ctx context.Context, url string) (*FetchResult, error)
// Fetch retrieves the content of the source image.
Fetch(ctx context.Context, source imagerequest.Source) (*FetchResult, error)
}
// FetchResult contains the result of fetching from upstream.
@@ -139,7 +140,8 @@ type Config struct {
Timeout time.Duration
// MaxResponseSize is the maximum allowed response body size.
MaxResponseSize int64
// AllowHTTP allows non-TLS connections (for testing only).
// AllowHTTP makes the fetcher fetch sources over http instead of https,
// and follow redirects to http (for testing only).
AllowHTTP bool
// MaxConnectionsPerHost limits concurrent connections to each upstream host.
MaxConnectionsPerHost int
@@ -211,7 +213,7 @@ func New(config *Config) *HTTPFetcher {
}
// Validate the redirect target
err := validateURL(req.Context(), req.URL.String(), config.AllowHTTP)
err := validateURL(req.Context(), req.URL, config.AllowHTTP)
if err != nil {
return fmt.Errorf("redirect blocked: %w", err)
}
@@ -229,18 +231,22 @@ func New(config *Config) *HTTPFetcher {
}
}
// Fetch retrieves content from the given URL with SSRF protection. When
// MaxConnections fetches are already in progress, it waits up to
// ConnectionWaitTimeout for one to finish, then fails with
// ErrTooManyConnections.
func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) {
// Fetch retrieves the source image with SSRF protection, over https, or over
// http when AllowHTTP is set. When MaxConnections fetches are already in
// progress, it waits up to ConnectionWaitTimeout for one to finish, then
// fails with ErrTooManyConnections.
func (f *HTTPFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
) (*FetchResult, error) {
sourceURL := source.URL(f.config.AllowHTTP)
// Validate URL before making request
err := validateURL(ctx, url, f.config.AllowHTTP)
err := validateURL(ctx, sourceURL, f.config.AllowHTTP)
if err != nil {
return nil, err
}
release, err := f.acquireConnection(ctx, extractHost(url))
release, err := f.acquireConnection(ctx, source.Host)
if err != nil {
return nil, err
}
@@ -254,14 +260,9 @@ func (f *HTTPFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
}
}()
parsedURL, err := neturl.Parse(url)
if err != nil {
return nil, fmt.Errorf("failed to parse URL: %w", err)
}
req := &http.Request{
Method: http.MethodGet,
URL: parsedURL,
URL: sourceURL,
Header: make(http.Header),
}
@@ -455,19 +456,14 @@ func (f *HTTPFetcher) isAllowedContentType(contentType string) bool {
}
// validateURL checks if a URL is safe to fetch (not internal/private).
func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
if !allowHTTP && !strings.HasPrefix(rawURL, "https://") {
func validateURL(ctx context.Context, fetchURL *neturl.URL, allowHTTP bool) error {
if !allowHTTP && fetchURL.Scheme != "https" {
return ErrUnsupportedScheme
}
parsedURL, err := neturl.Parse(rawURL)
if err != nil {
return fmt.Errorf("%w: %w", ErrInvalidHost, err)
}
// Hostname drops any port, and the brackets of an IPv6 address whether
// or not a port follows them
host := parsedURL.Hostname()
host := fetchURL.Hostname()
if host == "" {
return ErrInvalidHost
}
@@ -493,25 +489,6 @@ func validateURL(ctx context.Context, rawURL string, allowHTTP bool) error {
return nil
}
// extractHost extracts the host from a URL string.
func extractHost(rawURL string) string {
// Simple extraction without full URL parsing
url := rawURL
if idx := strings.Index(url, "://"); idx != -1 {
url = url[idx+3:]
}
if idx := strings.Index(url, "/"); idx != -1 {
url = url[:idx]
}
if idx := strings.Index(url, "?"); idx != -1 {
url = url[:idx]
}
return url
}
// isLocalhost checks if the host is localhost.
func isLocalhost(host string) bool {
host = strings.ToLower(host)
@@ -5,9 +5,12 @@ import (
"errors"
"io"
"net"
"net/url"
"testing"
"testing/fstest"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// testHost is the hostname used by mock fetch tests.
@@ -49,33 +52,6 @@ func TestIsAllowedContentType(t *testing.T) {
}
}
func TestExtractHost(t *testing.T) {
t.Parallel()
tests := []struct {
url string
want string
}{
{"https://example.com/path", testHost},
{"http://example.com:8080/path", "example.com:8080"},
{"https://example.com", testHost},
{"https://example.com?q=1", testHost},
{"example.com/path", testHost},
{"", ""},
}
for _, tc := range tests {
t.Run(tc.url, func(t *testing.T) {
t.Parallel()
got := extractHost(tc.url)
if got != tc.want {
t.Errorf("extractHost(%q) = %q, want %q", tc.url, got, tc.want)
}
})
}
}
func TestIsLocalhost(t *testing.T) {
t.Parallel()
@@ -188,7 +164,7 @@ func TestIsPrivateIPJudgesEmbeddedIPv4Address(t *testing.T) {
func TestValidateURL_RejectsNonHTTPS(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "http://example.com/path", false)
err := validateURL(t.Context(), &url.URL{Scheme: "http", Host: testHost}, false)
if !errors.Is(err, ErrUnsupportedScheme) {
t.Errorf("validateURL http = %v, want ErrUnsupportedScheme", err)
}
@@ -198,7 +174,8 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
t.Parallel()
// Use a host that won't resolve (explicit .invalid TLD) so we don't hit DNS.
err := validateURL(t.Context(), "http://nonexistent.invalid/path", true)
err := validateURL(t.Context(),
&url.URL{Scheme: "http", Host: "nonexistent.invalid"}, true)
// We expect a host resolution error, not ErrUnsupportedScheme.
if errors.Is(err, ErrUnsupportedScheme) {
t.Error("validateURL with AllowHTTP should not return ErrUnsupportedScheme")
@@ -208,7 +185,7 @@ func TestValidateURL_AllowsHTTPWhenConfigured(t *testing.T) {
func TestValidateURL_RejectsLocalhost(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "https://localhost/path", false)
err := validateURL(t.Context(), &url.URL{Scheme: "https", Host: "localhost"}, false)
if !errors.Is(err, ErrSSRFBlocked) {
t.Errorf("validateURL localhost = %v, want ErrSSRFBlocked", err)
}
@@ -217,7 +194,7 @@ func TestValidateURL_RejectsLocalhost(t *testing.T) {
func TestValidateURL_EmptyHost(t *testing.T) {
t.Parallel()
err := validateURL(t.Context(), "https:///path", false)
err := validateURL(t.Context(), &url.URL{Scheme: "https"}, false)
if !errors.Is(err, ErrInvalidHost) {
t.Errorf("validateURL empty host = %v, want ErrInvalidHost", err)
}
@@ -232,7 +209,8 @@ func TestMockFetcher_FetchesFile(t *testing.T) {
m := NewMock(mockFS)
result, err := m.Fetch(context.Background(), "https://example.com/images/photo.jpg")
result, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/images/photo.jpg"})
if err != nil {
t.Fatalf("Fetch() error = %v", err)
}
@@ -263,7 +241,8 @@ func TestMockFetcher_MissingFileReturnsUpstreamError(t *testing.T) {
mockFS := fstest.MapFS{}
m := NewMock(mockFS)
_, err := m.Fetch(context.Background(), "https://example.com/missing.jpg")
_, err := m.Fetch(context.Background(),
imagerequest.Source{Host: testHost, Path: "/missing.jpg"})
if !errors.Is(err, ErrUpstreamError) {
t.Errorf("Fetch() error = %v, want ErrUpstreamError", err)
}
@@ -280,7 +259,7 @@ func TestMockFetcher_RespectsContextCancellation(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
cancel()
_, err := m.Fetch(ctx, "https://example.com/photo.jpg")
_, err := m.Fetch(ctx, imagerequest.Source{Host: testHost, Path: "/photo.jpg"})
if !errors.Is(err, context.Canceled) {
t.Errorf("Fetch() error = %v, want context.Canceled", err)
}
@@ -8,14 +8,20 @@ import (
"sync"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// imageURLOnPort is the fake upstream's image route on testPublicHost at
// imageSourceOn is the fake upstream's image route on host.
func imageSourceOn(host string) imagerequest.Source {
return imagerequest.Source{Host: host, Path: "/image"}
}
// imageSourceOnPort is the fake upstream's image route on testPublicHost at
// port. Each port is a different host to the per-host limit, while the test
// dialer sends every port to the one test server.
func imageURLOnPort(port int) string {
return "http://" + net.JoinHostPort(testPublicHost, strconv.Itoa(port)) +
"/image"
func imageSourceOnPort(port int) imagerequest.Source {
return imageSourceOn(net.JoinHostPort(testPublicHost, strconv.Itoa(port)))
}
// TestFetchLimitsConnectionsToAllHostsTogether checks that MaxConnections
@@ -39,12 +45,12 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
first, err := f.Fetch(testContext(t), imageURLOnPort(81))
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
second, err := f.Fetch(testContext(t), imageURLOnPort(82))
second, err := f.Fetch(testContext(t), imageSourceOnPort(82))
if err != nil {
t.Fatalf("second Fetch() error = %v", err)
}
@@ -53,7 +59,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
start := time.Now()
_, err = f.Fetch(testContext(t), imageURLOnPort(83))
_, err = f.Fetch(testContext(t), imageSourceOnPort(83))
if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("third Fetch() error = %v, want ErrTooManyConnections", err)
}
@@ -72,7 +78,7 @@ func TestFetchLimitsConnectionsToAllHostsTogether(t *testing.T) {
t.Fatalf("close first body: %v", err)
}
third, err := f.Fetch(testContext(t), imageURLOnPort(83))
third, err := f.Fetch(testContext(t), imageSourceOnPort(83))
if err != nil {
t.Fatalf("Fetch() after a response was closed: error = %v", err)
}
@@ -99,7 +105,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
first, err := f.Fetch(testContext(t), imageURLOnPort(81))
first, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
@@ -109,7 +115,7 @@ func TestFetchFreesHostSlotWhenContextEndsWaitingForConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(82))
_, err = f.Fetch(ctx, imageSourceOnPort(82))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("second Fetch() error = %v, want context.DeadlineExceeded", err)
}
@@ -134,7 +140,7 @@ func TestFetchRemovesIdleHostSemaphores(t *testing.T) {
for i := range 100 {
wg.Go(func() {
res, err := f.Fetch(ctx, imageURLOnPort(1+i%50))
res, err := f.Fetch(ctx, imageSourceOnPort(1+i%50))
if err != nil {
t.Errorf("Fetch() error = %v", err)
@@ -173,12 +179,12 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
open, err := f.Fetch(testContext(t), imageURLOnPort(81))
open, err := f.Fetch(testContext(t), imageSourceOnPort(81))
if err != nil {
t.Fatalf("first Fetch() error = %v", err)
}
_, err = f.Fetch(testContext(t), imageURLOnPort(82))
_, err = f.Fetch(testContext(t), imageSourceOnPort(82))
if !errors.Is(err, ErrTooManyConnections) {
t.Fatalf("Fetch() from another host: error = %v, "+
"want ErrTooManyConnections", err)
@@ -187,7 +193,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel := context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(81))
_, err = f.Fetch(ctx, imageSourceOnPort(81))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from the busy host: error = %v, "+
"want context.DeadlineExceeded", err)
@@ -199,7 +205,7 @@ func TestFetchRemovesHostSemaphoreWhenNoConnection(t *testing.T) {
ctx, cancel = context.WithTimeout(t.Context(), 100*time.Millisecond)
defer cancel()
_, err = f.Fetch(ctx, imageURLOnPort(83))
_, err = f.Fetch(ctx, imageSourceOnPort(83))
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("Fetch() from a host with nothing open: error = %v, "+
"want context.DeadlineExceeded", err)
@@ -222,15 +228,15 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
t.Parallel()
cases := []struct {
name string
url string
want error
name string
source imagerequest.Source
want error
}{
{"upstream answers 500", upstreamURL("/status/500"), ErrUpstreamError},
{"upstream sends HTML", upstreamURL("/html"), ErrInvalidContentType},
{"upstream answers 500", upstreamSource("/status/500"), ErrUpstreamError},
{"upstream sends HTML", upstreamSource("/html"), ErrInvalidContentType},
// 198.51.100.7 (TEST-NET-2) passes the SSRF checks, and the test
// dialer refuses every host but testPublicHost.
{"connecting fails", "http://198.51.100.7/image", errUnexpectedDial},
{"connecting fails", imageSourceOn("198.51.100.7"), errUnexpectedDial},
}
for _, tc := range cases {
@@ -249,7 +255,7 @@ func TestFetchReleasesConnectionOnError(t *testing.T) {
f, _ := newServerFetcher(t, srv, cfg)
f.connectionWaitTimeout = 100 * time.Millisecond
_, err := f.Fetch(testContext(t), tc.url)
_, err := f.Fetch(testContext(t), tc.source)
if !errors.Is(err, tc.want) {
t.Fatalf("Fetch() error = %v, want %v", err, tc.want)
}
+9 -37
View File
@@ -7,14 +7,14 @@ import (
"io/fs"
"net/http"
"strings"
)
// errEmptyURLPath is returned when a mock URL has no usable path.
var errEmptyURLPath = errors.New("empty URL path")
"sneak.berlin/go/pixa/internal/imagerequest"
)
// MockFetcher implements Fetcher using an embedded filesystem.
// Files are organized as: hostname/path/to/file.ext
// URLs like https://example.com/images/photo.jpg map to example.com/images/photo.jpg.
// The source with host example.com and path /images/photo.jpg maps to
// example.com/images/photo.jpg; the source query is ignored.
type MockFetcher struct {
fs fs.FS
}
@@ -24,8 +24,10 @@ func NewMock(fsys fs.FS) *MockFetcher {
return &MockFetcher{fs: fsys}
}
// Fetch retrieves content from the mock filesystem.
func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, error) {
// Fetch retrieves the source's content from the mock filesystem.
func (m *MockFetcher) Fetch(
ctx context.Context, source imagerequest.Source,
) (*FetchResult, error) {
// Check context cancellation
select {
case <-ctx.Done():
@@ -33,11 +35,7 @@ func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
default:
}
// Parse URL to get filesystem path
path, err := urlToFSPath(url)
if err != nil {
return nil, err
}
path := source.Host + source.Path
// Open the file
f, err := m.fs.Open(path)
@@ -68,32 +66,6 @@ func (m *MockFetcher) Fetch(ctx context.Context, url string) (*FetchResult, erro
}, nil
}
// urlToFSPath converts a URL to a filesystem path.
// https://example.com/images/photo.jpg -> example.com/images/photo.jpg
func urlToFSPath(rawURL string) (string, error) {
// Strip scheme
url := rawURL
if idx := strings.Index(url, "://"); idx != -1 {
url = url[idx+3:]
}
// Remove query string
if idx := strings.Index(url, "?"); idx != -1 {
url = url[:idx]
}
// Remove fragment
if idx := strings.Index(url, "#"); idx != -1 {
url = url[:idx]
}
if url == "" {
return "", errEmptyURLPath
}
return url, nil
}
// detectContentTypeFromPath returns the MIME type based on file extension.
func detectContentTypeFromPath(path string) string {
path = strings.ToLower(path)
@@ -34,7 +34,7 @@ func TestFetchSendsRequestID(t *testing.T) {
ctx := context.WithValue(testContext(t), middleware.RequestIDKey, requestID)
res, err := f.Fetch(ctx, upstreamURL("/image"))
res, err := f.Fetch(ctx, upstreamSource("/image"))
if err != nil {
t.Fatalf("Fetch() error = %v", err)
}
@@ -0,0 +1,100 @@
package imageproxy
import (
"bytes"
"os"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// TestProxy_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestProxy_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := variantFilePath(svc, imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"),
imgcache.StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestProxy_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestProxy_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.db, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.db, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
@@ -1,4 +1,4 @@
package imgcache
package imageproxy
import (
"bytes"
@@ -36,7 +36,7 @@ type heldFetcher struct {
}
func (f *heldFetcher) Fetch(
ctx context.Context, url string,
ctx context.Context, source imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1)
@@ -48,7 +48,7 @@ func (f *heldFetcher) Fetch(
return nil, ctx.Err()
}
return f.upstream.Fetch(ctx, url)
return f.upstream.Fetch(ctx, source)
}
// releaseFetches lets every held fetch, and every later one, go on.
@@ -56,15 +56,15 @@ func (f *heldFetcher) releaseFetches() {
f.releaseOnce.Do(func() { close(f.release) })
}
// setupHeldFetchService returns a test service whose fetches go through a
// setupHeldFetchProxy returns a test proxy whose fetches go through a
// heldFetcher. Its database is limited to one connection: each connection to
// an in-memory SQLite database opens a new, empty one, so requests running at
// once must share the connection that holds the schema.
func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher) {
func setupHeldFetchProxy(t *testing.T) (*testProxy, *TestFixtures, *heldFetcher) {
t.Helper()
svc, fixtures := SetupTestService(t)
svc.cache.db.SetMaxOpenConns(1)
svc, fixtures := setupTestProxy(t)
svc.db.SetMaxOpenConns(1)
fetcher := &heldFetcher{
upstream: svc.fetcher,
@@ -79,8 +79,8 @@ func setupHeldFetchService(t *testing.T) (*Service, *TestFixtures, *heldFetcher)
}
// photoVariant asks for the test photo, 100x100, at 50x25 as a JPEG of the
// given quality and fit mode. Each call returns a new request, as Get writes
// to the request it is given.
// given quality and fit mode. Each call returns a new request, which a test
// may change.
func photoVariant(
fixtures *TestFixtures, quality int, fit imagerequest.Fit,
) *imagerequest.Request {
@@ -107,7 +107,7 @@ type getResult struct {
// startGet calls Get in a goroutine of its own and delivers what it returned
// on the channel.
func startGet(
ctx context.Context, svc *Service, req *imagerequest.Request,
ctx context.Context, svc *testProxy, req *imagerequest.Request,
) <-chan getResult {
results := make(chan getResult, 1)
@@ -139,14 +139,14 @@ func jpegSize(data []byte) (int, int) {
return config.Width, config.Height
}
// TestService_Get_ConcurrentMissesShareOneFetch starts several requests for
// TestProxy_Get_ConcurrentMissesShareOneFetch starts several requests for
// one uncached variant while the first one's fetch is held. Between them they
// must fetch the source once and transcode it once, every one must be answered
// with the same 50x25 JPEG, and each must count one miss.
func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
func TestProxy_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
const requests = 8
@@ -183,7 +183,7 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
t.Errorf("%d requests made %d upstream fetches, want 1", requests, fetches)
}
// NewTestFS builds the same files the test service's fetcher serves.
// NewTestFS builds the same files the test proxy's fetcher serves.
testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -193,18 +193,18 @@ func TestService_Get_ConcurrentMissesShareOneFetch(t *testing.T) {
want := counterValues{0, requests, 1, int64(len(photo)), 1}
if got := readCounters(svc.cache); got != want {
if got := readCounters(t, svc); got != want {
t.Errorf("counters = %+v, want %+v", got, want)
}
}
// TestService_Get_ConcurrentVariantsStayApart requests three variants of the
// TestProxy_Get_ConcurrentVariantsStayApart requests three variants of the
// test photo at once that differ only in quality or fit. Each must be made by
// a fetch and a transcode of its own, and each answer must be its own variant.
func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
func TestProxy_Get_ConcurrentVariantsStayApart(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
cover := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
lowQuality := startGet(t.Context(), svc,
@@ -253,14 +253,14 @@ func TestService_Get_ConcurrentVariantsStayApart(t *testing.T) {
}
}
// TestService_Get_WaiterStopsWhenItsContextEnds has a second request for a
// TestProxy_Get_WaiterStopsWhenItsContextEnds has a second request for a
// variant join the first one's held fetch, then ends the second request's
// context. The second request must return at once with the context's error,
// while the fetch is still held, and the first must still be answered.
func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
func TestProxy_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
first := startGet(t.Context(), svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -298,15 +298,15 @@ func TestService_Get_WaiterStopsWhenItsContextEnds(t *testing.T) {
}
}
// TestService_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// TestProxy_Get_FirstRequestLeavingKeepsTheWork ends the context of the
// request whose fetch is held, after a second request has joined it. The fetch
// and transcode must go on and answer the second request. The first request
// waits for its own work, as every request did before misses were shared, and
// is answered too.
func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
func TestProxy_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
firstCtx, cancelFirst := context.WithCancel(t.Context())
first := startGet(firstCtx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
@@ -337,11 +337,11 @@ func TestService_Get_FirstRequestLeavingKeepsTheWork(t *testing.T) {
}
}
// TestService_Get_ConcurrentMissesShareAFailure has several requests for an
// TestProxy_Get_ConcurrentMissesShareAFailure has several requests for an
// image that cannot be served arrive while its fetch is held: the one fetch
// answers all of them with its error. The request after them is answered from
// the negative cache, where the failure is kept.
func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
func TestProxy_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Parallel()
tests := []struct {
@@ -362,7 +362,7 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
request := func() *imagerequest.Request {
req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -398,12 +398,12 @@ func TestService_Get_ConcurrentMissesShareAFailure(t *testing.T) {
}
}
// TestService_Get_EndedRequestFetchesNothing checks that a request whose
// TestProxy_Get_EndedRequestFetchesNothing checks that a request whose
// context has already ended when it misses the cache starts no fetch.
func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
func TestProxy_Get_EndedRequestFetchesNothing(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
ctx, cancel := context.WithCancel(t.Context())
cancel()
@@ -418,14 +418,14 @@ func TestService_Get_EndedRequestFetchesNothing(t *testing.T) {
}
}
// TestService_Get_ReturnsByItsDeadline gives a request a deadline and holds
// TestProxy_Get_ReturnsByItsDeadline gives a request a deadline and holds
// its fetch until the fetch's context ends, as the fetcher does while it waits
// for a free connection to the host. The request must return by its deadline
// with the deadline's error.
func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
func TestProxy_Get_ReturnsByItsDeadline(t *testing.T) {
t.Parallel()
svc, fixtures, _ := setupHeldFetchService(t)
svc, fixtures, _ := setupHeldFetchProxy(t)
const timeout = 200 * time.Millisecond
@@ -444,16 +444,16 @@ func TestService_Get_ReturnsByItsDeadline(t *testing.T) {
}
}
// TestService_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// TestProxy_Get_WaiterGetsProcessingTimedOut has a request with no deadline
// join the held fetch of a first request with a short one. With the only
// processing slot held, the first request's transcode waits for it until the
// first request's deadline ends, and stops then. The waiting request, whose
// own context has not ended, must get ErrProcessingTimedOut, which the image
// routes answer 503 as when pixa is busy.
func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
func TestProxy_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
@@ -489,18 +489,18 @@ func TestService_Get_WaiterGetsProcessingTimedOut(t *testing.T) {
type panickingFetcher struct{}
func (panickingFetcher) Fetch(
context.Context, string,
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
panic("upstream fetcher panicked")
}
// TestService_Get_PanicBecomesAnError checks that a panic while a variant is
// TestProxy_Get_PanicBecomesAnError checks that a panic while a variant is
// being made reaches its request as an error naming the panic, instead of
// being raised again.
func TestService_Get_PanicBecomesAnError(t *testing.T) {
func TestProxy_Get_PanicBecomesAnError(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{}
var err error
@@ -520,13 +520,13 @@ func TestService_Get_PanicBecomesAnError(t *testing.T) {
}
}
// TestService_Get_PanicIsReportedToSentry checks that a panic while a variant
// TestProxy_Get_PanicIsReportedToSentry checks that a panic while a variant
// is being made is reported through the Sentry hub on the request's context,
// where the Sentry middleware puts one when sentry_dsn is set.
func TestService_Get_PanicIsReportedToSentry(t *testing.T) {
func TestProxy_Get_PanicIsReportedToSentry(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
svc.fetcher = panickingFetcher{}
transport := &sentry.MockTransport{}
@@ -1,4 +1,4 @@
package imgcache
package imageproxy
import (
"fmt"
@@ -1,4 +1,7 @@
package imgcache
// Package imageproxy fetches, transforms, caches and serves images. Its
// Proxy serves an image from the cache when it can, and otherwise makes it
// from the cached or fetched source image.
package imageproxy
import (
"bytes"
@@ -7,53 +10,40 @@ import (
"fmt"
"io"
"log/slog"
"net/url"
"runtime/debug"
"strings"
"time"
"github.com/dustin/go-humanize"
"github.com/getsentry/sentry-go"
"github.com/go-chi/chi/v5/middleware"
"golang.org/x/sync/singleflight"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/magic"
"sneak.berlin/go/pixa/internal/signature"
)
// Service orchestrates the cache, fetcher, and processor.
type Service struct {
cache *Cache
// Proxy orchestrates the cache, fetcher, and processor.
type Proxy struct {
cache *imgcache.Cache
fetcher httpfetcher.Fetcher
processor *imageprocessor.ImageProcessor
signer *signature.Signer
allowlist *allowlist.HostAllowList
log *slog.Logger
allowHTTP bool
maxResponseSize int64
// variantsInProgress lets the requests that miss the same variant at the
// same time share one fetch and one transcode.
variantsInProgress singleflight.Group
}
// ServiceConfig holds configuration for the image service.
type ServiceConfig struct {
// Config holds the proxy's configuration.
type Config struct {
// Cache is the cache instance
Cache *Cache
Cache *imgcache.Cache
// Fetcher fetches source images from upstream hosts
Fetcher httpfetcher.Fetcher
// Processor converts and resizes the images
Processor *imageprocessor.ImageProcessor
// SigningKey is the HMAC signing key (empty disables signing)
SigningKey string
// Allowlist is the list of hosts that don't require signatures
Allowlist []string
// AllowHTTP makes the service fetch sources over http instead of
// https (testing only), so the fetcher must allow http too
AllowHTTP bool
// MaxResponseSize is the largest source accepted, in bytes; a cached
// source larger than this is fetched again
MaxResponseSize int64
@@ -61,22 +51,15 @@ type ServiceConfig struct {
Logger *slog.Logger
}
// Static errors for service construction.
var (
errCacheRequired = errors.New("cache is required")
errSigningKeyRequired = errors.New("signing key is required")
)
// errCacheRequired is returned when the proxy is built without a cache.
var errCacheRequired = errors.New("cache is required")
// NewService creates a new image service.
func NewService(cfg *ServiceConfig) (*Service, error) {
// NewProxy creates a new image proxy.
func NewProxy(cfg *Config) (*Proxy, error) {
if cfg.Cache == nil {
return nil, errCacheRequired
}
if cfg.SigningKey == "" {
return nil, errSigningKeyRequired
}
log := cfg.Logger
if log == nil {
log = slog.Default()
@@ -89,14 +72,11 @@ func NewService(cfg *ServiceConfig) (*Service, error) {
return nil, err
}
return &Service{
return &Proxy{
cache: cfg.Cache,
fetcher: cfg.Fetcher,
processor: cfg.Processor,
signer: signature.New(cfg.SigningKey),
allowlist: allowlist.New(cfg.Allowlist),
log: log,
allowHTTP: cfg.AllowHTTP,
maxResponseSize: cfg.MaxResponseSize,
}, nil
}
@@ -110,28 +90,49 @@ var ErrNegativeCached = errors.New("request is in negative cache (recently faile
var ErrProcessingTimedOut = errors.New(
"image processing stopped at the deadline of the request doing it")
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
// Get retrieves a processed image, fetching and processing if necessary.
// While its source URL is in the negative cache, a cached variant is still
// served and a variant not in the cache is still made from a cached source;
// only a request that needs the source fetched is refused.
//
// A cache hit opens the variant file and makes no database call: its access
// time is handed to the goroutine that StartPendingAccessTimeWrites starts,
// without waiting for it, and its hit is counted in memory. A variant file
// that cannot be served, such as one whose first bytes are not those of an
// output format, is a miss, and the variant is made again.
func (s *Service) Get(
// A cache hit opens the variant file and makes no database call: the cache
// hands its access time to the goroutine that writes access times, without
// waiting for it, and its hit is counted in memory. A variant file that
// cannot be served, such as one whose first bytes are not those of an output
// format, is a miss, and the variant is made again.
func (p *Proxy) Get(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// Propagate AllowHTTP setting to the request
req.AllowHTTP = s.allowHTTP
cacheKey := imagerequest.CacheKey(req)
reader, size, contentType, err := s.cache.GetVariant(cacheKey)
reader, size, contentType, err := p.cache.Variant(cacheKey)
if err == nil {
s.cache.touchVariant(cacheKey)
s.cache.IncrementStats(true, 0)
p.cache.IncrementStats(true, 0)
return &ImageResponse{
Content: reader,
@@ -142,17 +143,17 @@ func (s *Service) Get(
}, nil
}
if !errors.Is(err, ErrNotFound) {
s.log.Warn("cached variant not served, making it again",
if !errors.Is(err, imgcache.ErrNotFound) {
p.log.Warn("cached variant not served, making it again",
"cache_key", cacheKey, "error", err)
}
// Cache miss - get the variant, processed once for all the requests that
// miss it at the same time, then count this request's miss, also when it
// failed or the request context has ended meanwhile
response, err := s.processOrWait(ctx, req)
response, err := p.processOrWait(ctx, req)
s.cache.IncrementStats(false, 0)
p.cache.IncrementStats(false, 0)
if err != nil {
return nil, err
@@ -165,35 +166,8 @@ func (s *Service) Get(
// WaitForProcessing waits until no image is being processed, or until ctx
// ends, and returns how many images were still being processed then.
func (s *Service) WaitForProcessing(ctx context.Context) int {
return s.processor.WaitForProcessing(ctx)
}
// ValidateRequest validates the request signature if required.
func (s *Service) ValidateRequest(req *imagerequest.Request) error {
// Check if host is allowed (no signature required)
sourceURL := req.SourceURL()
parsedURL, err := url.Parse(sourceURL)
if err != nil {
return fmt.Errorf("invalid source URL: %w", err)
}
// An allowlist entry names no port, so a host followed by ":", with or
// without a port, needs a signature. The port is read from the parsed
// URL, as the fetcher reads it: in an IPv6 address written without
// brackets, such as 2001:db8::1:8443, it is what follows the last ":".
// This is checked here and not in allowlist.IsAllowed, which also matches
// referer_blocklist, where a Referer must be blocked whatever its port.
hostHasPort := parsedURL.Port() != "" ||
strings.HasSuffix(parsedURL.Host, ":")
if !hostHasPort && s.allowlist.IsAllowed(parsedURL) {
return nil
}
// Signature required for non-allowed hosts
return s.signer.Verify(req)
func (p *Proxy) WaitForProcessing(ctx context.Context) int {
return p.processor.WaitForProcessing(ctx)
}
// errPanicked is returned when processing a variant panicked.
@@ -205,7 +179,7 @@ var errPanicked = errors.New("panic while processing image")
// and take no upstream connection or processing slot. The processing ignores
// the first request's cancellation, so the others are still served if that
// client goes away, but keeps its deadline.
func (s *Service) processOrWait(
func (p *Proxy) processOrWait(
ctx context.Context, req *imagerequest.Request,
) (*ImageResponse, error) {
// A request that has already ended starts no processing
@@ -219,7 +193,7 @@ func (s *Service) processOrWait(
// only when no other request is processing the variant
processing := make(chan struct{})
results := s.variantsInProgress.DoChan(string(cacheKey),
results := p.variantsInProgress.DoChan(string(cacheKey),
func() (_ any, err error) {
close(processing)
@@ -230,7 +204,7 @@ func (s *Service) processOrWait(
defer func() {
recovered := recover()
if recovered != nil {
s.log.Error("panic while processing image",
p.log.Error("panic while processing image",
"host", req.Source.Host, "path", req.Source.Path,
"panic", recovered, "stack", string(debug.Stack()))
@@ -245,7 +219,7 @@ func (s *Service) processOrWait(
processingCtx, cancel := withoutCancelKeepingDeadline(ctx)
defer cancel()
return s.processFromSourceOrFetch(processingCtx, req, cacheKey)
return p.processFromSourceOrFetch(processingCtx, req, cacheKey)
})
var result singleflight.Result
@@ -294,25 +268,31 @@ func withoutCancelKeepingDeadline(
return context.WithDeadline(context.WithoutCancel(ctx), deadline)
}
// loadCachedSource opens source content from cache, without reading it, and
// returns it with its size; nil if the cached data is unavailable, empty or
// exceeds maxResponseSize.
func (s *Service) loadCachedSource(
sourceKey imagerequest.SourceKey,
// loadCachedSource opens src's cached source, without reading it, and
// returns it with its size; nil if it is not in the cache, cannot be read,
// is empty or exceeds maxResponseSize.
func (p *Proxy) loadCachedSource(
ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64) {
reader, size, err := s.cache.GetSourceContent(sourceKey)
reader, size, err := p.cache.Source(ctx, src)
if errors.Is(err, imgcache.ErrNotFound) {
return nil, 0
}
if err != nil {
s.log.Warn("failed to load cached source, fetching", "error", err)
p.log.Warn("failed to load cached source, fetching", "error", err)
return nil, 0
}
if size > s.maxResponseSize {
p.log.Debug("using cached source", "source_key", src.Key())
if size > p.maxResponseSize {
_ = reader.Close()
s.log.Warn("cached source exceeds max response size, discarding",
"source_key", sourceKey,
"max_bytes", s.maxResponseSize,
p.log.Warn("cached source exceeds max response size, discarding",
"source_key", src.Key(),
"max_bytes", p.maxResponseSize,
)
return nil, 0
@@ -329,30 +309,16 @@ func (s *Service) loadCachedSource(
// processFromSourceOrFetch processes an image, using cached source content
// if available.
func (s *Service) processFromSourceOrFetch(
func (p *Proxy) processFromSourceOrFetch(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check if we have cached source content
sourceKey, err := s.cache.LookupSource(ctx, req)
if err != nil {
s.log.Warn("source lookup failed", "error", err)
}
var (
source io.ReadCloser
sourceSize int64
)
if sourceKey != "" {
s.log.Debug("using cached source", "source_key", sourceKey)
source, sourceSize = s.loadCachedSource(sourceKey)
}
source, sourceSize := p.loadCachedSource(ctx, req.Source)
// Fetch from upstream if we don't have source data or it's empty
if source == nil {
return s.fetchAndProcess(ctx, req, cacheKey)
return p.fetchAndProcess(ctx, req, cacheKey)
}
defer func() { _ = source.Close() }()
@@ -360,26 +326,26 @@ func (s *Service) processFromSourceOrFetch(
// Process using cached source; nothing was fetched from upstream. The
// image processor reads the source only once it has a processing slot,
// so a request waiting for one holds none of it in memory.
return s.processAndStore(ctx, req, cacheKey, source, sourceSize)
return p.processAndStore(ctx, req, cacheKey, source, sourceSize)
}
// fetchAndProcess fetches from upstream, processes, and caches the result.
// It counts the fetch with the bytes read from upstream, including when
// reading the response or a later step fails. It fetches nothing while the
// source URL is in the negative cache.
func (s *Service) fetchAndProcess(
func (p *Proxy) fetchAndProcess(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
) (*processedVariant, error) {
// Check negative cache - skip fetching for recently-failed URLs
negHit, err := s.cache.checkNegativeCache(ctx, req)
negHit, err := p.cache.Failure(ctx, req.Source)
if err != nil {
s.log.Warn("negative cache check failed", "error", err)
p.log.Warn("negative cache check failed", "error", err)
}
if negHit {
s.log.Debug("negative cache hit",
p.log.Debug("negative cache hit",
"host", req.Source.Host,
"path", req.Source.Path,
)
@@ -388,15 +354,14 @@ func (s *Service) fetchAndProcess(
}
// Fetch from upstream
sourceURL := req.SourceURL()
p.log.Debug("fetching from upstream", "host", req.Source.Host,
"path", req.Source.Path, "query", req.Source.Query)
s.log.Debug("fetching from upstream", "url", sourceURL)
fetchResult, err := s.fetcher.Fetch(ctx, sourceURL)
fetchResult, err := p.fetcher.Fetch(ctx, req.Source)
if err != nil {
// Store negative cache for certain errors
if isNegativeCacheable(err) {
s.storeNegative(ctx, req)
p.storeNegative(ctx, req)
}
return nil, fmt.Errorf("upstream fetch failed: %w", err)
@@ -413,7 +378,7 @@ func (s *Service) fetchAndProcess(
fetchBytes := int64(len(sourceData))
// Counted also when the request context has ended meanwhile
s.cache.IncrementUpstreamFetch(fetchBytes)
p.cache.IncrementUpstreamFetch(fetchBytes)
if err != nil {
return nil, fmt.Errorf("failed to read upstream response: %w", err)
@@ -429,7 +394,7 @@ func (s *Service) fetchAndProcess(
}
// Log upstream fetch details
s.log.Info("upstream fetched",
p.log.Info("upstream fetched",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
@@ -447,22 +412,22 @@ func (s *Service) fetchAndProcess(
// error status does.
err = magic.ValidateMagicBytes(sourceData, fetchResult.ContentType)
if err != nil {
s.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
p.log.Warn("source refused", "request_id", middleware.GetReqID(ctx),
"host", req.Source.Host, "path", req.Source.Path,
"content_type", fetchResult.ContentType, "error", err)
s.storeNegative(ctx, req)
p.storeNegative(ctx, req)
return nil, fmt.Errorf("content validation failed: %w", err)
}
// Store source content
_, err = s.cache.StoreSource(ctx, req, bytes.NewReader(sourceData))
_, err = p.cache.PutSource(ctx, req.Source, bytes.NewReader(sourceData))
if err != nil {
s.log.Warn("failed to store source content", "error", err)
p.log.Warn("failed to store source content", "error", err)
// Continue even if caching fails
}
return s.processAndStore(
return p.processAndStore(
ctx, req, cacheKey, bytes.NewReader(sourceData), fetchBytes,
)
}
@@ -470,10 +435,10 @@ func (s *Service) fetchAndProcess(
// storeNegative stores a negative cache entry for req's source, whose fetch
// failed, so the source is not fetched again until the entry expires. A
// failure to store the entry is only logged.
func (s *Service) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := s.cache.StoreNegative(ctx, req)
func (p *Proxy) storeNegative(ctx context.Context, req *imagerequest.Request) {
err := p.cache.PutFailure(ctx, req.Source)
if err != nil {
s.log.Warn("failed to store negative cache entry",
p.log.Warn("failed to store negative cache entry",
"host", req.Source.Host, "path", req.Source.Path, "error", err)
}
}
@@ -488,7 +453,7 @@ type processedVariant struct {
// processAndStore processes the image read from source and stores the
// result.
func (s *Service) processAndStore(
func (p *Proxy) processAndStore(
ctx context.Context,
req *imagerequest.Request,
cacheKey imagerequest.VariantKey,
@@ -500,7 +465,7 @@ func (s *Service) processAndStore(
// ctx has the deadline of the request doing the processing, which can
// end while a request waiting for its image has time left
processResult, err := s.processor.Process(ctx, source, &req.Transform)
processResult, err := p.processor.Process(ctx, source, &req.Transform)
if errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("%w: %w", ErrProcessingTimedOut, err)
}
@@ -512,7 +477,7 @@ func (s *Service) processAndStore(
processDuration := time.Since(processStart)
// Counted also when the request context has ended meanwhile
s.cache.IncrementTransformCount()
p.cache.IncrementTransformCount()
// Read processed content
processedData, err := io.ReadAll(processResult.Content)
@@ -531,7 +496,7 @@ func (s *Service) processAndStore(
sizePercent = float64(outputSize) / float64(fetchBytes) * 100.0
}
s.log.Info("image converted",
p.log.Info("image converted",
"request_id", middleware.GetReqID(ctx),
"host", req.Source.Host,
"path", req.Source.Path,
@@ -550,9 +515,9 @@ func (s *Service) processAndStore(
)
// Store variant to cache
err = s.cache.StoreVariant(ctx, cacheKey, bytes.NewReader(processedData))
err = p.cache.PutVariant(ctx, cacheKey, bytes.NewReader(processedData))
if err != nil {
s.log.Warn("failed to store variant", "error", err)
p.log.Warn("failed to store variant", "error", err)
// Continue even if caching fails
}
@@ -1,4 +1,4 @@
package imgcache
package imageproxy
import (
"context"
@@ -9,20 +9,18 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/magic"
"sneak.berlin/go/pixa/internal/signature"
)
// Test data literals used repeatedly in this file (goconst).
const (
testPathPhoto = "/images/photo.jpg"
testPathUpload = "/uploads/image.jpg"
testSigningKey = "test-signing-key-12345"
)
func TestService_Get_AllowlistedHost(t *testing.T) {
func TestProxy_Get_GoodHost(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
@@ -60,36 +58,12 @@ func TestService_Get_AllowlistedHost(t *testing.T) {
}
}
func TestService_Get_NonAllowlistedHost_NoSignature(t *testing.T) {
// TestProxy_Get_OtherHost checks that the proxy serves a second host as it
// serves GoodHost.
func TestProxy_Get_OtherHost(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t, WithSigningKey("test-key"))
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Should fail validation - not allowlisted and no signature
err := svc.ValidateRequest(req)
if err == nil {
t.Error("ValidateRequest() expected error for non-allowlisted host without signature")
}
}
func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
@@ -105,17 +79,6 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
},
}
// Generate a valid signature
signer := signature.New(signingKey)
req.Expires = time.Now().Add(time.Hour)
req.Signature = signer.Sign(req)
// Should pass validation
err := svc.ValidateRequest(req)
if err != nil {
t.Errorf("ValidateRequest() error = %v", err)
}
// Should fetch successfully
resp, err := svc.Get(ctx, req)
if err != nil {
@@ -134,248 +97,10 @@ func TestService_Get_NonAllowlistedHost_ValidSignature(t *testing.T) {
}
}
func TestService_Get_NonAllowlistedHost_ExpiredSignature(t *testing.T) {
func TestProxy_Get_InvalidFile(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Generate an expired signature
signer := signature.New(signingKey)
req.Expires = time.Now().Add(-time.Hour) // Already expired
req.Signature = signer.Sign(req)
// Should fail validation
err := svc.ValidateRequest(req)
if err == nil {
t.Error("ValidateRequest() expected error for expired signature")
}
}
func TestService_Get_NonAllowlistedHost_InvalidSignature(t *testing.T) {
t.Parallel()
signingKey := testSigningKey
svc, fixtures := SetupTestService(t, WithSigningKey(signingKey))
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
// Set an invalid signature
req.Signature = "invalid-signature"
req.Expires = time.Now().Add(time.Hour)
// Should fail validation
err := svc.ValidateRequest(req)
if err == nil {
t.Error("ValidateRequest() expected error for invalid signature")
}
}
// TestService_ValidateRequest_SignatureExactHostMatch verifies that
// ValidateRequest enforces exact host matching for signatures. A
// signature for one host must not verify for a different host, even
// if they share a domain suffix.
func TestService_ValidateRequest_SignatureExactHostMatch(t *testing.T) {
t.Parallel()
signingKey := "test-signing-key-must-be-32-chars"
svc, _ := SetupTestService(t,
WithSigningKey(signingKey),
WithNoAllowlist(),
)
signer := signature.New(signingKey)
// Sign a request for "cdn.example.com"
signedReq := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHostCDN,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: time.Now().Add(time.Hour),
}
signedReq.Signature = signer.Sign(signedReq)
// The original request should pass validation
t.Run("exact host passes", func(t *testing.T) {
t.Parallel()
err := svc.ValidateRequest(signedReq)
if err != nil {
t.Errorf("ValidateRequest() exact host failed: %v", err)
}
})
// Try to reuse the signature with different hosts
tests := []struct {
name string
host string
}{
{"parent domain", testHostExample},
{"sibling subdomain", "images.example.com"},
{"deeper subdomain", "a.cdn.example.com"},
{"evil suffix domain", "cdn.example.com.evil.com"},
{"prefixed host", "evilcdn.example.com"},
}
for _, tt := range tests {
t.Run(tt.name+" rejected", func(t *testing.T) {
t.Parallel()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: tt.host,
Path: signedReq.Source.Path,
Query: signedReq.Source.Query,
},
Transform: signedReq.Transform,
Expires: signedReq.Expires,
Signature: signedReq.Signature,
}
err := svc.ValidateRequest(req)
if err == nil {
t.Errorf(
"ValidateRequest() should reject signature for host %q (signed for %q)",
tt.host, signedReq.Source.Host)
}
})
}
}
// TestService_ValidateRequest_HostWithPortNeedsSignature verifies that a host
// followed by a port, or by a colon with no port, matches no allowlist entry,
// so that a request for it without a signature is refused, while the same
// host with no port is allowlisted.
func TestService_ValidateRequest_HostWithPortNeedsSignature(t *testing.T) {
t.Parallel()
svc, _ := SetupTestService(t,
WithSigningKey(testSigningKey),
WithAllowlist(testHostCDN, "203.0.113.5", "2001:db8::1"),
)
tests := []struct {
host string
allowlisted bool
}{
{testHostCDN, true},
{"cdn.example.com:8443", false},
{"cdn.example.com:443", false},
{"cdn.example.com:", false},
{"203.0.113.5", true},
{"203.0.113.5:8080", false},
{"[2001:db8::1]", true},
{"[2001:db8::1]:8443", false},
}
for _, tt := range tests {
t.Run(tt.host, func(t *testing.T) {
t.Parallel()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: tt.host,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
t.Logf("ValidateRequest() for host %q with no signature: %v",
tt.host, err)
if allowlisted := err == nil; allowlisted != tt.allowlisted {
t.Errorf("host %q served without a signature = %v, want %v",
tt.host, allowlisted, tt.allowlisted)
}
})
}
}
// TestService_ValidateRequest_UnbracketedIPv6PortNeedsSignature verifies that
// an allowlisted IPv6 address written without brackets and followed by a
// port, or by a colon with no port, needs a signature. Go's URL parser, which
// the fetcher uses, reads 2001:db8::1:8443 as the host 2001:db8::1 with the
// port 8443.
func TestService_ValidateRequest_UnbracketedIPv6PortNeedsSignature(
t *testing.T,
) {
t.Parallel()
svc, _ := SetupTestService(t,
WithSigningKey(testSigningKey),
WithAllowlist("2001:db8::1"),
)
for _, host := range []string{"2001:db8::1:8443", "2001:db8::1:"} {
t.Run(host, func(t *testing.T) {
t.Parallel()
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: host,
Path: testPathCat,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
t.Logf("ValidateRequest() for host %q with no signature: %v",
host, err)
if err == nil {
t.Errorf("host %q served without a signature, want refused",
host)
}
})
}
}
func TestService_Get_InvalidFile(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -389,10 +114,10 @@ func TestService_Get_InvalidFile(t *testing.T) {
}
}
func TestService_Get_NotFound(t *testing.T) {
func TestProxy_Get_NotFound(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := photoVariant(fixtures, 85, imagerequest.FitCover)
@@ -405,10 +130,10 @@ func TestService_Get_NotFound(t *testing.T) {
}
}
func TestService_Get_FormatConversion(t *testing.T) {
func TestProxy_Get_FormatConversion(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
tests := []struct {
@@ -488,10 +213,10 @@ func TestService_Get_FormatConversion(t *testing.T) {
}
}
func TestService_Get_Caching(t *testing.T) {
func TestProxy_Get_Caching(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
@@ -547,7 +272,7 @@ func TestService_Get_Caching(t *testing.T) {
}
}
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
// TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo caches a source and
// a variant of it and sets back two days every time pixa stored for them.
// For the source: last_accessed_at in source_content, and the access and
// modification times of its file. For the variant: last_accessed_at in
@@ -558,10 +283,10 @@ func TestService_Get_Caching(t *testing.T) {
// Left at the present: the files' status change and birth times, which a
// program cannot set, and the times of the directories holding the files,
// which other entries share.
func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
func TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
variant := photoVariant(fixtures, 85, imagerequest.FitCover)
resp, err := svc.Get(t.Context(), variant)
@@ -572,8 +297,8 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
_ = resp.Content.Close()
twoDaysAgo := time.Now().AddDate(0, 0, -2)
backdateSource(t, svc.cache, variant, twoDaysAgo)
backdateVariant(t, svc.cache, variant, twoDaysAgo)
backdateSource(t, svc, variant, twoDaysAgo)
backdateVariant(t, svc, variant, twoDaysAgo)
resp, err = svc.Get(t.Context(), variant)
if err != nil {
@@ -595,23 +320,23 @@ func TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo(t *testing.T) {
_ = resp.Content.Close()
fetches := readCounters(svc.cache).upstreamFetchCount
fetches := readCounters(t, svc).upstreamFetchCount
if fetches != 1 {
t.Errorf("upstream fetches = %d, want 1", fetches)
}
}
// backdateSource sets every time pixa stored for req's cached source to when,
// as listed on TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
// as listed on TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateSource(
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) {
t.Helper()
key := req.Source.Key()
setSourceLastAccessed(t, cache, key, when)
setSourceLastAccessed(t, svc.db, key, when)
path := cache.srcContent.keyToPath(key)
path := sourceFilePath(svc, key)
err := os.Chtimes(path, when, when)
if err != nil {
@@ -621,16 +346,16 @@ func backdateSource(
// backdateVariant sets every time pixa stored for the cached variant req asks
// for to when, as listed on
// TestService_Get_ServesSourceAndVariantCachedTwoDaysAgo.
// TestProxy_Get_ServesSourceAndVariantCachedTwoDaysAgo.
func backdateVariant(
t *testing.T, cache *Cache, req *imagerequest.Request, when time.Time,
t *testing.T, svc *testProxy, req *imagerequest.Request, when time.Time,
) {
t.Helper()
key := imagerequest.CacheKey(req)
path := cache.variants.keyToPath(key)
path := variantFilePath(svc, key)
setVariantLastAccessed(t, cache, key, when)
setVariantLastAccessed(t, svc.db, key, when)
err := os.Chtimes(path, when, when)
if err != nil {
@@ -638,10 +363,10 @@ func backdateVariant(
}
}
func TestService_Get_DifferentSizes(t *testing.T) {
func TestProxy_Get_DifferentSizes(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
// Request same image at different sizes
@@ -692,37 +417,10 @@ func TestService_Get_DifferentSizes(t *testing.T) {
}
}
func TestService_ValidateRequest_NoSigningKey(t *testing.T) {
func TestProxy_Get_ContextCancellation(t *testing.T) {
t.Parallel()
// Service with no signing key - all non-allowlisted requests should fail
svc, fixtures := SetupTestService(t, WithNoAllowlist())
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: fixtures.OtherHost,
Path: testPathUpload,
},
Transform: imagerequest.Transform{
Size: imagerequest.Size{Width: 50, Height: 50},
Format: imagerequest.FormatJPEG,
Quality: 85,
FitMode: imagerequest.FitCover,
},
}
err := svc.ValidateRequest(req)
if err == nil {
t.Error(
"ValidateRequest() expected error when no signing key and host not allowlisted",
)
}
}
func TestService_Get_ContextCancellation(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx, cancel := context.WithCancel(context.Background())
cancel() // Cancel immediately
@@ -746,10 +444,10 @@ func TestService_Get_ContextCancellation(t *testing.T) {
}
}
func TestService_Get_ReturnsETag(t *testing.T) {
func TestProxy_Get_ReturnsETag(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
@@ -783,10 +481,10 @@ func TestService_Get_ReturnsETag(t *testing.T) {
}
}
func TestService_Get_ETagConsistency(t *testing.T) {
func TestProxy_Get_ETagConsistency(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
req := &imagerequest.Request{
@@ -828,10 +526,10 @@ func TestService_Get_ETagConsistency(t *testing.T) {
}
}
func TestService_Get_DifferentETagsForDifferentContent(t *testing.T) {
func TestProxy_Get_DifferentETagsForDifferentContent(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx := context.Background()
// Request same image at different sizes - should get different ETags
@@ -1,4 +1,4 @@
package imgcache
package imageproxy
import (
"image/color"
@@ -55,16 +55,16 @@ func holdProcessingSlot(
return release
}
// TestService_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// TestProxy_Get_WaitsForSlotBeforeReadingCachedSource checks that a
// request whose source is cached holds none of it while it waits for a
// processing slot: it reads the cached file only once it has a slot. With
// the only slot held, a request for a new width of the cached 100x100 photo
// waits; the cached file is then rewritten as a 100x50 image before the slot
// is freed, so the request must answer with that image scaled to 40x20.
func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
func TestProxy_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
svc.processor = imageprocessor.New(
imageprocessor.Params{MaxInputBytes: 50 << 20, MaxConcurrentProcessing: 1},
)
@@ -77,13 +77,15 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
_ = resp.Content.Close()
sourceKey, err := svc.cache.LookupSource(t.Context(),
widthOnlyRequest(fixtures, 50))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source",
sourceKey, err)
cachedSource := widthOnlyRequest(fixtures, 50).Source
reader, _, err := svc.cache.Source(t.Context(), cachedSource)
if err != nil {
t.Fatalf("Source() error = %v; want the cached source", err)
}
_ = reader.Close()
release := holdProcessingSlot(t, svc.processor)
var (
@@ -103,7 +105,7 @@ func TestService_Get_WaitsForSlotBeforeReadingCachedSource(t *testing.T) {
// before waiting, it would have read it by now.
time.Sleep(100 * time.Millisecond)
err = os.WriteFile(svc.cache.srcContent.keyToPath(sourceKey),
err = os.WriteFile(sourceFilePath(svc, cachedSource.Key()),
generateTestJPEG(t, 100, 50, color.RGBA{0, 0, 255, 255}), 0o600)
if err != nil {
t.Fatalf("failed to rewrite the cached source: %v", err)
@@ -0,0 +1,96 @@
package imageproxy
import (
"errors"
"os"
"testing"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestProxy_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestProxy_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.PutFailure(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover).Source)
if err != nil {
t.Fatalf("PutFailure() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
source := photoVariant(fixtures, 85, imagerequest.FitCover).Source
reader, _, err := svc.cache.Source(t.Context(), source)
if err != nil {
t.Fatalf("Source() error = %v, want the cached source", err)
}
_ = reader.Close()
// Evicts the source as the cache's eviction does: its row, then its file
_, err = svc.db.ExecContext(t.Context(),
`DELETE FROM source_content WHERE source_key = ?`, string(source.Key()))
if err != nil {
t.Fatalf("deleting the source's row: %v", err)
}
err = os.Remove(sourceFilePath(svc, source.Key()))
if err != nil {
t.Fatalf("removing the source's file: %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
func TestProxy_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Proxy.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type
err := ErrNegativeCached
if !errors.Is(err, ErrNegativeCached) {
t.Error("ErrNegativeCached should be identifiable with errors.Is")
}
}
@@ -0,0 +1,273 @@
package imageproxy
import (
"bytes"
"errors"
"io"
"io/fs"
"log/slog"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// truncateFile cuts the file at path down to size bytes.
func truncateFile(t *testing.T, path string, size int64) {
t.Helper()
t.Logf("cutting %s from %d to %d bytes",
filepath.Base(path), fileSize(t, path), size)
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("truncating %s: %v", path, err)
}
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *testProxy, req *imagerequest.Request,
) string {
t.Helper()
reader, _, err := svc.cache.Source(t.Context(), req.Source)
if err != nil {
t.Fatalf("Source() error = %v; want the cached source", err)
}
_ = reader.Close()
return sourceFilePath(svc, req.Source.Key())
}
// reconcileAsAtStartup runs the reconciliation pass of a second cache over
// the database and the state directory of svc's cache, as pixad does when it
// starts. It waits for the pass to remove the file at path, stops the second
// cache, and returns what that cache logged.
//
// The database is limited to one connection, the one that holds the
// in-memory schema. A statement cancelled while it runs can close that
// connection, and the database with it. The test therefore takes the
// connection before it stops the second cache, so that the stop only cancels
// a wait for the connection.
func reconcileAsAtStartup(t *testing.T, svc *testProxy, path string) string {
t.Helper()
var logBuf bytes.Buffer
cache, err := imgcache.NewCache(svc.db, imgcache.Config{
StateDir: svc.stateDir,
MaxBytes: 1 << 30,
Logger: slog.New(slog.NewJSONHandler(&logBuf, nil)),
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create the second cache: %v", err)
}
svc.db.SetMaxOpenConns(1)
cache.StartEviction(imgcache.DefaultEvictionInterval)
deadline := time.Now().Add(5 * time.Second)
_, statErr := os.Stat(path)
for statErr == nil && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
_, statErr = os.Stat(path)
}
releaseDatabase := holdDatabase(t, svc.db)
err = cache.StopEviction(t.Context())
releaseDatabase()
if err != nil {
t.Fatalf("StopEviction() error = %v", err)
}
t.Logf("reconciliation log: %s", logBuf.String())
if !errors.Is(statErr, fs.ErrNotExist) {
t.Fatalf("the reconciliation pass did not remove %s: %v", path, statErr)
}
return logBuf.String()
}
// assertWarningNames checks that logs hold a warning naming the file at
// path.
func assertWarningNames(t *testing.T, logs, path string) {
t.Helper()
for line := range strings.Lines(logs) {
if strings.Contains(line, `"level":"WARN"`) &&
strings.Contains(line, `"path":"`+path+`"`) {
return
}
}
t.Errorf("reconciliation logged no warning naming %s", path)
}
// TestProxy_Get_FetchesAgainForEmptyCachedSource empties the cached source
// file, as a crash or a restored snapshot could, and runs the reconciliation
// pass, which must remove it with a warning naming it. It then asks for two
// new sizes of the source: the first must fetch it again and cache it whole,
// so that the second is made from the cache without a third fetch.
func TestProxy_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, 0)
assertWarningNames(t, reconcileAsAtStartup(t, svc, path), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
"was kept and fetched again on every new size", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestProxy_Get_FetchesAgainForPartialCachedSource cuts the cached source
// file to half its size, as a crash or a restored snapshot could, and runs
// the reconciliation pass, which must remove it with a warning naming it.
// It then asks for two new sizes of the source: the first must fetch it
// again rather than fail to decode the partial file, and the second is
// made from the cache.
func TestProxy_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := setupTestProxy(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc, path), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(t, svc).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// assertConvertedAgain caches a variant of the photo, cuts its file to size
// bytes and runs the reconciliation pass, which must remove it with a
// warning naming it. It then checks that Get converts the image again
// instead of serving the file, and that the next Get serves the whole
// variant from the cache.
func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := setupTestProxy(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := variantFilePath(svc, imagerequest.CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc, path), path)
got, status := getImage(t, svc, req)
t.Logf("after the cut: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestProxy_Get_ConvertsAgainForEmptyCachedVariant empties a cached
// variant file, as a crash or a restored snapshot could, and checks that it
// is converted again rather than served empty.
func TestProxy_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 0)
}
// TestProxy_Get_ConvertsAgainForPartialCachedVariant cuts a cached
// variant file to ten bytes, as a crash or a restored snapshot could, and
// checks that it is converted again rather than served cut short.
func TestProxy_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 10)
}
@@ -0,0 +1,78 @@
package imageproxy
import (
"context"
"database/sql"
"errors"
"testing"
"time"
"sneak.berlin/go/pixa/internal/imagerequest"
)
// pendingAccessTimesWait is how long a test waits for the database's
// connection, for a cache hit to be answered, for an access time to reach
// the database, or for a request to return past its deadline.
const pendingAccessTimesWait = 5 * time.Second
// holdDatabase takes the one connection of db, so that every other query
// waits for it, and returns the func that frees it.
func holdDatabase(t *testing.T, db *sql.DB) func() {
t.Helper()
ctx, cancel := context.WithTimeout(t.Context(), pendingAccessTimesWait)
defer cancel()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatalf("failed to take the database connection: %v", err)
}
return func() { _ = conn.Close() }
}
// TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestProxy_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchProxy(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.db)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
@@ -1,4 +1,4 @@
package imgcache
package imageproxy
import (
"bytes"
@@ -12,15 +12,15 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestService_Get_CountsStats walks Get through a miss that fetches the
// TestProxy_Get_CountsStats walks Get through a miss that fetches the
// source, a hit, a miss that reuses the cached source, and two misses whose
// source cannot be used, checking every counter of the cache after each.
func TestService_Get_CountsStats(t *testing.T) {
func TestProxy_Get_CountsStats(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
// NewTestFS builds the same files the test service's fetcher serves.
// NewTestFS builds the same files the test proxy's fetcher serves.
testFS, _ := NewTestFS(t)
photo, err := fs.ReadFile(testFS, fixtures.GoodHostJPEG)
@@ -76,7 +76,7 @@ func TestService_Get_CountsStats(t *testing.T) {
_ = resp.Content.Close()
}
got := readCounters(svc.cache)
got := readCounters(t, svc)
if got != step.want {
t.Fatalf("after the %s: counters = %+v, want %+v", step.name, got, step.want)
}
@@ -93,7 +93,7 @@ type fakeUpstream struct {
}
func (u *fakeUpstream) Fetch(
context.Context, string,
context.Context, imagerequest.Source,
) (*httpfetcher.FetchResult, error) {
return &httpfetcher.FetchResult{
Content: io.NopCloser(u),
@@ -114,10 +114,10 @@ func (u *fakeUpstream) Read(p []byte) (int, error) {
return 0, u.err
}
// TestService_Get_CountsInterruptedMisses checks every counter of the cache
// TestProxy_Get_CountsInterruptedMisses checks every counter of the cache
// after a miss whose request context ends during or after the upstream
// fetch, and after a miss whose upstream body is over the size limit.
func TestService_Get_CountsInterruptedMisses(t *testing.T) {
func TestProxy_Get_CountsInterruptedMisses(t *testing.T) {
t.Parallel()
photo := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
@@ -145,7 +145,7 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
@@ -182,7 +182,7 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
_ = resp.Content.Close()
}
got := readCounters(svc.cache)
got := readCounters(t, svc)
if got != tc.want {
t.Errorf("counters = %+v, want %+v", got, tc.want)
}
@@ -190,13 +190,13 @@ func TestService_Get_CountsInterruptedMisses(t *testing.T) {
}
}
// TestService_Get_CountsHitAfterRequestEnds checks every counter of the
// TestProxy_Get_CountsHitAfterRequestEnds checks every counter of the
// cache after a hit served with a request context that has already ended:
// only the hit count moves.
func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
func TestProxy_Get_CountsHitAfterRequestEnds(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
svc, fixtures := setupTestProxy(t)
req := &imagerequest.Request{
Source: imagerequest.Source{
@@ -219,7 +219,7 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
_ = resp.Content.Close()
want := readCounters(svc.cache)
want := readCounters(t, svc)
want.hitCount++
ctx, cancel := context.WithCancel(t.Context())
@@ -236,7 +236,7 @@ func TestService_Get_CountsHitAfterRequestEnds(t *testing.T) {
t.Fatalf("CacheStatus = %v, want %v", resp.CacheStatus, CacheHit)
}
got := readCounters(svc.cache)
got := readCounters(t, svc)
if got != want {
t.Errorf("counters = %+v, want %+v", got, want)
}
@@ -0,0 +1,354 @@
package imageproxy
import (
"bytes"
"context"
"database/sql"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"path/filepath"
"testing"
"testing/fstest"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
)
// Shared test data literals, extracted as constants for goconst.
const (
testContentTypeJPEG = "image/jpeg"
)
// sqliteTimestampFormat matches the format SQLite's CURRENT_TIMESTAMP
// produces, so injected timestamps compare correctly against ones the
// cache writes.
const sqliteTimestampFormat = "2006-01-02 15:04:05"
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
GoodHostJPEG string // valid JPEG on GoodHost
GoodHostPNG string // valid PNG on GoodHost
GoodHostGIF string // valid GIF on GoodHost
OtherHostJPEG string // valid JPEG on OtherHost
OtherHostPNG string // valid PNG on OtherHost
// Invalid/edge case files
InvalidFile string // file with wrong magic bytes
EmptyFile string // zero-byte file
TextFile string // text file masquerading as image
// Hostnames
GoodHost string // a host the mock fetcher serves
OtherHost string // a second host the mock fetcher serves
}
// DefaultFixtures returns the standard test fixture paths.
func DefaultFixtures() *TestFixtures {
return &TestFixtures{
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
GoodHostPNG: "goodhost.example.com/images/logo.png",
GoodHostGIF: "goodhost.example.com/images/animation.gif",
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
InvalidFile: "goodhost.example.com/images/fake.jpg",
EmptyFile: "goodhost.example.com/images/empty.jpg",
TextFile: "goodhost.example.com/images/text.png",
GoodHost: "goodhost.example.com",
OtherHost: "otherhost.example.com",
}
}
// NewTestFS creates a mock filesystem with test images.
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
t.Helper()
fixtures := DefaultFixtures()
// Generate test images
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
// Create the mock filesystem
mockFS := fstest.MapFS{
// Good host files
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
// Other host files
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
// Invalid files
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
}
return mockFS, fixtures
}
// generateTestJPEG creates a minimal valid JPEG image.
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: 85})
if err != nil {
t.Fatalf("failed to encode test JPEG: %v", err)
}
return buf.Bytes()
}
// generateTestPNG creates a minimal valid PNG image.
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// generateTestGIF creates a minimal valid GIF image.
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
return buf.Bytes()
}
// testProxy is a Proxy with the database, the state directory and the
// metrics registry of its cache, through which a test reads and changes the
// cache behind the proxy.
type testProxy struct {
*Proxy
db *sql.DB
stateDir string
registry *prometheus.Registry
}
// setupTestProxy creates a Proxy with mock fetcher for testing.
func setupTestProxy(t *testing.T) (*testProxy, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
// Create temp directory for cache
tmpDir := t.TempDir()
// Create in-memory database
db := setupTestDB(t)
registry := prometheus.NewRegistry()
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, registry)
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
proxy, err := NewProxy(&Config{
Cache: cache,
Fetcher: httpfetcher.NewMock(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("failed to create proxy: %v", err)
}
return &testProxy{
Proxy: proxy,
db: db,
stateDir: tmpDir,
registry: registry,
}, fixtures
}
// counterValues holds the values of a cache's counters.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads the counters of svc's cache from its metrics registry.
func readCounters(t *testing.T, svc *testProxy) counterValues {
t.Helper()
families, err := svc.registry.Gather()
if err != nil {
t.Fatalf("failed to gather the cache's counters: %v", err)
}
values := make(map[string]int64, len(families))
for _, family := range families {
values[family.GetName()] = int64(family.GetMetric()[0].GetCounter().GetValue())
}
return counterValues{
hitCount: values["pixa_cache_hits_total"],
missCount: values["pixa_cache_misses_total"],
upstreamFetchCount: values["pixa_upstream_fetches_total"],
upstreamFetchBytes: values["pixa_upstream_fetch_bytes_total"],
transformCount: values["pixa_transforms_total"],
}
}
// sourceFilePath returns the path of the file of the source key in svc's
// cache, as README.md gives it: <state_dir>/cache/sources/<ab>/<cd>/<key>.
func sourceFilePath(svc *testProxy, key imagerequest.SourceKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "sources", name[0:2], name[2:4], name)
}
// variantFilePath returns the path of the file of the variant key in svc's
// cache, as README.md gives it: <state_dir>/cache/variants/<ab>/<cd>/<key>.
func variantFilePath(svc *testProxy, key imagerequest.VariantKey) string {
name := string(key)
return filepath.Join(svc.stateDir, "cache", "variants", name[0:2], name[2:4], name)
}
// setVariantLastAccessed backdates the last access time of a tracked
// variant.
func setVariantLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.VariantKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE variant_content SET last_accessed_at = ? WHERE cache_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set variant last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("variant %s has no accounting row (affected=%d); "+
"stores must track variants in the database", key, affected)
}
}
// setSourceLastAccessed backdates the last access time of a tracked
// source content blob.
func setSourceLastAccessed(
t *testing.T, db *sql.DB, key imagerequest.SourceKey, when time.Time,
) {
t.Helper()
res, err := db.ExecContext(t.Context(),
`UPDATE source_content SET last_accessed_at = ? WHERE source_key = ?`,
when.UTC().Format(sqliteTimestampFormat), string(key),
)
if err != nil {
t.Fatalf("failed to set source last_accessed_at: %v", err)
}
affected, err := res.RowsAffected()
if err != nil {
t.Fatalf("failed to read affected rows: %v", err)
}
if affected != 1 {
t.Fatalf("source %s has no accounting row (affected=%d)", key, affected)
}
}
// countRows returns the number of rows the given query yields.
func countRows(t *testing.T, db *sql.DB, query string, args ...any) int {
t.Helper()
var n int
err := db.QueryRowContext(t.Context(), query, args...).Scan(&n)
if err != nil {
t.Fatalf("count query %q failed: %v", query, err)
}
return n
}
// setupTestDB creates an in-memory SQLite database for testing
// using the production schema.
func setupTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
return db
}
+15 -25
View File
@@ -1,13 +1,12 @@
// Package imagerequest holds the image request type, its parts, its checks,
// the parsing of the /v1/image/ path, the variant key and the source key. It
// imports no other package of this repo, so every package that handles an
// image request can import it.
// the parsing of the /v1/image/ path, the variant key, the source key and the
// source's upstream URL. It imports no other package of this repo, so every
// package that handles an image request can import it.
package imagerequest
import (
"errors"
"net/url"
"time"
)
// ErrInvalidFitMode is returned when an invalid fit mode is provided.
@@ -85,19 +84,13 @@ func ValidateImageRequest(req *Request) error {
}
// Request represents a request for a processed image: the source to fetch
// and the transform to apply to it.
// and the transform to apply to it. It holds exactly what the variant key
// covers.
type Request struct {
// Source is what to fetch
Source Source
// Transform is what to do to it
Transform Transform
// Signature is the HMAC signature for non-allowlisted hosts
Signature string
// Expires is when the URL expires: the exp of a signed URL, or the expiry
// of an encrypted URL; the zero time if it has none
Expires time.Time
// AllowHTTP indicates whether HTTP (non-TLS) is allowed for this request
AllowHTTP bool
}
// Source is the image a request fetches from the origin.
@@ -125,23 +118,20 @@ type Transform struct {
FitMode Fit
}
// SourceURL returns the full upstream URL to fetch: the source path
// percent-encoded again, as net/url writes a path, so each source path is one
// upstream path, and the source query as it is, with the characters
// escapeQuery encodes percent-encoded. Uses http:// scheme when AllowHTTP is
// true, otherwise https://.
func (r *Request) SourceURL() string {
// URL returns the source's upstream URL, with the scheme http when allowHTTP
// is true and https otherwise. net/url writes its path percent-encoded again,
// so each source path is one upstream path. Its query is the source query as
// it is, with the characters escapeQuery encodes percent-encoded.
func (s Source) URL(allowHTTP bool) *url.URL {
scheme := "https"
if r.AllowHTTP {
if allowHTTP {
scheme = "http"
}
sourceURL := url.URL{
return &url.URL{
Scheme: scheme,
Host: r.Source.Host,
Path: r.Source.Path,
RawQuery: escapeQuery(r.Source.Query),
Host: s.Host,
Path: s.Path,
RawQuery: escapeQuery(s.Query),
}
return sourceURL.String()
}
@@ -2,57 +2,49 @@ package imagerequest
import "testing"
func TestRequest_SourceURL_DefaultHTTPS(t *testing.T) {
func TestSource_URL_DefaultHTTPS(t *testing.T) {
t.Parallel()
req := &Request{
Source: Source{
Host: testHostCDN,
Path: testPathCat,
Query: "v=2",
},
source := Source{
Host: testHostCDN,
Path: testPathCat,
Query: "v=2",
}
got := req.SourceURL()
got := source.URL(false).String()
want := "https://cdn.example.com/photos/cat.jpg?v=2"
if got != want {
t.Errorf("SourceURL() = %q, want %q", got, want)
t.Errorf("URL() = %q, want %q", got, want)
}
}
func TestRequest_SourceURL_AllowHTTP(t *testing.T) {
func TestSource_URL_AllowHTTP(t *testing.T) {
t.Parallel()
req := &Request{
Source: Source{
Host: "localhost:8080",
Path: testPathCat,
},
AllowHTTP: true,
source := Source{
Host: "localhost:8080",
Path: testPathCat,
}
got := req.SourceURL()
got := source.URL(true).String()
want := "http://localhost:8080/photos/cat.jpg"
if got != want {
t.Errorf("SourceURL() = %q, want %q", got, want)
t.Errorf("URL() = %q, want %q", got, want)
}
}
func TestRequest_SourceURL_AllowHTTPFalse(t *testing.T) {
func TestSource_URL_AllowHTTPFalse(t *testing.T) {
t.Parallel()
req := &Request{
Source: Source{
Host: testHostCDN,
Path: "/img.jpg",
},
AllowHTTP: false,
source := Source{
Host: testHostCDN,
Path: "/img.jpg",
}
got := req.SourceURL()
got := source.URL(false).String()
if got != "https://cdn.example.com/img.jpg" {
t.Errorf("SourceURL() = %q, want https scheme", got)
t.Errorf("URL() = %q, want https scheme", got)
}
}
+6 -7
View File
@@ -126,15 +126,15 @@ func parseImageComponents(remainder string) (*Request, error) {
return nil, ErrMissingHost
}
// Only the forms hostRegex describes are accepted. A "?", "#" or "@" in
// the host would make Go's URL parser, which the allowlist check and the
// fetcher use, read a different host from the one allowlisted and cached
// Later steps put the host unparsed into the allowlist check, the cache key
// and the upstream URL, so only the forms hostRegex describes are accepted
hostMatch := hostRegex.FindStringSubmatch(host)
if hostMatch == nil {
return nil, fmt.Errorf("%w: %q", ErrInvalidHost, host)
}
// Go's URL parser refuses brackets around anything but an IPv6 address
// The allowlist check and the fetcher drop the brackets, so brackets
// around anything but an IPv6 address would give a host a second spelling
if inBrackets := hostMatch[1]; inBrackets != "" {
addr, err := netip.ParseAddr(inBrackets)
if err != nil || !addr.Is6() {
@@ -147,9 +147,8 @@ func parseImageComponents(remainder string) (*Request, error) {
// host
host = strings.ToLower(host)
// Go's URL parser, which the fetcher uses, reads a "#" as the start of a
// fragment and drops the rest, so each text after it would be another
// spelling, cached under its own keys, of one upstream request
// A "#" starts a URL's fragment, so neither the source path nor the source
// query may hold one
if strings.Contains(path, "#") {
return nil, fmt.Errorf(
`%w: the source path or query holds a "#"`, ErrInvalidPath)
+124 -117
View File
@@ -1,3 +1,10 @@
// Package imgcache is the image cache. It keeps:
// - the source images fetched from upstream
// - the variants made from them
// - the negative cache: the source URLs whose fetch failed recently
//
// Each image is a file with a row in the database. Eviction removes the
// least recently used images to keep the cache under its size limit.
package imgcache
import (
@@ -26,8 +33,12 @@ const maxPendingAccessTimes = 10000
// have expired.
const cleanExpiredBatchSize = 1000
// CacheConfig holds cache configuration.
type CacheConfig struct {
// DefaultNegativeTTL is how long pixad keeps a failed fetch in the negative
// cache.
const DefaultNegativeTTL = 5 * time.Minute
// Config holds cache configuration.
type Config struct {
StateDir string
NegativeTTL time.Duration
@@ -48,7 +59,7 @@ type Cache struct {
db *sql.DB
srcContent *ContentStorage // source images by source key
variants *VariantStorage // processed variants by cache key
config CacheConfig
config Config
log *slog.Logger
// Eviction machinery. The channels are created in NewCache so
@@ -97,7 +108,7 @@ type Cache struct {
// metricsRegistry. pixad passes Prometheus' default registry, the one
// /metrics serves. It makes no database call.
func NewCache(
db *sql.DB, config CacheConfig, metricsRegistry prometheus.Registerer,
db *sql.DB, config Config, metricsRegistry prometheus.Registerer,
) (*Cache, error) {
log := config.Logger
if log == nil {
@@ -147,75 +158,35 @@ func NewCache(
return c, nil
}
// GetVariant opens a cached variant and returns a reader, its size and its
// Variant opens a cached variant and returns a reader, its size and its
// content type, read from the variant file's first bytes. It opens that one
// file and makes no database call. A variant not in the cache, or any
// variant of a disabled cache, is ErrNotFound; a variant file whose first
// bytes are not those of an output format is another error.
func (c *Cache) GetVariant(
// file and makes no database call. A variant it opens has its access time
// handed to the background writer without waiting (see touchVariant). A
// variant not in the cache, or any variant of a disabled cache, is
// ErrNotFound; a variant file whose first bytes are not those of an output
// format is another error.
func (c *Cache) Variant(
cacheKey imagerequest.VariantKey,
) (io.ReadCloser, int64, string, error) {
if c.config.MaxBytes == 0 {
return nil, 0, "", ErrNotFound
}
return c.variants.LoadWithContentType(cacheKey)
reader, size, contentType, err := c.variants.LoadWithContentType(cacheKey)
if err != nil {
return nil, 0, "", err
}
c.touchVariant(cacheKey)
return reader, size, contentType, nil
}
// StoreSource stores fetched source content: it writes the file named by
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) StoreSource(
ctx context.Context,
req *imagerequest.Request,
content io.Reader,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
}
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
key := req.Source.Key()
size, err := c.srcContent.Store(key, data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (source_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(source_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), size)
if err != nil {
deleteErr := c.srcContent.Delete(key)
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
}
return "", fmt.Errorf("failed to insert source content: %w", err)
}
c.notifyWritePressure()
return key, nil
}
// StoreVariant stores a processed variant by its cache key and records
// PutVariant stores a processed variant by its cache key and records
// it in the size accounting. On a disabled cache it is a no-op. The
// accounting insert is best-effort (the startup reconciliation pass
// adopts any variant file that misses its accounting row).
func (c *Cache) StoreVariant(
func (c *Cache) PutVariant(
ctx context.Context, cacheKey imagerequest.VariantKey, content io.Reader,
) error {
if c.config.MaxBytes == 0 {
@@ -244,19 +215,18 @@ func (c *Cache) StoreVariant(
return nil
}
// LookupSource checks if we have cached source content for a request.
// Returns the source key if found, or an empty key if not; a row whose
// file is missing is not found. A source found has its access time
// handed to the background writer without waiting (see
// touchSourceContent); a disabled cache always reports no cached source.
func (c *Cache) LookupSource(
ctx context.Context, req *imagerequest.Request,
) (imagerequest.SourceKey, error) {
// Source opens the cached source of src and returns a reader and its size
// in bytes. A source whose row or file is missing is ErrNotFound, as is any
// source of a disabled cache. A source it opens has its access time handed
// to the background writer without waiting (see touchSourceContent).
func (c *Cache) Source(
ctx context.Context, src imagerequest.Source,
) (io.ReadCloser, int64, error) {
if c.config.MaxBytes == 0 {
return "", nil
return nil, 0, ErrNotFound
}
key := req.Source.Key()
key := src.Key()
var rowExists bool
@@ -264,25 +234,100 @@ func (c *Cache) LookupSource(
SELECT EXISTS(SELECT 1 FROM source_content WHERE source_key = ?)
`, string(key)).Scan(&rowExists)
if err != nil {
return "", fmt.Errorf("failed to lookup source: %w", err)
return nil, 0, fmt.Errorf("failed to lookup source: %w", err)
}
if !rowExists {
return "", nil
return nil, 0, ErrNotFound
}
// Verify the content file exists
if !c.srcContent.Exists(key) {
return "", nil
reader, size, err := c.srcContent.LoadWithSize(key)
if err != nil {
return nil, 0, err
}
c.touchSourceContent(key)
return reader, size, nil
}
// PutSource stores fetched source content: it writes the file named by
// the source's key, then the source's row. When the row cannot be written
// the file is removed and the store fails, as a file with no row is never
// served or evicted; if that removal fails too, the next reconciliation
// pass removes the file. On a disabled cache it is a no-op returning an
// empty key.
func (c *Cache) PutSource(
ctx context.Context,
src imagerequest.Source,
content io.Reader,
) (imagerequest.SourceKey, error) {
if c.config.MaxBytes == 0 {
return "", nil
}
data, err := io.ReadAll(content)
if err != nil {
return "", fmt.Errorf("failed to read source content: %w", err)
}
key := src.Key()
size, err := c.srcContent.Store(key, data)
if err != nil {
return "", fmt.Errorf("failed to store source content: %w", err)
}
_, err = c.db.ExecContext(ctx, `
INSERT INTO source_content (source_key, size_bytes)
VALUES (?, ?)
ON CONFLICT(source_key) DO UPDATE SET
size_bytes = excluded.size_bytes,
last_accessed_at = CURRENT_TIMESTAMP
`, string(key), size)
if err != nil {
deleteErr := c.srcContent.Delete(key)
if deleteErr != nil {
c.log.Warn("failed to remove source file whose row was not written",
"source_key", key, "error", deleteErr)
}
return "", fmt.Errorf("failed to insert source content: %w", err)
}
c.notifyWritePressure()
return key, nil
}
// StoreNegative stores a negative cache entry for a failed fetch.
func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) error {
// Failure reports whether src is in the negative cache. It only reads: a
// row past its expiry is not a hit, and the eviction goroutine deletes it
// with the other expired rows (see CleanExpired).
func (c *Cache) Failure(
ctx context.Context, src imagerequest.Source,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, src.Host, src.Path, src.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
expired := time.Now().After(expiresAt)
return !expired, nil
}
// PutFailure stores a negative cache entry for src, whose fetch failed.
func (c *Cache) PutFailure(ctx context.Context, src imagerequest.Source) error {
expiresAt := time.Now().UTC().Add(c.config.NegativeTTL)
_, err := c.db.ExecContext(ctx, `
@@ -291,7 +336,7 @@ func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) er
VALUES (?, ?, ?, ?)
ON CONFLICT(source_host, source_path, source_query) DO UPDATE SET
expires_at = excluded.expires_at
`, req.Source.Host, req.Source.Path, req.Source.Query, expiresAt)
`, src.Host, src.Path, src.Query, expiresAt)
if err != nil {
return fmt.Errorf("failed to insert negative cache: %w", err)
}
@@ -299,18 +344,6 @@ func (c *Cache) StoreNegative(ctx context.Context, req *imagerequest.Request) er
return nil
}
// GetSourceContent returns a reader for cached source content by its key,
// and the content's size in bytes.
func (c *Cache) GetSourceContent(
key imagerequest.SourceKey,
) (io.ReadCloser, int64, error) {
if c.config.MaxBytes == 0 {
return nil, 0, ErrNotFound
}
return c.srcContent.LoadWithSize(key)
}
// CleanExpired removes the negative cache rows past their expiry, one
// batch per statement, until none is left. Once ctx ends it starts no
// further batch and returns ctx's error. The eviction goroutine calls it
@@ -388,32 +421,6 @@ func (c *Cache) touchSourceContent(key imagerequest.SourceKey) {
}
}
// checkNegativeCache checks if a request is in the negative cache. It only
// reads: a row past its expiry is not a hit, and the eviction goroutine
// deletes it with the other expired rows (see CleanExpired).
func (c *Cache) checkNegativeCache(
ctx context.Context, req *imagerequest.Request,
) (bool, error) {
var expiresAt time.Time
err := c.db.QueryRowContext(ctx, `
SELECT expires_at FROM negative_cache
WHERE source_host = ? AND source_path = ? AND source_query = ?
`, req.Source.Host, req.Source.Path, req.Source.Query).Scan(&expiresAt)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("failed to check negative cache: %w", err)
}
expired := time.Now().After(expiresAt)
return !expired, nil
}
// cleanExpiredBatch deletes up to cleanExpiredBatchSize negative cache
// rows past their expiry in one statement, and returns how many it
// deleted. DELETE ... LIMIT needs a SQLite build option, so the subquery
+16 -105
View File
@@ -6,7 +6,6 @@ import (
"fmt"
"image/color"
"io"
"os"
"testing"
"time"
@@ -15,10 +14,10 @@ import (
"sneak.berlin/go/pixa/internal/magic"
)
// TestGetVariant_ContentTypeFromFirstBytes stores an image in each output
// format as a variant and checks that GetVariant serves it whole with the
// TestVariant_ContentTypeFromFirstBytes stores an image in each output
// format as a variant and checks that Variant serves it whole with the
// content type of its format, read from the file.
func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
func TestVariant_ContentTypeFromFirstBytes(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -50,14 +49,14 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
key := imagerequest.VariantKey("aabbccdd10" + string(format))
err = cache.StoreVariant(t.Context(), key, bytes.NewReader(image))
err = cache.PutVariant(t.Context(), key, bytes.NewReader(image))
if err != nil {
t.Fatalf("StoreVariant(%s) error = %v", format, err)
t.Fatalf("PutVariant(%s) error = %v", format, err)
}
reader, size, contentType, err := cache.GetVariant(key)
reader, size, contentType, err := cache.Variant(key)
if err != nil {
t.Fatalf("GetVariant(%s) error = %v", format, err)
t.Fatalf("Variant(%s) error = %v", format, err)
}
served, err := io.ReadAll(reader)
@@ -80,11 +79,11 @@ func TestGetVariant_ContentTypeFromFirstBytes(t *testing.T) {
}
}
// TestGetVariant_RefusesFileOfNoOutputFormat checks that GetVariant refuses
// TestVariant_RefusesFileOfNoOutputFormat checks that Variant refuses
// a variant file whose first bytes are not those of an output format, with
// an error other than ErrNotFound, and that a variant with no file, or any
// variant of a disabled cache, is ErrNotFound.
func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
func TestVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -110,118 +109,30 @@ func TestGetVariant_RefusesFileOfNoOutputFormat(t *testing.T) {
t.Fatalf("%s: storing the variant file: %v", tt.name, err)
}
reader, _, contentType, err := cache.GetVariant(key)
t.Logf("%s: GetVariant() error = %v", tt.name, err)
reader, _, contentType, err := cache.Variant(key)
t.Logf("%s: Variant() error = %v", tt.name, err)
if reader != nil || contentType != "" || !errors.Is(err, tt.wantErr) {
t.Errorf("%s: GetVariant() = reader %v, content type %q, error %v; "+
t.Errorf("%s: Variant() = reader %v, content type %q, error %v; "+
"want no reader and %v", tt.name, reader, contentType, err, tt.wantErr)
}
}
reader, _, _, err := cache.GetVariant(testVariantKeyFour)
reader, _, _, err := cache.Variant(testVariantKeyFour)
if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("GetVariant() of a variant with no file = reader %v, error %v; "+
t.Errorf("Variant() of a variant with no file = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound)
}
disabled, _ := newEvictionTestCache(t, 0)
reader, _, _, err = disabled.GetVariant(testVariantKeyOne)
reader, _, _, err = disabled.Variant(testVariantKeyOne)
if reader != nil || !errors.Is(err, ErrNotFound) {
t.Errorf("GetVariant() on a disabled cache = reader %v, error %v; "+
t.Errorf("Variant() on a disabled cache = reader %v, error %v; "+
"want no reader and %v", reader, err, ErrNotFound)
}
}
// TestService_Get_MakesAgainVariantOfNoOutputFormat replaces a cached
// variant's file with text, and checks that the next request is a miss
// answered with the variant made again, and the one after a hit.
func TestService_Get_MakesAgainVariantOfNoOutputFormat(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
err := os.WriteFile(path, []byte("this is text, not an image"), StorageFilePerm)
if err != nil {
t.Fatalf("replacing the variant file: %v", err)
}
got, status := getImage(t, svc, req)
t.Logf("after the replacement: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the replacement: Get() = %d bytes, %s; want the %d bytes "+
"made again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_HitWaitsForNoDatabaseWrite holds the database's one
// connection while a cached variant is requested. The hit must be answered
// while the connection is still held, and its access time must reach the
// database once it is free.
func TestService_Get_HitWaitsForNoDatabaseWrite(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
key := imagerequest.CacheKey(req)
getImage(t, svc, req)
// Each connection to an in-memory SQLite database opens a new, empty one,
// so the goroutine that writes the access times must share the
// connection that holds the schema.
svc.cache.db.SetMaxOpenConns(1)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
anHourAgo := time.Now().Add(-time.Hour)
setVariantLastAccessed(t, svc.cache, key, anHourAgo)
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
results := startGet(t.Context(), svc, req)
select {
case got := <-results:
if got.err != nil {
t.Fatalf("Get() of the cached variant error = %v", got.err)
}
case <-time.After(pendingAccessTimesWait):
t.Fatal("the cache hit waited for the database")
}
releaseDatabase()
deadline := time.Now().Add(pendingAccessTimesWait)
for countRows(t, svc.cache, `
SELECT COUNT(*) FROM variant_content
WHERE cache_key = ? AND last_accessed_at > ?
`, string(key), anHourAgo.UTC().Format(sqliteTimestampFormat)) == 0 {
if time.Now().After(deadline) {
t.Fatal("the hit's access time was not written")
}
time.Sleep(10 * time.Millisecond)
}
}
// TestTouchVariantAndTouchSourceContent_DropAccessTimesPastTheLimit hands one
// more access time of a variant, and one more of a source, than may wait to
// be written. touchVariant and touchSourceContent must return each time
+2 -2
View File
@@ -50,7 +50,7 @@ func TestCache_CleanExpired(t *testing.T) {
tmpDir := t.TempDir()
db := setupTestDB(t)
cache, _ := NewCache(db, CacheConfig{
cache, _ := NewCache(db, Config{
StateDir: tmpDir,
NegativeTTL: 1 * time.Millisecond,
MaxBytes: 1 << 30,
@@ -103,7 +103,7 @@ func TestCache_StorageDirectoriesCreated(t *testing.T) {
tmpDir := t.TempDir()
db := setupTestDB(t)
_, err := NewCache(db, CacheConfig{
_, err := NewCache(db, Config{
StateDir: tmpDir,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
+3 -23
View File
@@ -8,26 +8,6 @@ import (
"github.com/prometheus/client_golang/prometheus/testutil"
)
// counterValues holds the values of a cache's counters.
type counterValues struct {
hitCount int64
missCount int64
upstreamFetchCount int64
upstreamFetchBytes int64
transformCount int64
}
// readCounters reads cache's counters.
func readCounters(cache *Cache) counterValues {
return counterValues{
hitCount: int64(testutil.ToFloat64(cache.cacheHits)),
missCount: int64(testutil.ToFloat64(cache.cacheMisses)),
upstreamFetchCount: int64(testutil.ToFloat64(cache.upstreamFetches)),
upstreamFetchBytes: int64(testutil.ToFloat64(cache.upstreamFetchBytes)),
transformCount: int64(testutil.ToFloat64(cache.transforms)),
}
}
// TestNewCache_RegistersItsCountersOnItsRegistry checks that NewCache
// registers the cache's counters on the registry it is given, under the
// names README.md gives them.
@@ -37,7 +17,7 @@ func TestNewCache_RegistersItsCountersOnItsRegistry(t *testing.T) {
registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil {
t.Fatalf("NewCache() error = %v", err)
}
@@ -70,13 +50,13 @@ func TestNewCache_FailsWhenItsCountersCannotBeRegistered(t *testing.T) {
registry := prometheus.NewRegistry()
_, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
if err != nil {
t.Fatalf("NewCache() of the first cache error = %v", err)
}
_, err = NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30}, registry)
t.Logf("NewCache() of the second cache error = %v", err)
var alreadyRegistered prometheus.AlreadyRegisteredError
+2 -2
View File
@@ -317,7 +317,7 @@ func (c *Cache) evictVariant(
// evictSourceBlob removes one source: its row first, then its file, as
// evictVariant does for a variant. It holds no lock: a store of the same
// source in between can leave the source's row with no file, which
// LookupSource treats as a miss.
// Source treats as a miss.
func (c *Cache) evictSourceBlob(ctx context.Context, key imagerequest.SourceKey) error {
_, err := c.db.ExecContext(ctx,
`DELETE FROM source_content WHERE source_key = ?`, string(key))
@@ -479,7 +479,7 @@ func (c *Cache) runCleanExpired(ctx context.Context) {
// then on the first periodic eviction tick that comes at least ten times
// as long after the last pass ended as that pass took, off the request
// hot path: it adopts variant files that predate the accounting table
// (or whose accounting insert failed, e.g. StoreVariant's best-effort
// (or whose accounting insert failed, e.g. PutVariant's best-effort
// insert under transient DB contention), drops variant accounting rows
// whose files are missing, removes source blob files the database does
// not know, removes source and variant files that are empty or not the
+19 -19
View File
@@ -67,7 +67,7 @@ func newEvictionTestCache(t *testing.T, maxBytes int64) (*Cache, string) {
db := evictionTestDB(t)
// maxBytes zero turns the disk cache off, as cache_max_bytes: 0 does.
cache, err := NewCache(db, CacheConfig{
cache, err := NewCache(db, Config{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: maxBytes,
@@ -98,11 +98,11 @@ func storeEvictionTestSource(
},
}
key, err := cache.StoreSource(
context.Background(), req, bytes.NewReader(content),
key, err := cache.PutSource(
context.Background(), req.Source, bytes.NewReader(content),
)
if err != nil {
t.Fatalf("StoreSource(%s%s) failed: %v", host, path, err)
t.Fatalf("PutSource(%s%s) failed: %v", host, path, err)
}
return key
@@ -115,9 +115,9 @@ func storeEvictionTestVariant(
) {
t.Helper()
err := cache.StoreVariant(t.Context(), key, bytes.NewReader(content))
err := cache.PutVariant(t.Context(), key, bytes.NewReader(content))
if err != nil {
t.Fatalf("StoreVariant(%s) failed: %v", key, err)
t.Fatalf("PutVariant(%s) failed: %v", key, err)
}
}
@@ -423,20 +423,20 @@ func assertDisabledCacheWritesAreNoOps(
ctx := t.Context()
err := cache.StoreVariant(
err := cache.PutVariant(
ctx, imagerequest.CacheKey(req), bytes.NewReader([]byte("data")),
)
if err != nil {
t.Fatalf("StoreVariant on disabled cache must be a no-op, got error: %v", err)
t.Fatalf("PutVariant on disabled cache must be a no-op, got error: %v", err)
}
key, err := cache.StoreSource(ctx, req, bytes.NewReader([]byte("data")))
key, err := cache.PutSource(ctx, req.Source, bytes.NewReader([]byte("data")))
if err != nil {
t.Fatalf("StoreSource on disabled cache must be a no-op, got error: %v", err)
t.Fatalf("PutSource on disabled cache must be a no-op, got error: %v", err)
}
if key != "" {
t.Errorf("StoreSource on disabled cache returned key %q, want empty", key)
t.Errorf("PutSource on disabled cache returned key %q, want empty", key)
}
}
@@ -449,13 +449,13 @@ func assertDisabledCacheReadsAlwaysMiss(
ctx := t.Context()
key, err := cache.LookupSource(ctx, req)
if err != nil {
t.Fatalf("LookupSource on disabled cache failed: %v", err)
reader, _, err := cache.Source(ctx, req.Source)
if !errors.Is(err, ErrNotFound) {
t.Fatalf("Source on disabled cache error = %v, want %v", err, ErrNotFound)
}
if key != "" {
t.Errorf("LookupSource on disabled cache = %q, want empty", key)
if reader != nil {
t.Errorf("Source on disabled cache = reader %v, want none", reader)
}
}
@@ -544,7 +544,7 @@ func TestEvictionRunsUnderWritePressure(t *testing.T) {
}
// TestEvictionRunsOnPeriodicSchedule writes three variant files straight
// to disk, bypassing StoreVariant, so they have no accounting rows and no
// to disk, bypassing PutVariant, so they have no accounting rows and no
// write-pressure notification fires. Only a periodic reconciliation pass
// can then adopt them, and only the eviction pass that follows it can
// evict them.
@@ -697,7 +697,7 @@ func TestStartEvictionReconcilesAccountingWithDisk(t *testing.T) {
// TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup proves
// reconciliation is not a one-shot startup-only pass: it must also run
// on the periodic ticker, so a variant file that lands on disk with no
// accounting row well after startup (e.g. because StoreVariant's
// accounting row well after startup (e.g. because PutVariant's
// best-effort accounting insert failed under transient contention, or
// any other cause of an untracked file appearing during steady-state
// operation) is still adopted into accounting eventually, rather than
@@ -733,7 +733,7 @@ func TestPeriodicReconciliationAdoptsFileThatAppearsAfterStartup(t *testing.T) {
// Simulate a variant whose accounting insert failed after the
// process was already running and serving requests: the content
// file is written directly, bypassing StoreVariant's (and thus its
// file is written directly, bypassing PutVariant's (and thus its
// accounting insert) entirely, exactly as would happen if that
// insert had failed and only the file write had succeeded.
untracked := bytes.Repeat([]byte{0x41}, 900)
-29
View File
@@ -1,29 +0,0 @@
// Package imgcache provides types for the image caching proxy.
package imgcache
import "io"
// ImageResponse represents a processed image ready to serve
type ImageResponse struct {
// Content is the image data reader
Content io.ReadCloser
// ContentLength is the size in bytes (-1 if unknown)
ContentLength int64
// ContentType is the MIME type of the response
ContentType string
// ETag is the entity tag for caching
ETag string
// CacheStatus indicates HIT or MISS
CacheStatus CacheStatus
// FetchedBytes is the number of bytes fetched from upstream (0 if cache hit)
FetchedBytes int64
}
// CacheStatus indicates whether the response was served from cache.
type CacheStatus string
// Cache status values for response headers.
const (
CacheHit CacheStatus = "HIT"
CacheMiss CacheStatus = "MISS"
)
-8
View File
@@ -1,8 +0,0 @@
package imgcache
import "time"
// CacheConfig defaults.
const (
DefaultNegativeTTL = 5 * time.Minute
)
@@ -17,7 +17,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
db := setupTestDB(t)
dir := t.TempDir()
cache, err := NewCache(db, CacheConfig{
cache, err := NewCache(db, Config{
StateDir: dir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
@@ -35,7 +35,7 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
}
// Initially should not be in negative cache
hit, err := cache.checkNegativeCache(ctx, req)
hit, err := cache.Failure(ctx, req.Source)
if err != nil {
t.Fatal(err)
}
@@ -45,13 +45,13 @@ func TestNegativeCache_StoreAndCheck(t *testing.T) {
}
// Store a negative entry
err = cache.StoreNegative(ctx, req)
err = cache.PutFailure(ctx, req.Source)
if err != nil {
t.Fatal(err)
}
// Now should be in negative cache
hit, err = cache.checkNegativeCache(ctx, req)
hit, err = cache.Failure(ctx, req.Source)
if err != nil {
t.Fatal(err)
}
@@ -67,7 +67,7 @@ func TestNegativeCache_Expired(t *testing.T) {
db := setupTestDB(t)
dir := t.TempDir()
cache, err := NewCache(db, CacheConfig{
cache, err := NewCache(db, Config{
StateDir: dir,
NegativeTTL: 1 * time.Millisecond, // very short TTL
MaxBytes: 1 << 30,
@@ -85,7 +85,7 @@ func TestNegativeCache_Expired(t *testing.T) {
}
// Store a negative entry with very short TTL
err = cache.StoreNegative(ctx, req)
err = cache.PutFailure(ctx, req.Source)
if err != nil {
t.Fatal(err)
}
@@ -94,7 +94,7 @@ func TestNegativeCache_Expired(t *testing.T) {
time.Sleep(10 * time.Millisecond)
// Should no longer be in negative cache
hit, err := cache.checkNegativeCache(ctx, req)
hit, err := cache.Failure(ctx, req.Source)
if err != nil {
t.Fatal(err)
}
@@ -104,71 +104,6 @@ func TestNegativeCache_Expired(t *testing.T) {
}
}
// TestService_Get_ServesCachedVariantOfNegativeCachedURL caches a variant of
// the test photo, then keeps a failed fetch of the photo's URL in the negative
// cache, as a failed fetch for another variant does once the source has been
// evicted. The cached variant is still served, and a variant not in the cache
// is made from the cached source. Once the source is evicted, a variant not in
// the cache needs a fetch and is refused.
func TestService_Get_ServesCachedVariantOfNegativeCachedURL(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
resp, err := svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() first request error = %v", err)
}
_ = resp.Content.Close()
err = svc.cache.StoreNegative(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("StoreNegative() error = %v", err)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil {
t.Fatalf("Get() of the cached variant error = %v", err)
}
_ = resp.Content.Close()
if resp.CacheStatus != CacheHit {
t.Errorf("cached variant CacheStatus = %q, want %q",
resp.CacheStatus, CacheHit)
}
resp, err = svc.Get(t.Context(), photoVariant(fixtures, 40, imagerequest.FitCover))
t.Logf("Get() of a variant made from the cached source: error = %v", err)
if err != nil {
t.Fatalf("Get() of a variant made from the cached source error = %v", err)
}
_ = resp.Content.Close()
sourceKey, err := svc.cache.LookupSource(t.Context(),
photoVariant(fixtures, 85, imagerequest.FitCover))
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v, want the cached source", sourceKey, err)
}
err = svc.cache.evictSourceBlob(t.Context(), sourceKey)
if err != nil {
t.Fatalf("evictSourceBlob() error = %v", err)
}
_, err = svc.Get(t.Context(), photoVariant(fixtures, 60, imagerequest.FitCover))
t.Logf("Get() of a variant whose source was evicted: error = %v", err)
if !errors.Is(err, ErrNegativeCached) {
t.Errorf("Get() of a variant whose source was evicted error = %v, want %v",
err, ErrNegativeCached)
}
}
// TestEvictionLoopRemovesExpiredNegativeCacheRows stores a failed fetch
// that expires a second later and starts the eviction loop with a short
// interval: a later tick must remove its row although its URL is never
@@ -190,7 +125,7 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t), CacheConfig{
cache, err := NewCache(evictionTestDB(t), Config{
StateDir: t.TempDir(),
NegativeTTL: time.Second,
MaxBytes: tt.maxBytes,
@@ -203,9 +138,9 @@ func TestEvictionLoopRemovesExpiredNegativeCacheRows(t *testing.T) {
Source: imagerequest.Source{Host: testHostExample, Path: "/gone.jpg"},
}
err = cache.StoreNegative(t.Context(), expired)
err = cache.PutFailure(t.Context(), expired.Source)
if err != nil {
t.Fatalf("StoreNegative failed: %v", err)
t.Fatalf("PutFailure failed: %v", err)
}
const notExpiredPath = "/later.jpg"
@@ -269,7 +204,7 @@ func TestCleanExpiredDeletesInBatches(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -325,7 +260,7 @@ func TestCleanExpiredRemovesMoreThanOneBatch(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -357,7 +292,7 @@ func TestCleanExpiredStopsOnceContextEnds(t *testing.T) {
t.Parallel()
cache, err := NewCache(evictionTestDB(t),
CacheConfig{StateDir: t.TempDir(), MaxBytes: 1 << 30},
Config{StateDir: t.TempDir(), MaxBytes: 1 << 30},
prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
@@ -410,15 +345,3 @@ func negativeCacheTableRowCount(tb testing.TB, cache *Cache) int {
return count
}
func TestService_Get_ReturnsErrorForNegativeCachedURL(t *testing.T) {
t.Parallel()
// This test verifies that Service.Get() checks the negative cache
// We can't easily test the full pipeline without vips, but we can
// verify the error type
err := ErrNegativeCached
if !errors.Is(err, ErrNegativeCached) {
t.Error("ErrNegativeCached should be identifiable with errors.Is")
}
}
@@ -12,67 +12,6 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// getImage calls Get for req and returns the image it answered with and
// its cache status.
func getImage(
t *testing.T, svc *Service, req *imagerequest.Request,
) ([]byte, CacheStatus) {
t.Helper()
resp, err := svc.Get(t.Context(), req)
if err != nil {
t.Fatalf("Get() of quality %d error = %v", req.Transform.Quality, err)
}
defer func() { _ = resp.Content.Close() }()
body, err := io.ReadAll(resp.Content)
if err != nil {
t.Fatalf("reading the image of quality %d: %v", req.Transform.Quality, err)
}
return body, resp.CacheStatus
}
// fileSize returns the size of the file at path.
func fileSize(t *testing.T, path string) int64 {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
return info.Size()
}
// truncateFile cuts the file at path down to size bytes.
func truncateFile(t *testing.T, path string, size int64) {
t.Helper()
t.Logf("cutting %s from %d to %d bytes",
filepath.Base(path), fileSize(t, path), size)
err := os.Truncate(path, size)
if err != nil {
t.Fatalf("truncating %s: %v", path, err)
}
}
// cachedSourcePath returns the path of the cached source file req uses.
func cachedSourcePath(
t *testing.T, svc *Service, req *imagerequest.Request,
) string {
t.Helper()
sourceKey, err := svc.cache.LookupSource(t.Context(), req)
if err != nil || sourceKey == "" {
t.Fatalf("LookupSource() = %q, %v; want the cached source", sourceKey, err)
}
return svc.cache.srcContent.keyToPath(sourceKey)
}
// reconcileAsAtStartup runs the cache's reconciliation pass, as pixa does
// when it starts, and returns what the pass logged.
func reconcileAsAtStartup(t *testing.T, cache *Cache) string {
@@ -107,129 +46,6 @@ func assertWarningNames(t *testing.T, logs, path string) {
t.Errorf("reconciliation logged no warning naming %s", path)
}
// TestService_Get_FetchesAgainForEmptyCachedSource empties the cached source
// file, as a crash or a restored snapshot could, and runs the reconciliation
// pass, which must remove it with a warning naming it. It then asks for two
// new sizes of the source: the first must fetch it again and cache it whole,
// so that the second is made from the cache without a third fetch.
func TestService_Get_FetchesAgainForEmptyCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, 0)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2: the empty cached source "+
"was kept and fetched again on every new size", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// TestService_Get_FetchesAgainForPartialCachedSource cuts the cached source
// file to half its size, as a crash or a restored snapshot could, and runs
// the reconciliation pass, which must remove it with a warning naming it.
// It then asks for two new sizes of the source: the first must fetch it
// again rather than fail to decode the partial file, and the second is
// made from the cache.
func TestService_Get_FetchesAgainForPartialCachedSource(t *testing.T) {
t.Parallel()
svc, fixtures := SetupTestService(t)
getImage(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
path := cachedSourcePath(t, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
wholeSize := fileSize(t, path)
truncateFile(t, path, wholeSize/2)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
getImage(t, svc, photoVariant(fixtures, 40, imagerequest.FitCover))
getImage(t, svc, photoVariant(fixtures, 30, imagerequest.FitCover))
fetches := readCounters(svc.cache).upstreamFetchCount
t.Logf("upstream fetches = %d", fetches)
if fetches != 2 {
t.Errorf("upstream fetches = %d, want 2", fetches)
}
size := fileSize(t, path)
if size != wholeSize {
t.Errorf("cached source is %d bytes, want %d", size, wholeSize)
}
}
// assertConvertedAgain caches a variant of the photo, cuts its file to size
// bytes and runs the reconciliation pass, which must remove it with a
// warning naming it. It then checks that Get converts the image again
// instead of serving the file, and that the next Get serves the whole
// variant from the cache.
func assertConvertedAgain(t *testing.T, size int64) {
t.Helper()
svc, fixtures := SetupTestService(t)
req := photoVariant(fixtures, 85, imagerequest.FitCover)
want, _ := getImage(t, svc, req)
path := svc.cache.variants.keyToPath(imagerequest.CacheKey(req))
truncateFile(t, path, size)
assertWarningNames(t, reconcileAsAtStartup(t, svc.cache), path)
got, status := getImage(t, svc, req)
t.Logf("after the cut: %d bytes, %s", len(got), status)
if status != CacheMiss || !bytes.Equal(got, want) {
t.Errorf("after the cut: Get() = %d bytes, %s; want the %d bytes "+
"converted again, %s", len(got), status, len(want), CacheMiss)
}
got, status = getImage(t, svc, req)
t.Logf("next request: %d bytes, %s", len(got), status)
if status != CacheHit || !bytes.Equal(got, want) {
t.Errorf("next request: Get() = %d bytes, %s; want the %d bytes, %s",
len(got), status, len(want), CacheHit)
}
}
// TestService_Get_ConvertsAgainForEmptyCachedVariant empties a cached
// variant file, as a crash or a restored snapshot could, and checks that it
// is converted again rather than served empty.
func TestService_Get_ConvertsAgainForEmptyCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 0)
}
// TestService_Get_ConvertsAgainForPartialCachedVariant cuts a cached
// variant file to ten bytes, as a crash or a restored snapshot could, and
// checks that it is converted again rather than served cut short.
func TestService_Get_ConvertsAgainForPartialCachedVariant(t *testing.T) {
t.Parallel()
assertConvertedAgain(t, 10)
}
// TestReconciliationRemovesEmptyVariantFileWithoutRow checks that the
// reconciliation pass removes an empty variant file that has no
// accounting row, with a warning naming it, instead of adding a row for
@@ -5,6 +5,7 @@ import (
"context"
"database/sql"
"errors"
"io"
"log/slog"
"strings"
"testing"
@@ -15,8 +16,7 @@ import (
// pendingAccessTimesWait is how long a test waits for the database's
// connection, for an access time to reach the database, for a write to start
// waiting for the connection, for a request to return past its deadline, or
// for StopPendingAccessTimeWrites to return.
// waiting for the connection, or for StopPendingAccessTimeWrites to return.
const pendingAccessTimesWait = 5 * time.Second
// otherWritesWait is how long a test gives other writes, once one write waits
@@ -56,52 +56,6 @@ func waitForConnectionWaits(t *testing.T, db *sql.DB, waits int64) {
}
}
// TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy holds the
// database's one connection while a request whose fetch is held reaches its
// deadline. The request must return with the deadline's error while the
// connection is still held, as it waits for no database write past its
// deadline.
func TestService_Get_ReturnsByItsDeadlineWhileTheDatabaseIsBusy(t *testing.T) {
t.Parallel()
svc, fixtures, fetcher := setupHeldFetchService(t)
svc.cache.StartPendingAccessTimeWrites()
defer func() { _ = svc.cache.StopPendingAccessTimeWrites(t.Context()) }()
// Room for the request to reach its fetch on a busy host
const timeout = 2 * time.Second
ctx, cancel := context.WithTimeout(t.Context(), timeout)
defer cancel()
deadline, _ := ctx.Deadline()
results := startGet(ctx, svc, photoVariant(fixtures, 85, imagerequest.FitCover))
// The request has made its database reads by the time it fetches.
select {
case <-fetcher.started:
case <-time.After(timeout):
t.Fatal("request did not reach its fetch by its deadline")
}
releaseDatabase := holdDatabase(t, svc.cache)
defer releaseDatabase()
select {
case got := <-results:
t.Logf("Get() returned %v after its deadline, error = %v",
time.Since(deadline), got.err)
if !errors.Is(got.err, context.DeadlineExceeded) {
t.Errorf("Get() error = %v, want %v", got.err, context.DeadlineExceeded)
}
case <-time.After(time.Until(deadline) + pendingAccessTimesWait):
t.Fatal("request did not return while the database was busy")
}
}
// TestStopPendingAccessTimeWrites_LetsTheWriteUnderWayFinish holds the
// database's one connection while the goroutine that writes the pending
// access times waits for it, hands over another access time, then stops that
@@ -242,11 +196,11 @@ func TestStopPendingAccessTimeWrites_ReturnsWhenItsContextEnds(t *testing.T) {
}
}
// TestLookupSource_HandsAccessTimeToTheBackgroundWriter finds a cached source
// whose access time is an hour old. LookupSource must not write the access
// TestSource_HandsAccessTimeToTheBackgroundWriter finds a cached source
// whose access time is an hour old. Source must not write the access
// time itself but hand it over to be written, and StopPendingAccessTimeWrites
// must then write it.
func TestLookupSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
func TestSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -261,15 +215,21 @@ func TestLookupSource_HandsAccessTimeToTheBackgroundWriter(t *testing.T) {
`
anHourAgoInDatabase := anHourAgo.UTC().Format(sqliteTimestampFormat)
found, err := cache.LookupSource(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{Host: "example.com", Path: "/photo.jpg"},
})
if err != nil || found != key {
t.Fatalf("LookupSource() = %q, %v; want %q", found, err, key)
reader, _, err := cache.Source(t.Context(),
imagerequest.Source{Host: "example.com", Path: "/photo.jpg"})
if err != nil {
t.Fatalf("Source() error = %v; want the source of %q", err, key)
}
found, err := io.ReadAll(reader)
_ = reader.Close()
if err != nil || string(found) != "source bytes" {
t.Fatalf("Source() reads %q, %v; want the source of %q", found, err, key)
}
if countRows(t, cache, touchedQuery, string(key), anHourAgoInDatabase) != 0 {
t.Error("LookupSource wrote the source's access time itself")
t.Error("Source wrote the source's access time itself")
}
if waiting := len(cache.pendingSourceAccessTimes); waiting != 1 {
@@ -10,11 +10,11 @@ import (
"sneak.berlin/go/pixa/internal/imagerequest"
)
// TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
// TestPutSource_StoresIdenticalBytesOfTwoURLsTwice stores the same bytes
// for two source URLs and checks that each URL has a source_content row of
// its own and a file of its own, named by its source key under
// cache/sources/<ab>/<cd>/.
func TestStoreSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) {
func TestPutSource_StoresIdenticalBytesOfTwoURLsTwice(t *testing.T) {
t.Parallel()
cache, stateDir := newEvictionTestCache(t, 1<<30)
@@ -84,18 +84,12 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
t.Errorf("the evicted source still has %d source_content rows", n)
}
kept, err := cache.LookupSource(t.Context(), &imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"},
})
t.Logf("LookupSource() of the kept source = %q, error %v", kept, err)
reader, _, err := cache.Source(t.Context(),
imagerequest.Source{Host: testHostCDN, Path: "/kept.jpg"})
t.Logf("Source() of the kept source: error %v", err)
if err != nil || kept == "" {
t.Fatalf("LookupSource() of the kept source = %q, %v; want it cached", kept, err)
}
reader, _, err := cache.GetSourceContent(kept)
if err != nil {
t.Fatalf("GetSourceContent() of the kept source error = %v", err)
t.Fatalf("Source() of the kept source error = %v; want it cached", err)
}
defer func() { _ = reader.Close() }()
@@ -111,10 +105,10 @@ func TestEvictSourceBlob_LeavesSourceWithIdenticalBytesServed(t *testing.T) {
}
}
// TestStoreSource_FailedRowWriteLeavesNoFile drops source_content, so that
// StoreSource cannot write the source's row, and checks that the store fails
// TestPutSource_FailedRowWriteLeavesNoFile drops source_content, so that
// PutSource cannot write the source's row, and checks that the store fails
// and leaves no file under the source's key.
func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) {
func TestPutSource_FailedRowWriteLeavesNoFile(t *testing.T) {
t.Parallel()
cache, _ := newEvictionTestCache(t, 1<<30)
@@ -127,12 +121,12 @@ func TestStoreSource_FailedRowWriteLeavesNoFile(t *testing.T) {
req := &imagerequest.Request{
Source: imagerequest.Source{Host: testHostCDN, Path: testPathCat},
}
key, err := cache.StoreSource(t.Context(), req,
key, err := cache.PutSource(t.Context(), req.Source,
bytes.NewReader([]byte("source bytes")))
t.Logf("StoreSource() = %q, error = %v", key, err)
t.Logf("PutSource() = %q, error = %v", key, err)
if err == nil {
t.Error("StoreSource() error = nil, want the failed row write")
t.Error("PutSource() error = nil, want the failed row write")
}
if cache.srcContent.Exists(req.Source.Key()) {
+3 -225
View File
@@ -2,98 +2,19 @@ package imgcache
import (
"bytes"
"context"
"database/sql"
"image"
"image/color"
"image/gif"
"image/jpeg"
"image/png"
"io/fs"
"testing"
"testing/fstest"
"time"
"github.com/prometheus/client_golang/prometheus"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
)
// Shared test data literals, extracted as constants for goconst.
const (
testHostCDN = "cdn.example.com"
testHostExample = "example.com"
testPathCat = "/photos/cat.jpg"
testContentTypeJPEG = "image/jpeg"
testHostCDN = "cdn.example.com"
testHostExample = "example.com"
testPathCat = "/photos/cat.jpg"
)
// TestFixtures contains paths to test files in the mock filesystem.
type TestFixtures struct {
// Valid image files
GoodHostJPEG string // allowlisted host, valid JPEG
GoodHostPNG string // allowlisted host, valid PNG
GoodHostGIF string // allowlisted host, valid GIF
OtherHostJPEG string // non-allowlisted host, valid JPEG
OtherHostPNG string // non-allowlisted host, valid PNG
// Invalid/edge case files
InvalidFile string // file with wrong magic bytes
EmptyFile string // zero-byte file
TextFile string // text file masquerading as image
// Hostnames
GoodHost string // allowlisted hostname
OtherHost string // non-allowlisted hostname
}
// DefaultFixtures returns the standard test fixture paths.
func DefaultFixtures() *TestFixtures {
return &TestFixtures{
GoodHostJPEG: "goodhost.example.com/images/photo.jpg",
GoodHostPNG: "goodhost.example.com/images/logo.png",
GoodHostGIF: "goodhost.example.com/images/animation.gif",
OtherHostJPEG: "otherhost.example.com/uploads/image.jpg",
OtherHostPNG: "otherhost.example.com/uploads/icon.png",
InvalidFile: "goodhost.example.com/images/fake.jpg",
EmptyFile: "goodhost.example.com/images/empty.jpg",
TextFile: "goodhost.example.com/images/text.png",
GoodHost: "goodhost.example.com",
OtherHost: "otherhost.example.com",
}
}
// NewTestFS creates a mock filesystem with test images.
func NewTestFS(t *testing.T) (fs.FS, *TestFixtures) {
t.Helper()
fixtures := DefaultFixtures()
// Generate test images
jpegData := generateTestJPEG(t, 100, 100, color.RGBA{255, 0, 0, 255})
pngData := generateTestPNG(t, 100, 100, color.RGBA{0, 255, 0, 255})
gifData := generateTestGIF(t, 100, 100, color.RGBA{0, 0, 255, 255})
// Create the mock filesystem
mockFS := fstest.MapFS{
// Good host files
fixtures.GoodHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.GoodHostPNG: &fstest.MapFile{Data: pngData},
fixtures.GoodHostGIF: &fstest.MapFile{Data: gifData},
// Other host files
fixtures.OtherHostJPEG: &fstest.MapFile{Data: jpegData},
fixtures.OtherHostPNG: &fstest.MapFile{Data: pngData},
// Invalid files
fixtures.InvalidFile: &fstest.MapFile{Data: []byte("not a real image file content")},
fixtures.EmptyFile: &fstest.MapFile{Data: []byte{}},
fixtures.TextFile: &fstest.MapFile{Data: []byte("This is just text, not a PNG")},
}
return mockFS, fixtures
}
// generateTestJPEG creates a minimal valid JPEG image.
func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
@@ -114,146 +35,3 @@ func generateTestJPEG(t *testing.T, width, height int, c color.Color) []byte {
return buf.Bytes()
}
// generateTestPNG creates a minimal valid PNG image.
func generateTestPNG(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewRGBA(image.Rect(0, 0, width, height))
for y := range height {
for x := range width {
img.Set(x, y, c)
}
}
var buf bytes.Buffer
err := png.Encode(&buf, img)
if err != nil {
t.Fatalf("failed to encode test PNG: %v", err)
}
return buf.Bytes()
}
// generateTestGIF creates a minimal valid GIF image.
func generateTestGIF(t *testing.T, width, height int, c color.Color) []byte {
t.Helper()
img := image.NewPaletted(
image.Rect(0, 0, width, height),
[]color.Color{c, color.White},
)
for y := range height {
for x := range width {
img.SetColorIndex(x, y, 0)
}
}
var buf bytes.Buffer
err := gif.Encode(&buf, img, nil)
if err != nil {
t.Fatalf("failed to encode test GIF: %v", err)
}
return buf.Bytes()
}
// SetupTestService creates a Service with mock fetcher for testing.
func SetupTestService(
t *testing.T, opts ...TestServiceOption,
) (*Service, *TestFixtures) {
t.Helper()
mockFS, fixtures := NewTestFS(t)
cfg := &testServiceConfig{
allowlist: []string{fixtures.GoodHost},
signingKey: "test-signing-key-must-be-32-chars",
}
for _, opt := range opts {
opt(cfg)
}
// Create temp directory for cache
tmpDir := t.TempDir()
// Create in-memory database
db := setupServiceTestDB(t)
cache, err := NewCache(db, CacheConfig{
StateDir: tmpDir,
NegativeTTL: 5 * time.Minute,
MaxBytes: 1 << 30,
}, prometheus.NewRegistry())
if err != nil {
t.Fatalf("failed to create cache: %v", err)
}
svc, err := NewService(&ServiceConfig{
Cache: cache,
Fetcher: httpfetcher.NewMock(mockFS),
Processor: imageprocessor.New(imageprocessor.Params{
MaxInputBytes: 50 << 20,
MaxConcurrentProcessing: 2,
}),
SigningKey: cfg.signingKey,
Allowlist: cfg.allowlist,
MaxResponseSize: 50 << 20,
})
if err != nil {
t.Fatalf("failed to create service: %v", err)
}
return svc, fixtures
}
// setupServiceTestDB creates an in-memory SQLite database for testing
// using the production schema.
func setupServiceTestDB(t *testing.T) *sql.DB {
t.Helper()
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open test db: %v", err)
}
// Use the real production schema via migrations
err = database.ApplyMigrations(context.Background(), db, nil)
if err != nil {
t.Fatalf("failed to apply migrations: %v", err)
}
return db
}
type testServiceConfig struct {
allowlist []string
signingKey string
}
// TestServiceOption configures the test service.
type TestServiceOption func(*testServiceConfig)
// WithAllowlist sets the allowlist for the test service.
func WithAllowlist(hosts ...string) TestServiceOption {
return func(c *testServiceConfig) {
c.allowlist = hosts
}
}
// WithSigningKey sets the signing key for the test service.
func WithSigningKey(key string) TestServiceOption {
return func(c *testServiceConfig) {
c.signingKey = key
}
}
// WithNoAllowlist removes all allowlisted hosts.
func WithNoAllowlist() TestServiceOption {
return func(c *testServiceConfig) {
c.allowlist = nil
}
}
+26 -21
View File
@@ -1,5 +1,6 @@
// Package providers builds from the config the parts pixad serves images
// and logins with, and provides them to the fx graph:
// - the keys derived from the signing key
// - the image cache, with its default limit
// - the fetcher
// - the processor
@@ -25,8 +26,10 @@ import (
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imageprocessor"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/logger"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -35,6 +38,7 @@ import (
// the cache registers its counters on.
func Constructors() []any {
return []any{
newKeys,
newCache,
newFetcherConfig,
httpfetcher.New,
@@ -45,6 +49,12 @@ func Constructors() []any {
}
}
// newKeys derives the keys of seal.Keys from the signing key. fx calls it
// once, at startup.
func newKeys(cfg *config.Config) (seal.Keys, error) {
return seal.DeriveKeys(cfg.SigningKey)
}
// cacheParams defines dependencies for newCache.
type cacheParams struct {
fx.In
@@ -70,7 +80,7 @@ func newCache(lc fx.Lifecycle, params cacheParams) (*imgcache.Cache, error) {
return nil, err
}
cache, err := imgcache.NewCache(db, imgcache.CacheConfig{
cache, err := imgcache.NewCache(db, imgcache.Config{
StateDir: cfg.StateDir,
NegativeTTL: imgcache.DefaultNegativeTTL,
MaxBytes: maxBytes,
@@ -212,21 +222,15 @@ type imageServiceParams struct {
}
// newImageService builds the image service from the cache, the fetcher
// and the processor, with the signing key, the allowlist and the upstream
// settings of the config.
// and the processor, with the largest upstream response the config allows.
func newImageService(
_ fx.Lifecycle, params imageServiceParams,
) (*imgcache.Service, error) {
cfg := params.Config
return imgcache.NewService(&imgcache.ServiceConfig{
) (*imageproxy.Proxy, error) {
return imageproxy.NewProxy(&imageproxy.Config{
Cache: params.Cache,
Fetcher: params.Fetcher,
Processor: params.Processor,
SigningKey: cfg.SigningKey,
Allowlist: cfg.AllowlistHosts,
AllowHTTP: cfg.AllowHTTP,
MaxResponseSize: cfg.UpstreamMaxResponseSize,
MaxResponseSize: params.Config.UpstreamMaxResponseSize,
Logger: params.Logger.Get(),
})
}
@@ -235,22 +239,23 @@ func newImageService(
type sessionManagerParams struct {
fx.In
Config *config.Config
Keys seal.Keys
Database *database.Database
}
// newSessionManager builds the session manager. Session cookies are
// always Secure, HttpOnly and SameSite=Strict. The random value that each
// logout replaces is saved in the database, so it outlasts a restart.
// newSessionManager builds the session manager with its two keys. Session
// cookies are always Secure, HttpOnly and SameSite=Strict. The random value
// that each logout replaces is saved in the database, so it outlasts a
// restart.
func newSessionManager(
_ fx.Lifecycle, params sessionManagerParams,
) (*session.Manager, error) {
return session.NewManagerWithDatabase(
context.Background(), params.Config.SigningKey, params.Database.DB())
return session.NewManagerWithDatabase(context.Background(),
params.Keys.SessionHashKey, params.Keys.SessionBlockKey,
params.Database.DB())
}
// newEncryptedURLGenerator builds the encrypted URL generator, with a key
// derived from the signing key.
func newEncryptedURLGenerator(cfg *config.Config) (*encurl.Generator, error) {
return encurl.NewGenerator(cfg.SigningKey)
// newEncryptedURLGenerator builds the encrypted URL generator with its key.
func newEncryptedURLGenerator(keys seal.Keys) *encurl.Generator {
return encurl.NewGenerator(keys.EncryptedURLKey)
}
+57
View File
@@ -0,0 +1,57 @@
package seal
// The salt of each key that DeriveKeys derives. Each key has a salt of its
// own, so no two uses share a key.
const (
sessionHashKeySalt = "pixa-session-hash-v1"
sessionBlockKeySalt = "pixa-session-block-v1"
encryptedURLKeySalt = "pixa-urlenc-v1"
csrfKeySalt = "pixa-csrf-v1"
)
// Keys holds the keys derived from the signing key, one for each use. The
// signing key itself is used for two things only: it is the HMAC key of signed
// URLs and the password of the login page.
type Keys struct {
// SessionHashKey signs the session cookie.
SessionHashKey [KeySize]byte
// SessionBlockKey encrypts the session cookie.
SessionBlockKey [KeySize]byte
// EncryptedURLKey encrypts the token of an encrypted URL.
EncryptedURLKey [KeySize]byte
// CSRFKey signs the CSRF token cookie of the login and generator pages.
CSRFKey [KeySize]byte
}
// DeriveKeys derives each key of Keys from signingKey, with DeriveKey and the
// salt of that key.
func DeriveKeys(signingKey string) (Keys, error) {
masterKey := []byte(signingKey)
sessionHashKey, err := DeriveKey(masterKey, sessionHashKeySalt)
if err != nil {
return Keys{}, err
}
sessionBlockKey, err := DeriveKey(masterKey, sessionBlockKeySalt)
if err != nil {
return Keys{}, err
}
encryptedURLKey, err := DeriveKey(masterKey, encryptedURLKeySalt)
if err != nil {
return Keys{}, err
}
csrfKey, err := DeriveKey(masterKey, csrfKeySalt)
if err != nil {
return Keys{}, err
}
return Keys{
SessionHashKey: sessionHashKey,
SessionBlockKey: sessionBlockKey,
EncryptedURLKey: encryptedURLKey,
CSRFKey: csrfKey,
}, nil
}
+42
View File
@@ -0,0 +1,42 @@
package seal_test
import (
"bytes"
"testing"
"sneak.berlin/go/pixa/internal/seal"
)
// TestDeriveKeys_KeysDiffer verifies that the keys DeriveKeys derives differ
// from each other and from the signing key.
func TestDeriveKeys_KeysDiffer(t *testing.T) {
t.Parallel()
// 32 bytes, as long as a derived key, so that no key differs from it by
// its length alone
const signingKey = "0123456789abcdef0123456789abcdef"
keys, err := seal.DeriveKeys(signingKey)
if err != nil {
t.Fatalf("DeriveKeys() error = %v", err)
}
named := []struct {
name string
key []byte
}{
{"the signing key", []byte(signingKey)},
{"SessionHashKey", keys.SessionHashKey[:]},
{"SessionBlockKey", keys.SessionBlockKey[:]},
{"EncryptedURLKey", keys.EncryptedURLKey[:]},
{"CSRFKey", keys.CSRFKey[:]},
}
for i, a := range named {
for _, b := range named[i+1:] {
if bytes.Equal(a.key, b.key) {
t.Errorf("%s equals %s", a.name, b.name)
}
}
}
}
@@ -7,9 +7,11 @@ import (
"net/http"
"net/http/httptest"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// TestImageProxyRefusesComputedSideAboveMaximum requests sizes of a 3000x600
@@ -47,11 +49,13 @@ func TestImageProxyRefusesComputedSideAboveMaximum(t *testing.T) {
s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
t.Fatalf("seal.DeriveKeys: %v", err)
}
generator := encurl.NewGenerator(keys.EncryptedURLKey)
tests := []struct {
path string
// size is the size of the /v1/image/ URL, and width and height are
@@ -76,7 +80,7 @@ func TestImageProxyRefusesComputedSideAboveMaximum(t *testing.T) {
Size: imagerequest.Size{Width: tt.width, Height: tt.height},
Format: imagerequest.FormatJPEG,
},
})
}, time.Time{})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
+7 -3
View File
@@ -5,9 +5,11 @@ import (
"net/http"
"net/http/httptest"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// TestImageProxyRefusesFitOutsideAboveMaximum requests sizes with fit=outside
@@ -41,11 +43,13 @@ func TestImageProxyRefusesFitOutsideAboveMaximum(t *testing.T) {
s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
t.Fatalf("seal.DeriveKeys: %v", err)
}
generator := encurl.NewGenerator(keys.EncryptedURLKey)
tests := []struct {
path string
width, height int
@@ -66,7 +70,7 @@ func TestImageProxyRefusesFitOutsideAboveMaximum(t *testing.T) {
Format: imagerequest.FormatJPEG,
FitMode: imagerequest.FitOutside,
},
})
}, time.Time{})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
+11 -9
View File
@@ -14,8 +14,9 @@ import (
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imageproxy"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/imgcache"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
"sneak.berlin/go/pixa/internal/signature"
)
@@ -226,7 +227,7 @@ func TestSourcePathsHoldingSpaceAndPercent20AreFetchedApart(t *testing.T) {
for _, imageURL := range imageURLs {
cacheStatus := getImage(t, s, imageURL).Header().Get("X-Pixa-Cache")
if cacheStatus != string(imgcache.CacheHit) {
if cacheStatus != string(imageproxy.CacheHit) {
t.Errorf("second GET %s: X-Pixa-Cache = %q, want HIT",
imageURL, cacheStatus)
}
@@ -255,7 +256,7 @@ func TestSourceQueryWrittenTwoWaysIsOneUpstreamRequest(t *testing.T) {
second := getImage(t, s,
"/v1/image/"+upstreamHost+"/photo.png%3Fname=my%20cat/orig.png")
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
if got := second.Header().Get("X-Pixa-Cache"); got != string(imageproxy.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
@@ -364,7 +365,7 @@ func TestGeneratorMakesOneImageURLForTwoSpellingsOfASourceURL(t *testing.T) {
getImage(t, s, imageURLs[0])
second := getImage(t, s, imageURLs[1])
if got := second.Header().Get("X-Pixa-Cache"); got != string(imgcache.CacheHit) {
if got := second.Header().Get("X-Pixa-Cache"); got != string(imageproxy.CacheHit) {
t.Errorf("second X-Pixa-Cache = %q, want HIT", got)
}
@@ -524,8 +525,7 @@ func signedImageURL(t *testing.T, sourceHost string) string {
Quality: 85,
FitMode: imagerequest.FitCover,
},
Expires: expires,
})
}, expires)
// Go's percent-encoding of a URL path writes the brackets of the IPv6
// host as %5B and %5D.
@@ -541,15 +541,17 @@ func signedImageURL(t *testing.T, sourceHost string) string {
func encryptedImageURL(t *testing.T, sourceHost string) string {
t.Helper()
generator, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
t.Fatalf("seal.DeriveKeys: %v", err)
}
generator := encurl.NewGenerator(keys.EncryptedURLKey)
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: sourceHost, Path: ipv6SourcePath},
Transform: imagerequest.Transform{Format: imagerequest.FormatPNG},
})
}, time.Time{})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
+7 -3
View File
@@ -6,9 +6,11 @@ import (
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/imagerequest"
"sneak.berlin/go/pixa/internal/seal"
)
// svgServedAsAVIF is an SVG whose first bytes pass the first-bytes check for
@@ -33,11 +35,13 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
s, _, _ := startImageProxy(t, upstream)
generator, err := encurl.NewGenerator(testSigningKey)
keys, err := seal.DeriveKeys(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator: %v", err)
t.Fatalf("seal.DeriveKeys: %v", err)
}
generator := encurl.NewGenerator(keys.EncryptedURLKey)
encryptedURL := func(format imagerequest.Format) string {
token, err := generator.Generate(&imagerequest.Request{
Source: imagerequest.Source{Host: upstreamHost, Path: "/drawing.avif"},
@@ -45,7 +49,7 @@ func TestImageProxyRefusesSVGServedAsAVIF(t *testing.T) {
Size: imagerequest.Size{Width: 32, Height: 24},
Format: format,
},
})
}, time.Time{})
if err != nil {
t.Fatalf("encurl Generate: %v", err)
}
+9 -28
View File
@@ -22,10 +22,6 @@ import (
const (
CookieName = "pixa_session"
SessionTTL = 30 * 24 * time.Hour // 30 days
// HKDF salts for key derivation
hashKeySalt = "pixa-session-hash-v1"
blockKeySalt = "pixa-session-block-v1"
)
// Errors returned by session operations.
@@ -66,35 +62,23 @@ type Manager struct {
logoutValue string
}
// NewManager creates a session manager with keys derived from the signing key.
// It makes a new random value and keeps it in memory only, so a restart ends
// every session; NewManagerWithDatabase keeps the value in the database.
// NewManager creates a session manager whose session cookies hashKey signs
// and blockKey encrypts. It makes a new random value and keeps it in memory
// only, so a restart ends every session; NewManagerWithDatabase keeps the
// value in the database.
//
// Session cookies always carry the Secure, HttpOnly, and SameSite=Strict
// attributes; this cannot be configured. Browsers treat http://localhost as a
// trustworthy origin and accept Secure cookies there, so local development
// keeps working.
func NewManager(signingKey string) (*Manager, error) {
masterKey := []byte(signingKey)
// Derive separate keys for HMAC (hash) and encryption (block)
hashKey, err := seal.DeriveKey(masterKey, hashKeySalt)
if err != nil {
return nil, err
}
blockKey, err := seal.DeriveKey(masterKey, blockKeySalt)
if err != nil {
return nil, err
}
func NewManager(hashKey, blockKey [seal.KeySize]byte) *Manager {
sc := securecookie.New(hashKey[:], blockKey[:])
sc.MaxAge(int(SessionTTL.Seconds()))
return &Manager{
sc: sc,
logoutValue: rand.Text(),
}, nil
}
}
// NewManagerWithDatabase creates a session manager like NewManager that keeps
@@ -102,14 +86,11 @@ func NewManager(signingKey string) (*Manager, error) {
// NewManager made when db has none, and otherwise starts from the one saved
// there.
func NewManagerWithDatabase(
ctx context.Context, signingKey string, db *sql.DB,
ctx context.Context, hashKey, blockKey [seal.KeySize]byte, db *sql.DB,
) (*Manager, error) {
m, err := NewManager(signingKey)
if err != nil {
return nil, err
}
m := NewManager(hashKey, blockKey)
_, err = db.ExecContext(ctx,
_, err := db.ExecContext(ctx,
"INSERT OR IGNORE INTO logout (id, logout_value) VALUES (1, ?)",
m.logoutValue)
if err != nil {
@@ -5,6 +5,7 @@ import (
"net/http/httptest"
"testing"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -20,11 +21,13 @@ import (
func TestSessionCookieAttributesAlwaysSecure(t *testing.T) {
t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345")
keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil {
t.Fatalf("NewManager() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
writePaths := []struct {
name string
setCookie func(t *testing.T, w http.ResponseWriter)
+15 -4
View File
@@ -8,6 +8,7 @@ import (
"testing"
"sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
@@ -80,8 +81,13 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
db := newTestDatabase(t)
keys, err := seal.DeriveKeys(logoutTestSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db)
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err)
}
@@ -109,7 +115,7 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
}
restarted, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, db)
t.Context(), keys.SessionHashKey, keys.SessionBlockKey, db)
if err != nil {
t.Fatalf("NewManagerWithDatabase() after restart error = %v", err)
}
@@ -129,8 +135,13 @@ func TestLogout_EndsEverySessionMadeBeforeIt(t *testing.T) {
func TestLogout_WithoutSession_EndsNoSession(t *testing.T) {
t.Parallel()
mgr, err := session.NewManagerWithDatabase(
t.Context(), logoutTestSigningKey, newTestDatabase(t))
keys, err := seal.DeriveKeys(logoutTestSigningKey)
if err != nil {
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr, err := session.NewManagerWithDatabase(t.Context(),
keys.SessionHashKey, keys.SessionBlockKey, newTestDatabase(t))
if err != nil {
t.Fatalf("NewManagerWithDatabase() error = %v", err)
}
+19 -9
View File
@@ -7,17 +7,20 @@ import (
"testing"
"time"
"sneak.berlin/go/pixa/internal/seal"
"sneak.berlin/go/pixa/internal/session"
)
func TestManager_CreateAndValidate(t *testing.T) {
t.Parallel()
mgr, err := session.NewManager("test-signing-key-12345")
keys, err := seal.DeriveKeys("test-signing-key-12345")
if err != nil {
t.Fatalf("NewManager() error = %v", err)
t.Fatalf("seal.DeriveKeys() error = %v", err)
}
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
// Create a session
w := httptest.NewRecorder()
@@ -69,7 +72,8 @@ func TestManager_CreateAndValidate(t *testing.T) {
func TestManager_ValidateSession_NoCookie(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
@@ -86,7 +90,8 @@ func TestManager_ValidateSession_NoCookie(t *testing.T) {
func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
req.AddCookie(&http.Cookie{
@@ -110,8 +115,10 @@ func TestManager_ValidateSession_TamperedCookie(t *testing.T) {
func TestManager_ValidateSession_WrongKey(t *testing.T) {
t.Parallel()
mgr1, _ := session.NewManager("signing-key-1")
mgr2, _ := session.NewManager("signing-key-2")
keys1, _ := seal.DeriveKeys("signing-key-1")
keys2, _ := seal.DeriveKeys("signing-key-2")
mgr1 := session.NewManager(keys1.SessionHashKey, keys1.SessionBlockKey)
mgr2 := session.NewManager(keys2.SessionHashKey, keys2.SessionBlockKey)
// Create session with mgr1
w := httptest.NewRecorder()
@@ -142,7 +149,8 @@ func TestManager_ValidateSession_WrongKey(t *testing.T) {
func TestManager_ClearSession(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
w := httptest.NewRecorder()
mgr.ClearSession(w)
@@ -172,7 +180,8 @@ func TestManager_ClearSession(t *testing.T) {
func TestManager_IsAuthenticated(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-signing-key-12345")
keys, _ := seal.DeriveKeys("test-signing-key-12345")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
// No session - should return false
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, "/", nil)
@@ -208,7 +217,8 @@ func TestManager_IsAuthenticated(t *testing.T) {
func TestManager_CookieAttributes(t *testing.T) {
t.Parallel()
mgr, _ := session.NewManager("test-key")
keys, _ := seal.DeriveKeys("test-key")
mgr := session.NewManager(keys.SessionHashKey, keys.SessionBlockKey)
w := httptest.NewRecorder()
_ = mgr.CreateSession(w)
+6 -7
View File
@@ -28,20 +28,20 @@ func TestSigner_Verify_ColonCannotMovePathQueryBoundary(t *testing.T) {
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Now().Add(1 * time.Hour),
}
signed.Signature = signer.Sign(signed)
expires := time.Now().Add(1 * time.Hour)
sig := signer.Sign(signed, expires)
replayed := *signed
replayed.Source.Path = "/a"
replayed.Source.Query = "b:"
t.Logf("signature for path %q query %q: %s",
signed.Source.Path, signed.Source.Query, signed.Signature)
signed.Source.Path, signed.Source.Query, sig)
t.Logf("signature for path %q query %q: %s",
replayed.Source.Path, replayed.Source.Query, signer.Sign(&replayed))
replayed.Source.Path, replayed.Source.Query, signer.Sign(&replayed, expires))
err := signer.Verify(&replayed)
err := signer.Verify(&replayed, sig, expires)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() of path %q query %q with the signature for "+
"path %q = %v, want %v",
@@ -106,10 +106,9 @@ func TestSigner_EscapedGoldenVectors(t *testing.T) {
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Unix(goldenExpiresUnix, 0),
}
gotSignature := signer.Sign(req)
gotSignature := signer.Sign(req, time.Unix(goldenExpiresUnix, 0))
t.Logf("Sign() of path %q query %q = %s", tt.path, tt.query, gotSignature)
if gotSignature != tt.wantSignature {
+3 -4
View File
@@ -19,8 +19,8 @@ type goldenVector struct {
name string
req imagerequest.Request
// wantSignature is the exact base64url (RFC 4648 URL-safe,
// padded) HMAC-SHA256 signature for the request with Expires
// set to goldenExpiresUnix.
// padded) HMAC-SHA256 signature for the request expiring at
// goldenExpiresUnix.
wantSignature string
}
@@ -124,9 +124,8 @@ func TestSigner_GoldenVectors(t *testing.T) {
t.Parallel()
signReq := tt.req
signReq.Expires = time.Unix(goldenExpiresUnix, 0)
gotSignature := signer.Sign(&signReq)
gotSignature := signer.Sign(&signReq, time.Unix(goldenExpiresUnix, 0))
if gotSignature != tt.wantSignature {
t.Errorf("Sign() = %q, want %q (signed byte format changed?)",
gotSignature, tt.wantSignature)
+10 -9
View File
@@ -9,10 +9,12 @@ import (
"sneak.berlin/go/pixa/internal/signature"
)
// signedQualityFitRequest returns a request signed for quality 85 and fit
// mode "cover", the effective defaults the handler applies before
// verification.
func signedQualityFitRequest(signer *signature.Signer) *imagerequest.Request {
// signedQualityFitRequest returns a request, its signature for quality 85 and
// fit mode "cover", the effective defaults the handler applies before
// verification, and the expiry it was signed with.
func signedQualityFitRequest(
signer *signature.Signer,
) (*imagerequest.Request, string, time.Time) {
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
@@ -24,11 +26,10 @@ func signedQualityFitRequest(signer *signature.Signer) *imagerequest.Request {
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
expires := time.Now().Add(1 * time.Hour)
return req
return req, signer.Sign(req, expires), expires
}
// TestSigner_Verify_QualityAndFitAreSigned proves that quality and fit are
@@ -60,10 +61,10 @@ func TestSigner_Verify_QualityAndFitAreSigned(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := signedQualityFitRequest(signer)
req, sig, expires := signedQualityFitRequest(signer)
tt.tamper(req)
err := signer.Verify(req)
err := signer.Verify(req, sig, expires)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() = %v, want %v", err, signature.ErrInvalid)
}
+20 -15
View File
@@ -34,14 +34,14 @@ func New(secretKey string) *Signer {
}
}
// Sign generates an HMAC-SHA256 signature for the given request.
// The signature covers: host + path + query + width + height + format +
// expiration + quality + fit. The quality and fit mode are the effective
// values the request resolves to: callers pass the defaults when the request
// omits those parameters, so an omitted quality or fit mode signs
// Sign generates an HMAC-SHA256 signature for the given request, expiring at
// expires. The signature covers: host + path + query + width + height +
// format + expiration + quality + fit. The quality and fit mode are the
// effective values the request resolves to: callers pass the defaults when
// the request omits those parameters, so an omitted quality or fit mode signs
// identically to that same value stated explicitly.
func (s *Signer) Sign(req *imagerequest.Request) string {
data := s.buildSignatureData(req)
func (s *Signer) Sign(req *imagerequest.Request, expires time.Time) string {
data := s.buildSignatureData(req, expires)
mac := hmac.New(sha256.New, s.secretKey)
mac.Write([]byte(data))
sig := mac.Sum(nil)
@@ -49,28 +49,31 @@ func (s *Signer) Sign(req *imagerequest.Request) string {
return base64.URLEncoding.EncodeToString(sig)
}
// Verify checks if the signature on the request is valid and not expired.
// Verify checks that sig is the signature of the request expiring at expires,
// and that expires has not passed.
// Signatures are exact-match only: every component of the signed data
// (host, path, query, dimensions, format, expiration, quality, fit) must
// match exactly.
// No suffix matching, wildcard matching, or partial matching is supported.
// A signature for "cdn.example.com" will NOT verify for "example.com" or
// "other.cdn.example.com", and vice versa.
func (s *Signer) Verify(req *imagerequest.Request) error {
func (s *Signer) Verify(
req *imagerequest.Request, sig string, expires time.Time,
) error {
// Check expiration first
if req.Expires.IsZero() {
if expires.IsZero() {
return ErrMissingExpiration
}
if time.Now().After(req.Expires) {
if time.Now().After(expires) {
return ErrExpired
}
// Compute expected signature
expected := s.Sign(req)
expected := s.Sign(req, expires)
// Constant-time comparison to prevent timing attacks
if !hmac.Equal([]byte(req.Signature), []byte(expected)) {
if !hmac.Equal([]byte(sig), []byte(expected)) {
return ErrInvalid
}
@@ -85,7 +88,9 @@ func (s *Signer) Verify(req *imagerequest.Request) error {
// expansion is performed. Quality and fit are the effective transform
// values, so replaying a signed URL with a different quality or fit mode
// fails verification.
func (s *Signer) buildSignatureData(req *imagerequest.Request) string {
func (s *Signer) buildSignatureData(
req *imagerequest.Request, expires time.Time,
) string {
return fmt.Sprintf("%s:%s:%s:%d:%d:%s:%d:%d:%s",
req.Source.Host,
escapePercentAndColon(req.Source.Path),
@@ -93,7 +98,7 @@ func (s *Signer) buildSignatureData(req *imagerequest.Request) string {
req.Transform.Size.Width,
req.Transform.Size.Height,
req.Transform.Format,
req.Expires.Unix(),
expires.Unix(),
req.Transform.Quality,
req.Transform.FitMode,
)
+45 -53
View File
@@ -34,11 +34,11 @@ func TestSigner_Sign(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0), // Fixed timestamp for reproducibility
}
expires := time.Unix(1704067200, 0) // Fixed timestamp for reproducibility
sig1 := signer.Sign(req)
sig2 := signer.Sign(req)
sig1 := signer.Sign(req, expires)
sig2 := signer.Sign(req, expires)
// Same input should produce same signature
if sig1 != sig2 {
@@ -62,18 +62,17 @@ func TestSigner_Sign(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0),
}
sig3 := signer.Sign(req2)
sig3 := signer.Sign(req2, expires)
if sig1 == sig3 {
t.Error("Sign() produced same signature for different input")
}
}
// validVerifyRequest returns a fully-populated request that verifies
// successfully once signed.
func validVerifyRequest() *imagerequest.Request {
// validVerifyRequest returns a fully-populated request, and an expiry an hour
// ahead, that verify successfully once signed.
func validVerifyRequest() (*imagerequest.Request, time.Time) {
return &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
@@ -83,13 +82,13 @@ func validVerifyRequest() *imagerequest.Request {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
}, time.Now().Add(1 * time.Hour)
}
type verifyCase struct {
name string
setup func() *imagerequest.Request
name string
// setup returns the request, the signature and the expiry to verify
setup func() (*imagerequest.Request, string, time.Time)
wantErr error
}
@@ -97,54 +96,49 @@ func verifyCases(signer *signature.Signer) []verifyCase {
return []verifyCase{
{
name: "valid signature",
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = signer.Sign(req)
setup: func() (*imagerequest.Request, string, time.Time) {
req, expires := validVerifyRequest()
return req
return req, signer.Sign(req, expires), expires
},
wantErr: nil,
},
{
name: "expired signature",
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Expires = time.Now().Add(-1 * time.Hour)
req.Signature = signer.Sign(req)
setup: func() (*imagerequest.Request, string, time.Time) {
req, _ := validVerifyRequest()
expires := time.Now().Add(-1 * time.Hour)
return req
return req, signer.Sign(req, expires), expires
},
wantErr: signature.ErrExpired,
},
{
name: "invalid signature",
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = "invalid-signature"
setup: func() (*imagerequest.Request, string, time.Time) {
req, expires := validVerifyRequest()
return req
return req, "invalid-signature", expires
},
wantErr: signature.ErrInvalid,
},
{
name: "missing expiration",
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Expires = time.Time{}
req.Signature = "some-signature"
setup: func() (*imagerequest.Request, string, time.Time) {
req, _ := validVerifyRequest()
return req
return req, "some-signature", time.Time{}
},
wantErr: signature.ErrMissingExpiration,
},
{
name: "tampered request",
setup: func() *imagerequest.Request {
req := validVerifyRequest()
req.Signature = signer.Sign(req)
setup: func() (*imagerequest.Request, string, time.Time) {
req, expires := validVerifyRequest()
sig := signer.Sign(req, expires)
req.Source.Path = "/photos/secret.jpg"
return req
return req, sig, expires
},
wantErr: signature.ErrInvalid,
},
@@ -160,8 +154,8 @@ func TestSigner_Verify(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := tt.setup()
err := signer.Verify(req)
req, sig, expires := tt.setup()
err := signer.Verify(req, sig, expires)
if tt.wantErr == nil {
if err != nil {
@@ -254,8 +248,9 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
signer := signature.New("test-secret-key")
// Base request that we'll sign, then tamper with individual fields.
baseReq := func() *imagerequest.Request {
// Base request that we'll sign, then tamper with individual fields; it
// is returned with its signature and expiry.
baseReq := func() (*imagerequest.Request, string, time.Time) {
req := &imagerequest.Request{
Source: imagerequest.Source{
Host: testHost,
@@ -266,21 +261,20 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
req.Signature = signer.Sign(req)
expires := time.Now().Add(1 * time.Hour)
return req
return req, signer.Sign(req, expires), expires
}
for _, tt := range exactMatchTamperCases() {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
req := baseReq()
req, sig, expires := baseReq()
tt.tamper(req)
err := signer.Verify(req)
err := signer.Verify(req, sig, expires)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() = %v, want %v", err, signature.ErrInvalid)
}
@@ -291,9 +285,9 @@ func TestSigner_Verify_ExactMatchOnly(t *testing.T) {
t.Run("unmodified request passes", func(t *testing.T) {
t.Parallel()
req := baseReq()
req, sig, expires := baseReq()
err := signer.Verify(req)
err := signer.Verify(req, sig, expires)
if err != nil {
t.Errorf("Verify() unmodified request failed: %v", err)
}
@@ -329,10 +323,9 @@ func TestSigner_Sign_ExactHostInData(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Unix(1704067200, 0),
}
sig := signer.Sign(req)
sig := signer.Sign(req, time.Unix(1704067200, 0))
if existing, ok := sigs[sig]; ok {
t.Errorf("hosts %q and %q produced the same signature", existing, host)
}
@@ -356,20 +349,20 @@ func TestSigner_DifferentKeys(t *testing.T) {
Size: imagerequest.Size{Width: 800, Height: 600},
Format: testFormatWebP,
},
Expires: time.Now().Add(1 * time.Hour),
}
expires := time.Now().Add(1 * time.Hour)
// Sign with key 1
req.Signature = signer1.Sign(req)
sig := signer1.Sign(req, expires)
// Verify with key 1 should succeed
err := signer1.Verify(req)
err := signer1.Verify(req, sig, expires)
if err != nil {
t.Errorf("Verify() with same key failed: %v", err)
}
// Verify with key 2 should fail
err = signer2.Verify(req)
err = signer2.Verify(req, sig, expires)
if !errors.Is(err, signature.ErrInvalid) {
t.Errorf("Verify() with different key should fail, got: %v", err)
}
@@ -395,12 +388,11 @@ func TestSign_ReadmeQueryExample(t *testing.T) {
Quality: 85,
FitMode: testFitCover,
},
Expires: time.Unix(1704067200, 0),
}
const wantSig = "37aFa9GSVCZM1kRY9SzMOk2fOg_M7Wt5gKNR4xM-FYA="
gotSig := signer.Sign(&req)
gotSig := signer.Sign(&req, time.Unix(1704067200, 0))
if gotSig != wantSig {
t.Errorf("Sign() = %q, want %q", gotSig, wantSig)
}