Limit each client's requests to the image routes (closes #236)
check / check (push) Successful in 54m15s

The new setting image_requests_per_minute (PIXA_IMAGE_REQUESTS_PER_MINUTE),
default 600, caps the GET and HEAD requests one client sends to /v1/image/
and /v1/e/ per minute; the next is answered 429 with Retry-After and the
image routes' JSON error body. It uses the login limit's RateLimit
middleware, so clients are told apart the same way, behind the CORS
middleware, so a preflight is not counted. 0 turns it off, as
script/loadtest does; a negative or non-integer value stops startup.

TestEnvironmentSetsEveryKey gains the new variable, under the standing
approval in #64. To stay within the
linters' limits, the sentry_dsn check moves unchanged into
validateSentryDSN, and the cache_max_bytes explicit flag is set in one
line.

Model: opus-5-5
This commit was merged in pull request #237.
This commit is contained in:
2026-10-13 01:36:18 +02:00
parent bdde021b45
commit 40ad39843b
11 changed files with 431 additions and 35 deletions
+34 -17
View File
@@ -46,9 +46,9 @@ everything in this list without further settings. The reverse proxy must:
builds encrypted URLs from `Host`, and checks `Referer` against builds encrypted URLs from `Host`, and checks `Referer` against
`referer_blocklist`; `referer_blocklist`;
- set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to - set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to
the address pixa sees the proxy's requests come from, so the login limit the address pixa sees the proxy's requests come from, so the login limit and
counts each client by its own address (see `trusted_proxies` under `image_requests_per_minute` count each client by its own address (see
Configuration); `trusted_proxies` under Configuration);
- wait for pixa's answer for at least `downstream_timeout` (default `60s`), the - wait for pixa's answer for at least `downstream_timeout` (default `60s`), the
longest pixa takes to fetch, convert and send an image. longest pixa takes to fetch, convert and send an image.
@@ -58,8 +58,9 @@ password there.
pixa does the rest itself: it checks signatures and encrypted URLs, applies the pixa does the rest itself: it checks signatures and encrypted URLs, applies the
allowlist, refuses upstream hosts with private or local addresses, limits login allowlist, refuses upstream hosts with private or local addresses, limits login
attempts, upstream response size and image dimensions, and sends the security attempts, each client's image requests, upstream response size and image
headers, `Strict-Transport-Security` included, with every response. dimensions, and sends the security headers, `Strict-Transport-Security`
included, with every response.
The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the The state directory (`state_dir`, `/var/lib/pixa` in the container) holds the
database and the disk cache: database and the disk cache:
@@ -186,10 +187,10 @@ path under `/v1/` answers 200, in maintenance mode too.
`Referer` names a host in `referer_blocklist`, checked before the signature, `Referer` names a host in `referer_blocklist`, checked before the signature,
the cache and the upstream fetch; 403 when the upstream host, or a host it the cache and the upstream fetch; 403 when the upstream host, or a host it
redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an redirects to, is `localhost`, ends in `.localhost` or `.local`, or has an
address in a blocked network (see `blocked_networks`); 502 when the upstream address in a blocked network (see `blocked_networks`); 429 over
answered with an error status, and for 5 minutes after that for the same `image_requests_per_minute` (below); 502 when the upstream answered with an
source URL; 503 when pixa is busy or in maintenance mode; 500 for any other error status, and for 5 minutes after that for the same source URL; 503 when
failure. pixa is busy or in maintenance mode; 500 for any other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL - `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when (see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
`If-None-Match` matches the image's `ETag`; 400 for a token that does not `If-None-Match` matches the image's `ETag`; 400 for a token that does not
@@ -197,7 +198,7 @@ path under `/v1/` answers 200, in maintenance mode too.
expired; 504 when the upstream has not sent its response headers within expired; 504 when the upstream has not sent its response headers within
`upstream_fetch_timeout`, but 500 when that time runs out while the image `upstream_fetch_timeout`, but 500 when that time runs out while the image
itself is still arriving; 400 for an `Accept` header that is not valid, and itself is still arriving; 400 for an `Accept` header that is not valid, and
406, 403, 502, 503 and 500, as for `/v1/image/`. 406, 403, 429, 502, 503 and 500, as for `/v1/image/`.
- `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs: - `GET /robots.txt` — asks every crawler to stay away (`Disallow: /`). Needs:
nothing. Answers: 200. nothing. Answers: 200.
- `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`, - `GET /.well-known/healthcheck.json` — JSON with `status` (`ok`), `now`,
@@ -313,6 +314,15 @@ its own `X-Forwarded-For`, whether it connects directly or through the proxy,
because its own address is trusted too. Setting `trusted_proxies` to only the because its own address is trusted too. Setting `trusted_proxies` to only the
address pixa sees for requests that come through the proxy closes this. address pixa sees for requests that come through the proxy closes this.
One client may send `image_requests_per_minute` requests per minute
(default 600) to the image routes, `/v1/image/` and `/v1/e/` together, `GET` and
`HEAD`, cache hits included; a request over the limit is refused with 429 and a
`Retry-After` header. Clients are counted as for the login limit. A browser's
CORS preflight is not counted. Behind a CDN or proxy whose addresses are not in
`trusted_proxies`, all requests arriving from one proxy or CDN address count as
one client, so such a deployment needs a higher limit, or `0`, which turns the
limit off.
### Encrypted URLs ### Encrypted URLs
An encrypted URL is an image URL made on pixa's own web page by someone who An encrypted URL is an image URL made on pixa's own web page by someone who
@@ -482,6 +492,7 @@ tell. With no file, pixa uses the environment and the defaults.
| `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` | | `PIXA_UPSTREAM_CONNECTIONS_PER_HOST` | `upstream_connections_per_host` | Concurrent connections per upstream host; default `20` |
| `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` | | `PIXA_UPSTREAM_CONNECTIONS` | `upstream_connections` | Concurrent connections to all upstream hosts together; default `64` |
| `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs | | `PIXA_MAX_CONCURRENT_PROCESSING` | `max_concurrent_processing` | Images processed at once; default the number of CPUs |
| `PIXA_IMAGE_REQUESTS_PER_MINUTE` | `image_requests_per_minute` | Image requests per client per minute; `0` turns it off; default `600` |
| `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` | | `PIXA_UPSTREAM_FETCH_TIMEOUT` | `upstream_fetch_timeout` | Time allowed for one fetch from an upstream host; default `30s` |
| `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB | | `PIXA_UPSTREAM_MAX_RESPONSE_SIZE` | `upstream_max_response_size` | Largest upstream response accepted, in bytes; default 50 MiB |
| `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` | | `PIXA_DOWNSTREAM_TIMEOUT` | `downstream_timeout` | Time allowed for answering one client request; default `60s` |
@@ -531,13 +542,13 @@ Key settings in more detail:
explicit list replaces the default. An invalid CIDR aborts startup. Set this explicit list replaces the default. An invalid CIDR aborts startup. Set this
to the address pixa sees for requests that come through your proxy, such as to the address pixa sees for requests that come through your proxy, such as
`172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else `172.17.0.1/32`, when the defaults do not cover it, or to trust nothing else
(see the login limit under Routes). For a proxy on the Docker host that (see the login limit and `image_requests_per_minute` under Routes). For a
connects to pixa over `127.0.0.1`, that address is the gateway of the proxy on the Docker host that connects to pixa over `127.0.0.1`, that address
container's Docker network (`172.17.0.1` on the default bridge), not the is the gateway of the container's Docker network (`172.17.0.1` on the default
proxy's own address; a proxy that connects through another of the host's bridge), not the proxy's own address; a proxy that connects through another of
addresses is seen with that address. To be sure which address it is, set this the host's addresses is seen with that address. To be sure which address it
to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request through the is, set this to `[]` (or `PIXA_TRUSTED_PROXIES` to empty), send a request
proxy, and read `remoteIP` in pixa's request log line for it through the proxy, and read `remoteIP` in pixa's request log line for it
- `upstream_fetch_timeout` — time allowed for one fetch from an upstream host, - `upstream_fetch_timeout` — time allowed for one fetch from an upstream host,
as a duration such as `30s` (the default) or `2m` as a duration such as `30s` (the default) or `2m`
- `upstream_max_response_size` — largest upstream response accepted, in bytes; - `upstream_max_response_size` — largest upstream response accepted, in bytes;
@@ -570,6 +581,12 @@ Key settings in more detail:
container's CPU limit. A request that finds all of them in use waits up to 10 container's CPU limit. A request that finds all of them in use waits up to 10
seconds for one to free up; if none does, and `downstream_timeout` has not seconds for one to free up; if none does, and `downstream_timeout` has not
ended first, it is answered 503 the same way ended first, it is answered 503 the same way
- `image_requests_per_minute` — how many requests one client may send to the
image routes per minute (see the limit on them under Routes); default `600`.
`0` turns the limit off, and a negative or non-integer value aborts startup.
Behind a CDN or proxy whose addresses are not in `trusted_proxies`, all
requests arriving from one proxy or CDN address count as one client, so raise
the limit there, or set `0`
- `maintenance_mode` — while `true`, the image routes (`/v1/image/` and - `maintenance_mode` — while `true`, the image routes (`/v1/image/` and
`/v1/e/`) answer every request for an image with 503, a `Retry-After` header `/v1/e/`) answer every request for an image with 503, a `Retry-After` header
and a JSON error body. The health check (`/.well-known/healthcheck.json`) and a JSON error body. The health check (`/.well-known/healthcheck.json`)
+9 -3
View File
@@ -26,10 +26,18 @@ resolved. The disk cache is now size-bounded with LRU eviction
# Next Step # Next Step
P2: security: per-IP rate limiting on the image routes P2: security: per-origin rate limiting
# Completed Steps # Completed Steps
- 2026-10-12 per-client limit on the image routes (closes #236): the new setting
`image_requests_per_minute` (`PIXA_IMAGE_REQUESTS_PER_MINUTE`), default 600,
is how many `GET` and `HEAD` requests one client may send to `/v1/image/` and
`/v1/e/` together per minute, cache hits included; the next is answered 429
with `Retry-After`. It uses the login limit's `RateLimit` middleware, so
clients are told apart the same way, and runs after the CORS middleware, so a
preflight is not counted. `0` turns the limit off; a negative or non-integer
value aborts startup.
- 2026-10-08 every output format is saved with settings pixa sets on purpose - 2026-10-08 every output format is saved with settings pixa sets on purpose
(closes #232): each format has its own govips export, as JPEG XL does, in (closes #232): each format has its own govips export, as JPEG XL does, in
place of govips' generic `Export`, which sent libvips a zero for some settings place of govips' generic `Export`, which sent libvips a zero for some settings
@@ -736,8 +744,6 @@ P2: security: per-IP rate limiting on the image routes
# Future Steps # Future Steps
- P2: security
- per-origin rate limiting
- P2: HTTP response handling - P2: HTTP response handling
- Last-Modified headers - Last-Modified headers
- P2: configuration - P2: configuration
+8
View File
@@ -115,6 +115,14 @@ upstream_connections: 64
# frees up it is answered 503, unless downstream_timeout has ended first. # frees up it is answered 503, unless downstream_timeout has ended first.
# max_concurrent_processing: 4 # max_concurrent_processing: 4
# How many requests to the image routes (/v1/image/ and /v1/e/) one client
# may send per minute; a request over the limit is answered 429 with a
# Retry-After header. 0 turns the limit off. Clients are told apart as for
# the login limit, so behind a CDN or proxy whose addresses are not in
# trusted_proxies all requests arriving from one proxy or CDN address count
# as one client: raise the limit there, or set 0. (default: 600)
image_requests_per_minute: 600
# Time allowed for one fetch from an upstream host (default: 30s) # Time allowed for one fetch from an upstream host (default: 30s)
upstream_fetch_timeout: 30s upstream_fetch_timeout: 30s
+39 -10
View File
@@ -34,6 +34,7 @@ const (
DefaultUpstreamFetchTimeout = 30 * time.Second DefaultUpstreamFetchTimeout = 30 * time.Second
DefaultUpstreamMaxResponseSize = 50 << 20 // 50 MiB DefaultUpstreamMaxResponseSize = 50 << 20 // 50 MiB
DefaultDownstreamTimeout = 60 * time.Second DefaultDownstreamTimeout = 60 * time.Second
DefaultImageRequestsPerMinute = 600
) )
// Configuration key names. // Configuration key names.
@@ -61,6 +62,7 @@ const (
keyUpstreamFetchTimeout = "upstream_fetch_timeout" keyUpstreamFetchTimeout = "upstream_fetch_timeout"
keyUpstreamMaxResponseSize = "upstream_max_response_size" keyUpstreamMaxResponseSize = "upstream_max_response_size"
keyDownstreamTimeout = "downstream_timeout" keyDownstreamTimeout = "downstream_timeout"
keyImageRequestsPerMinute = "image_requests_per_minute"
) )
// placeholderSigningKey is the dummy signing_key shipped in // placeholderSigningKey is the dummy signing_key shipped in
@@ -158,6 +160,11 @@ type Config struct {
// of the per-request timeout middleware. // of the per-request timeout middleware.
DownstreamTimeout time.Duration DownstreamTimeout time.Duration
// ImageRequestsPerMinute is how many requests one client may make to
// the image routes per minute; the next is refused with 429 Too Many
// Requests. 0 turns the limit off.
ImageRequestsPerMinute int
// BlockedNetworks are operator-supplied CIDR ranges to refuse in // BlockedNetworks are operator-supplied CIDR ranges to refuse in
// addition to the built-in SSRF blocklist. Enforced by the upstream // addition to the built-in SSRF blocklist. Enforced by the upstream
// fetcher's dialer; the built-in ranges always apply. // fetcher's dialer; the built-in ranges always apply.
@@ -302,6 +309,8 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
keyAccessControlAllowOrigin, DefaultAccessControlAllowOrigin), keyAccessControlAllowOrigin, DefaultAccessControlAllowOrigin),
DownstreamTimeout: loader.durationVal( DownstreamTimeout: loader.durationVal(
keyDownstreamTimeout, DefaultDownstreamTimeout), keyDownstreamTimeout, DefaultDownstreamTimeout),
ImageRequestsPerMinute: loader.intVal(
keyImageRequestsPerMinute, DefaultImageRequestsPerMinute),
CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0), CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0),
BlockedNetworks: blockedNetworks, BlockedNetworks: blockedNetworks,
TrustedProxies: trustedProxies, TrustedProxies: trustedProxies,
@@ -311,9 +320,7 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
// The default for an omitted cache_max_bytes is worked out when // The default for an omitted cache_max_bytes is worked out when
// the cache opens; here we only record whether the operator set // the cache opens; here we only record whether the operator set
// the key explicitly. // the key explicitly.
if _, present := lookupValue(sc, keyCacheMaxBytes); present { _, c.CacheMaxBytesExplicit = lookupValue(sc, keyCacheMaxBytes)
c.CacheMaxBytesExplicit = true
}
// Build DBURL from StateDir if not explicitly set. The derived URL // Build DBURL from StateDir if not explicitly set. The derived URL
// is a default: it applies only when db_url is omitted, never to an // is a default: it applies only when db_url is omitted, never to an
@@ -426,7 +433,7 @@ func isKnownConfigKey(key string) bool {
keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks, keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks,
keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout, keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout,
keyUpstreamMaxResponseSize, keyDownstreamTimeout, keyRefererBlocklist, keyUpstreamMaxResponseSize, keyDownstreamTimeout, keyRefererBlocklist,
"env": keyImageRequestsPerMinute, "env":
return true return true
} }
@@ -461,6 +468,7 @@ func envVarNames() map[string]string {
keyUpstreamFetchTimeout: "PIXA_UPSTREAM_FETCH_TIMEOUT", keyUpstreamFetchTimeout: "PIXA_UPSTREAM_FETCH_TIMEOUT",
keyUpstreamMaxResponseSize: "PIXA_UPSTREAM_MAX_RESPONSE_SIZE", keyUpstreamMaxResponseSize: "PIXA_UPSTREAM_MAX_RESPONSE_SIZE",
keyDownstreamTimeout: "PIXA_DOWNSTREAM_TIMEOUT", keyDownstreamTimeout: "PIXA_DOWNSTREAM_TIMEOUT",
keyImageRequestsPerMinute: "PIXA_IMAGE_REQUESTS_PER_MINUTE",
} }
} }
@@ -613,6 +621,14 @@ func (c *Config) validate() error {
settingName(keyCacheMaxBytes), c.CacheMaxBytes, errMustNotBeNegative) settingName(keyCacheMaxBytes), c.CacheMaxBytes, errMustNotBeNegative)
} }
// Zero is valid (it turns the limit off); only negative values are
// rejected.
if c.ImageRequestsPerMinute < 0 {
return fmt.Errorf("%s: value %d %w",
settingName(keyImageRequestsPerMinute), c.ImageRequestsPerMinute,
errMustNotBeNegative)
}
err = c.validateUpstreamMaxResponseSize() err = c.validateUpstreamMaxResponseSize()
if err != nil { if err != nil {
return err return err
@@ -625,12 +641,9 @@ func (c *Config) validate() error {
} }
} }
if c.SentryDSN != "" { err = c.validateSentryDSN()
parsed, err := url.Parse(c.SentryDSN) if err != nil {
if err != nil || parsed.Scheme == "" || parsed.Host == "" { return err
return fmt.Errorf("%s: value %q is %w",
settingName(keySentryDSN), c.SentryDSN, errNotAValidURL)
}
} }
if (c.MetricsUsername == "") != (c.MetricsPassword == "") { if (c.MetricsUsername == "") != (c.MetricsPassword == "") {
@@ -642,6 +655,22 @@ func (c *Config) validate() error {
return c.validateAccessControlAllowOrigin() return c.validateAccessControlAllowOrigin()
} }
// validateSentryDSN checks that sentry_dsn, when set, is a URL with a scheme
// and a host.
func (c *Config) validateSentryDSN() error {
if c.SentryDSN == "" {
return nil
}
parsed, err := url.Parse(c.SentryDSN)
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
return fmt.Errorf("%s: value %q is %w",
settingName(keySentryDSN), c.SentryDSN, errNotAValidURL)
}
return nil
}
// validateUpstreamMaxResponseSize checks that upstream_max_response_size // validateUpstreamMaxResponseSize checks that upstream_max_response_size
// is from 1 byte to 1 GiB. An upstream response is read whole into // is from 1 byte to 1 GiB. An upstream response is read whole into
// memory, and the image processor reads one byte past this limit, which // memory, and the image processor reads one byte past this limit, which
+2
View File
@@ -70,6 +70,7 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5") t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5")
t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10") t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10")
t.Setenv("PIXA_MAX_CONCURRENT_PROCESSING", "3") t.Setenv("PIXA_MAX_CONCURRENT_PROCESSING", "3")
t.Setenv("PIXA_IMAGE_REQUESTS_PER_MINUTE", "30")
t.Setenv("PIXA_CACHE_MAX_BYTES", "1024") t.Setenv("PIXA_CACHE_MAX_BYTES", "1024")
t.Setenv("PIXA_BLOCKED_NETWORKS", "203.0.113.0/24") t.Setenv("PIXA_BLOCKED_NETWORKS", "203.0.113.0/24")
t.Setenv("PIXA_TRUSTED_PROXIES", "192.0.2.0/24") t.Setenv("PIXA_TRUSTED_PROXIES", "192.0.2.0/24")
@@ -99,6 +100,7 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
UpstreamConnectionsPerHost: 5, UpstreamConnectionsPerHost: 5,
UpstreamConnections: 10, UpstreamConnections: 10,
MaxConcurrentProcessing: 3, MaxConcurrentProcessing: 3,
ImageRequestsPerMinute: 30,
CacheMaxBytes: 1024, CacheMaxBytes: 1024,
CacheMaxBytesExplicit: true, CacheMaxBytesExplicit: true,
BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")}, BlockedNetworks: []netip.Prefix{netip.MustParsePrefix("203.0.113.0/24")},
@@ -0,0 +1,107 @@
package config
import "testing"
// testImageRequestsPerMinuteVar is the variable that sets
// image_requests_per_minute.
const testImageRequestsPerMinuteVar = "PIXA_IMAGE_REQUESTS_PER_MINUTE"
// TestImageRequestsPerMinuteValues checks that an omitted
// image_requests_per_minute is 600, and that 0, which turns the limit off,
// and any larger whole number are used as given.
func TestImageRequestsPerMinuteValues(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
if c.ImageRequestsPerMinute != 600 {
t.Errorf("ImageRequestsPerMinute = %d, want 600", c.ImageRequestsPerMinute)
}
explicit := []struct {
value string
want int
}{
{"0", 0},
{"30", 30},
}
for _, tc := range explicit {
c, err := configFromYAML(t,
signingKeyLine+"image_requests_per_minute: "+tc.value+"\n")
if err != nil {
t.Fatalf("image_requests_per_minute: %s should load, got error: %v",
tc.value, err)
}
if c.ImageRequestsPerMinute != tc.want {
t.Errorf("ImageRequestsPerMinute = %d, want %d",
c.ImageRequestsPerMinute, tc.want)
}
}
}
// TestInvalidImageRequestsPerMinuteAbortsStartup checks that a value that is
// negative, not a whole number, or null aborts startup naming the key and
// the value, and the variable too where the value could have come from it.
func TestInvalidImageRequestsPerMinuteAbortsStartup(t *testing.T) {
t.Parallel()
key := keyImageRequestsPerMinute
runAbortCases(t, []abortCase{
{
name: "negative",
yaml: signingKeyLine + key + ": -1\n",
wantErrSubstrings: []string{
key, testImageRequestsPerMinuteVar, "value -1",
},
},
{
name: "not a number",
yaml: signingKeyLine + key + ": plenty\n",
wantErrSubstrings: []string{
key, testImageRequestsPerMinuteVar, "plenty",
},
},
{
name: "fractional",
yaml: signingKeyLine + key + ": 2.5\n",
wantErrSubstrings: []string{key, "2.5"},
},
{
name: "explicit null",
yaml: signingKeyLine + key + ": null\n",
wantErrSubstrings: []string{key, nullValueText},
},
})
}
// TestImageRequestsPerMinuteFromEnvironment checks that the variable sets the
// limit over the config file, and that an invalid value in it aborts startup
// naming the variable and the value.
func TestImageRequestsPerMinuteFromEnvironment(t *testing.T) {
t.Setenv(testImageRequestsPerMinuteVar, "0")
c, err := configFromYAML(t, signingKeyLine+"image_requests_per_minute: 30\n")
if err != nil {
t.Fatalf("limit from the environment should load: %v", err)
}
if c.ImageRequestsPerMinute != 0 {
t.Errorf("ImageRequestsPerMinute = %d, want 0 from the environment",
c.ImageRequestsPerMinute)
}
for _, value := range []string{"-5", "ten"} {
t.Run(value, func(t *testing.T) {
t.Setenv(testImageRequestsPerMinuteVar, value)
_, err := configFromYAML(t, signingKeyLine)
wantStartupError(t, err, testImageRequestsPerMinuteVar, value)
})
}
}
+10
View File
@@ -206,6 +206,16 @@ func parseExpires(query url.Values) (time.Time, error) {
return time.Unix(exp, 0), nil return time.Unix(exp, 0), nil
} }
// HandleTooManyRequests answers 429 Too Many Requests with the JSON error
// body the image routes send for their other errors. The image routes' rate
// limit calls it for a request over the limit.
func (s *Handlers) HandleTooManyRequests() http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
s.respondError(w, "too many requests, try again later",
http.StatusTooManyRequests)
}
}
// respondImageError maps image retrieval errors to HTTP responses. // respondImageError maps image retrieval errors to HTTP responses.
func (s *Handlers) respondImageError( func (s *Handlers) respondImageError(
w http.ResponseWriter, req *imgcache.ImageRequest, err error, w http.ResponseWriter, req *imgcache.ImageRequest, err error,
+5 -3
View File
@@ -98,9 +98,10 @@ func (s *Middleware) ClientIP() func(http.Handler) http.Handler {
// holds whole; an IPv4-mapped address (::ffff:a.b.c.d) is counted as the // holds whole; an IPv4-mapped address (::ffff:a.b.c.d) is counted as the
// IPv4 address it carries, since every such address falls in the same /64. // IPv4 address it carries, since every such address falls in the same /64.
// Counts are kept only for the current and the previous window, so memory // Counts are kept only for the current and the previous window, so memory
// stays bounded. // stays bounded. limitHandler writes the 429 after Retry-After is set; nil
// keeps the rate limit library's plain-text "Too Many Requests".
func (s *Middleware) RateLimit( func (s *Middleware) RateLimit(
requestLimit int, window time.Duration, requestLimit int, window time.Duration, limitHandler http.HandlerFunc,
) func(http.Handler) http.Handler { ) func(http.Handler) http.Handler {
return httprate.LimitBy(requestLimit, window, return httprate.LimitBy(requestLimit, window,
func(r *http.Request) (string, error) { func(r *http.Request) (string, error) {
@@ -112,7 +113,8 @@ func (s *Middleware) RateLimit(
} }
return httprate.CanonicalizeIP(ip), nil return httprate.CanonicalizeIP(ip), nil
}) },
httprate.WithLimitHandler(limitHandler))
} }
// requestIDPattern is what a request's own X-Request-Id must look like to be // requestIDPattern is what a request's own X-Request-Id must look like to be
@@ -0,0 +1,204 @@
package server
import (
"encoding/json"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"sneak.berlin/go/pixa/internal/config"
)
// requestImageFrom sends method path to s from remoteAddr and returns the
// response. The image handlers refuse an unsigned image URL with 401 and a
// token they cannot decrypt with 400, so either status shows the limit let
// the request through.
func requestImageFrom(
t *testing.T, s *Server, method, path, remoteAddr string,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, path, nil)
req.RemoteAddr = remoteAddr
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
return rec
}
// TestImageRequestLimitRefusesRequestOverLimit verifies that GET and HEAD on
// both image routes count toward one limit per client, and that the request
// after image_requests_per_minute of them is refused with 429 and a
// Retry-After header.
func TestImageRequestLimitRefusesRequestOverLimit(t *testing.T) {
t.Parallel()
requests := []struct {
method string
path string
want int
}{
{http.MethodGet, unsignedImagePath, http.StatusUnauthorized},
{http.MethodHead, unsignedImagePath, http.StatusUnauthorized},
{http.MethodGet, encryptedImagePath, http.StatusBadRequest},
}
s := newTestServer(t)
s.config.ImageRequestsPerMinute = len(requests)
s.SetupRoutes()
for _, tc := range requests {
rec := requestImageFrom(t, s, tc.method, tc.path, firstClient)
if rec.Code != tc.want {
t.Fatalf("%s %s within the limit status = %d, want %d",
tc.method, tc.path, rec.Code, tc.want)
}
}
rec := requestImageFrom(t, s, http.MethodHead, encryptedImagePath, firstClient)
if rec.Code != http.StatusTooManyRequests {
t.Fatalf("request over the limit status = %d, want %d",
rec.Code, http.StatusTooManyRequests)
}
retryAfter := rec.Header().Get("Retry-After")
seconds, err := strconv.Atoi(retryAfter)
if err != nil || seconds <= 0 {
t.Errorf("Retry-After = %q, want a positive number of seconds",
retryAfter)
}
}
// TestImageRequestLimitAnswersWithJSONError verifies that the 429 for a
// request over the limit carries Retry-After and the JSON error body the
// image routes send for their other errors.
func TestImageRequestLimitAnswersWithJSONError(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.ImageRequestsPerMinute = 1
s.SetupRoutes()
requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
rec := requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
t.Logf("status %d, body %s", rec.Code, rec.Body.String())
if rec.Code != http.StatusTooManyRequests {
t.Fatalf("request over the limit status = %d, want %d",
rec.Code, http.StatusTooManyRequests)
}
retryAfter := rec.Header().Get("Retry-After")
seconds, err := strconv.Atoi(retryAfter)
if err != nil || seconds <= 0 {
t.Errorf("Retry-After = %q, want a positive number of seconds",
retryAfter)
}
contentType := rec.Header().Get("Content-Type")
if contentType != "application/json" {
t.Errorf("Content-Type = %q, want %q", contentType, "application/json")
}
var body struct {
Error string `json:"error"`
Status int `json:"status"`
Timestamp string `json:"timestamp"`
}
err = json.NewDecoder(rec.Body).Decode(&body)
if err != nil {
t.Fatalf("body is not JSON: %v", err)
}
if body.Error == "" || body.Status != http.StatusTooManyRequests ||
body.Timestamp == "" {
t.Errorf("body = %+v, want an error, status %d and a timestamp",
body, http.StatusTooManyRequests)
}
}
// TestImageRequestLimitCountsClientsSeparately verifies that one client going
// over the limit does not limit another.
func TestImageRequestLimitCountsClientsSeparately(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.ImageRequestsPerMinute = 1
s.SetupRoutes()
rec := requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("first client's first request status = %d, want %d",
rec.Code, http.StatusUnauthorized)
}
rec = requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
if rec.Code != http.StatusTooManyRequests {
t.Fatalf("first client's second request status = %d, want %d",
rec.Code, http.StatusTooManyRequests)
}
rec = requestImageFrom(t, s, http.MethodGet, unsignedImagePath, secondClient)
if rec.Code != http.StatusUnauthorized {
t.Errorf("second client's first request status = %d, want %d",
rec.Code, http.StatusUnauthorized)
}
}
// TestImageRequestLimitZeroLetsEveryRequestThrough verifies that with
// image_requests_per_minute set to 0 no image request is refused with 429,
// not even past the default limit.
func TestImageRequestLimitZeroLetsEveryRequestThrough(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.ImageRequestsPerMinute = 0
s.SetupRoutes()
for request := range config.DefaultImageRequestsPerMinute + 1 {
rec := requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
if rec.Code != http.StatusUnauthorized {
t.Fatalf("request %d status = %d, want %d",
request+1, rec.Code, http.StatusUnauthorized)
}
}
}
// TestImageRequestLimitDoesNotCountPreflight verifies that a browser's CORS
// preflight request to an image route is answered without being counted, so
// a client that has sent several still gets its image request through.
func TestImageRequestLimitDoesNotCountPreflight(t *testing.T) {
t.Parallel()
s := newTestServer(t)
s.config.AccessControlAllowOrigin = config.DefaultAccessControlAllowOrigin
s.config.ImageRequestsPerMinute = 1
s.SetupRoutes()
for range 3 {
req := httptest.NewRequestWithContext(
t.Context(), http.MethodOptions, unsignedImagePath, nil)
req.RemoteAddr = firstClient
req.Header.Set("Origin", "https://app.example.com")
req.Header.Set("Access-Control-Request-Method", http.MethodGet)
rec := httptest.NewRecorder()
s.ServeHTTP(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("preflight status = %d, want %d", rec.Code, http.StatusOK)
}
}
rec := requestImageFrom(t, s, http.MethodGet, unsignedImagePath, firstClient)
if rec.Code != http.StatusUnauthorized {
t.Errorf("image request after preflights status = %d, want %d",
rec.Code, http.StatusUnauthorized)
}
}
+10 -1
View File
@@ -66,7 +66,7 @@ func (s *Server) SetupRoutes() {
r.Use(s.h.LimitBody(handlers.MaxFormBytes)) r.Use(s.h.LimitBody(handlers.MaxFormBytes))
r.Use(s.h.CSRF()) r.Use(s.h.CSRF())
r.Get("/", s.h.HandleRoot()) r.Get("/", s.h.HandleRoot())
r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute)). r.With(s.mw.RateLimit(LoginAttemptsPerMinute, time.Minute, nil)).
Post("/", s.h.HandleRoot()) Post("/", s.h.HandleRoot())
r.Post("/generate", s.h.HandleGenerateURL()) r.Post("/generate", s.h.HandleGenerateURL())
}) })
@@ -88,6 +88,15 @@ func (s *Server) SetupRoutes() {
r.Group(func(r chi.Router) { r.Group(func(r chi.Router) {
r.Use(s.refuseDuringMaintenance) r.Use(s.refuseDuringMaintenance)
// Limited per client, as the login POST is, but the 429 has
// the image routes' JSON error body; 0 turns the limit off.
// The CORS middleware answers a preflight before it gets
// here, so a preflight is not counted.
if s.config.ImageRequestsPerMinute > 0 {
r.Use(s.mw.RateLimit(s.config.ImageRequestsPerMinute,
time.Minute, s.h.HandleTooManyRequests()))
}
// Main image proxy route // Main image proxy route
// /v1/image/<host>/<path>/<width>x<height>.<format>, or with no // /v1/image/<host>/<path>/<width>x<height>.<format>, or with no
// format /v1/image/<host>/<path>/<width>x<height> // format /v1/image/<host>/<path>/<width>x<height>
+3 -1
View File
@@ -94,7 +94,8 @@ main() {
} }
# start_containers starts a new origin and a new pixad, and waits up to 30 # start_containers starts a new origin and a new pixad, and waits up to 30
# seconds for pixad's health check to pass. # seconds for pixad's health check to pass. pixad sees all of vegeta's
# requests come from one address, so its image request limit is off.
start_containers() { start_containers() {
docker run -d --name "$run-origin" \ docker run -d --name "$run-origin" \
--network "$run" --network-alias origin \ --network "$run" --network-alias origin \
@@ -104,6 +105,7 @@ start_containers() {
--network "$run" -p 127.0.0.1::8080 --health-interval=1s \ --network "$run" -p 127.0.0.1::8080 --health-interval=1s \
-e PIXA_SIGNING_KEY="$(head -c 32 /dev/urandom | base64)" \ -e PIXA_SIGNING_KEY="$(head -c 32 /dev/urandom | base64)" \
-e PIXA_ALLOWLIST_HOSTS=origin -e PIXA_ALLOW_HTTP=true \ -e PIXA_ALLOWLIST_HOSTS=origin -e PIXA_ALLOW_HTTP=true \
-e PIXA_IMAGE_REQUESTS_PER_MINUTE=0 \
"$image" >/dev/null "$image" >/dev/null
waited=0 waited=0