5 Commits
Author SHA1 Message Date
clawbot b3e744a744 Refuse host entries that are not a host name or an IP address
check / check (push) Failing after 2s
An entry of allowlist_hosts or referer_blocklist that is neither a host
name (letters, digits, hyphens and dots, with at most one leading dot)
nor an IP address now aborts startup naming the setting and the entry,
so a `*.` wildcard or a port no longer loads and silently matches
nothing. README.md, config.example.yml and the TODO.md entry now say the
Referer check comes before the signature, the cache and the upstream
fetch, since maintenance mode answers first; config.example.yml says
the list does not cover the login and generator pages.

Model: opus-5-5
2026-10-04 19:04:34 +00:00
clawbot a31dbcb70a Add failing tests for host entries that can never match
check / check (push) Failing after 1s
A `*.` entry, an entry with a port and one with two leading dots must
abort startup for referer_blocklist, and the first two for
allowlist_hosts; IPv4 and IPv6 address entries must still load.

Model: opus-5-5
2026-10-04 19:01:23 +00:00
clawbot 8f66a795ad Read referer_blocklist through strictLoader and fix test lint
newFromSmartConfig went over the line limit, so referer_blocklist is now
read through strictLoader, like the other typed settings, instead of its
own parse step. The tests stop repeating string literals that goconst
counts: the environment test uses other hosts, two case names change, and
the handler test names each image route's URL instead of its path.

Model: opus-5-5
2026-10-04 18:59:29 +00:00
clawbot d9fa7d9130 Refuse image requests whose Referer is on referer_blocklist (closes #90)
A new setting, referer_blocklist (PIXA_REFERER_BLOCKLIST), lists hosts
written and matched as allowlist_hosts are, with the same matcher and the
same entry check; a bad entry aborts startup naming the setting and the
entry. Both image routes check the Referer first and answer 403 with the
JSON error, so a blocked request fetches nothing and is refused whether or
not the image is cached. No Referer, or one that does not parse as a URL
with a host, is served; README.md and config.example.yml say this makes the
list easy to get around. The CIDR-list entry reader is renamed listEntries
now that host lists use it too.

Model: opus-5-5
2026-10-04 18:59:29 +00:00
clawbot 7d2a9a2a96 Add failing tests for referer_blocklist
Both image routes must refuse a request whose Referer names a host on
referer_blocklist with 403 and the JSON error, without fetching from the
upstream host and whether or not the image is cached, and must serve a
request with no Referer, one that does not parse, or one naming another
host. Hosts match as allowlist_hosts matches them. The config tests check
the list is read from the file and from PIXA_REFERER_BLOCKLIST, and that an
entry that is not a host aborts startup naming the setting and the entry.
These do not compile until the setting exists.

Model: opus-5-5
2026-10-04 18:59:29 +00:00
11 changed files with 519 additions and 48 deletions
+27 -6
View File
@@ -45,7 +45,8 @@ everything in this list without further settings. The reverse proxy must:
Routes); Routes);
- pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses - pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses
a form from those pages unless `Origin` or `Referer` names the host in `Host`, a form from those pages unless `Origin` or `Referer` names the host in `Host`,
and builds encrypted URLs from `Host`; builds encrypted URLs from `Host`, and checks `Referer` against
`referer_blocklist`;
- set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to - set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to
the address pixa sees the proxy's requests come from, so the login limit the address pixa sees the proxy's requests come from, so the login limit
counts each client by its own address (see `trusted_proxies` under counts each client by its own address (see `trusted_proxies` under
@@ -175,11 +176,13 @@ path under `/v1/` answers 200, in maintenance mode too.
allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
the image's `ETag`; 400 for a URL or parameter that is not valid; 401 for a the image's `ETag`; 400 for a URL or parameter that is not valid; 401 for a
missing or wrong signature, a missing `exp` or an `exp` in the past; 403 when missing or wrong signature, a missing `exp` or an `exp` in the past; 403 when
the upstream host, or a host it redirects to, is `localhost`, ends in the request's `Referer` names a host in `referer_blocklist`, checked before
`.localhost` or `.local`, or has an address in a blocked network (see the signature, the cache and the upstream fetch; 403 when the upstream host,
`blocked_networks`); 502 when the upstream answered with an error status, and or a host it redirects to, is `localhost`, ends in `.localhost` or `.local`,
for 5 minutes after that for the same source URL; 503 when pixa is busy or in or has an address in a blocked network (see `blocked_networks`); 502 when the
maintenance mode; 500 for any other failure. upstream answered with an error status, and for 5 minutes after that for the
same source URL; 503 when pixa is busy or in maintenance mode; 500 for any
other failure.
- `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL - `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
(see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when (see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
`If-None-Match` matches the image's `ETag`; 400 for a token that does not `If-None-Match` matches the image's `ETag`; 400 for a token that does not
@@ -392,6 +395,11 @@ and the URL is
- **Suffix match**: `.example.com` — matches `cdn.example.com`, - **Suffix match**: `.example.com` — matches `cdn.example.com`,
`images.example.com`, and `example.com` `images.example.com`, and `example.com`
An IP address is matched exactly; write an IPv6 address without brackets. An
entry that is neither a host name (letters, digits, hyphens and dots, with at
most one leading dot) nor an IP address, such as one with a port or a `*.`
wildcard, aborts startup.
### Configuration ### Configuration
Every setting can be given as an environment variable, in a YAML config Every setting can be given as an environment variable, in a YAML config
@@ -428,6 +436,7 @@ tell. With no file, pixa uses the environment and the defaults.
| `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory | | `PIXA_DB_URL` | `db_url` | SQLite database URL; default `state.sqlite3` in the state directory |
| `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) | | `PIXA_CACHE_MAX_BYTES` | `cache_max_bytes` | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached) |
| `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature | | `PIXA_ALLOWLIST_HOSTS` | `allowlist_hosts` | Upstream hosts served without a signature |
| `PIXA_REFERER_BLOCKLIST` | `referer_blocklist` | Hosts whose pages the image routes refuse with 403, by `Referer` |
| `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones | | `PIXA_BLOCKED_NETWORKS` | `blocked_networks` | CIDR ranges never fetched from, on top of the built-in ones |
| `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 | | `PIXA_TRUSTED_PROXIES` | `trusted_proxies` | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
| `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` | | `PIXA_ALLOW_HTTP` | `allow_http` | Allow plain-HTTP upstreams, for testing only; default `false` |
@@ -456,6 +465,18 @@ Key settings in more detail:
that has no leading zero and is not the scheme's default. Any other value, that has no leading zero and is not the scheme's default. Any other value,
including another scheme such as a browser extension's, aborts startup including another scheme such as a browser extension's, aborts startup
- `allowlist_hosts` — list of allowed upstream hosts - `allowlist_hosts` — list of allowed upstream hosts
- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
stop other sites hotlinking them. Entries are written and matched as for
`allowlist_hosts` (see Allowlist patterns), and an entry that is neither a
host name nor an IP address aborts startup. A request to `/v1/image/` or
`/v1/e/` whose `Referer` header names a listed host is refused with 403 before
its signature or token is checked and before the cache or the upstream host is
used, so it fetches nothing, and it is refused even when the image is cached.
A request with no `Referer`, or one that does not parse as a URL
with a host, is served, as many clients send none. So this is easily got
around: a site whose pages send no `Referer` (for example with
`Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
and generator pages. Default: empty
- `blocked_networks` — list of CIDR ranges to refuse for SSRF protection, - `blocked_networks` — list of CIDR ranges to refuse for SSRF protection,
added to the always-enforced built-in ranges (loopback, private, added to the always-enforced built-in ranges (loopback, private,
link-local, CGNAT, benchmark, NAT64, and the like); an invalid CIDR link-local, CGNAT, benchmark, NAT64, and the like); an invalid CIDR
+13 -2
View File
@@ -27,10 +27,22 @@ The disk cache is now size-bounded with LRU eviction
# Next Step # Next Step
P2: security: referer blacklist P2: security: per-IP rate limiting on the image routes
# Completed Steps # Completed Steps
- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
(`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
`allowlist_hosts` with the same matcher; an entry of either list that is
neither a host name (letters, digits, hyphens and dots, with at most one
leading dot) nor an IP address, such as one with a port or a `*.` wildcard,
aborts startup naming the setting and the entry. Both image routes refuse a
request whose `Referer` names a listed host with 403 and a JSON error before
the signature, the cache and the upstream fetch, so it fetches nothing and is
refused whether or not the image is cached. A request with no `Referer`, or
one that does not parse as a URL with a host, is served, so the list is easily
got around; `README.md` and `config.example.yml` say so. It does not apply to
the login and generator pages.
- 2026-10-04 SQLite writes no longer fail with "database is locked" (closes - 2026-10-04 SQLite writes no longer fail with "database is locked" (closes
#198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write #198): pixa adds `_pragma=busy_timeout(5000)` to every `db_url`, so a write
that finds another in progress on another connection waits up to five seconds that finds another in progress on another connection waits up to five seconds
@@ -551,7 +563,6 @@ P2: security: referer blacklist
# Future Steps # Future Steps
- P2: security - P2: security
- per-IP rate limiting on the image routes
- per-origin rate limiting - per-origin rate limiting
- P2: HTTP response handling - P2: HTTP response handling
- Last-Modified headers - Last-Modified headers
+12
View File
@@ -46,6 +46,8 @@ signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32"
# Hosts that don't require signatures (default: none) # Hosts that don't require signatures (default: none)
# Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com") # Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
# An entry that is neither a host name nor an IP address (IPv6 without
# brackets), such as one with a port or a "*." wildcard, aborts startup.
allowlist_hosts: allowlist_hosts:
- s3.sneak.cloud - s3.sneak.cloud
- static.sneak.cloud - static.sneak.cloud
@@ -53,6 +55,16 @@ allowlist_hosts:
- github.com - github.com
- user-images.githubusercontent.com - user-images.githubusercontent.com
# Hosts whose pages may not show pixa's images, written as for
# allowlist_hosts. A request to /v1/image/ or /v1/e/ whose Referer header
# names one of them is answered 403 before anything is fetched, even when
# the image is cached. A request with no Referer, or one that does not
# parse, is served, so a site whose pages send no Referer is not stopped.
# The login and generator pages are not covered. (default: none)
# referer_blocklist:
# - leech.example
# - .hotlinker.example
# Additional CIDR ranges to refuse when fetching upstream, extending the # Additional CIDR ranges to refuse when fetching upstream, extending the
# SSRF protection. These are added to the always-enforced built-in ranges # SSRF protection. These are added to the always-enforced built-in ranges
# (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and # (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and
+83 -35
View File
@@ -11,6 +11,7 @@ import (
"net/url" "net/url"
"os" "os"
"path/filepath" "path/filepath"
"regexp"
"runtime" "runtime"
"sort" "sort"
"strconv" "strconv"
@@ -48,6 +49,7 @@ const (
keyMetricsPassword = "metrics.password" keyMetricsPassword = "metrics.password"
keySigningKey = "signing_key" keySigningKey = "signing_key"
keyAllowlistHosts = "allowlist_hosts" keyAllowlistHosts = "allowlist_hosts"
keyRefererBlocklist = "referer_blocklist"
keyAllowHTTP = "allow_http" keyAllowHTTP = "allow_http"
keyUpstreamConnectionsPerHost = "upstream_connections_per_host" keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
keyUpstreamConnections = "upstream_connections" keyUpstreamConnections = "upstream_connections"
@@ -97,12 +99,11 @@ var (
"value is null; omit the key entirely to use the default") "value is null; omit the key entirely to use the default")
errValuesNull = errors.New( errValuesNull = errors.New(
"value is null; omit a key entirely to use its default") "value is null; omit a key entirely to use its default")
errNotBareHostname = errors.New( errNotAHost = errors.New("must be a host name such as " +
"must be a bare hostname without scheme, path, or whitespace") "cdn.example.com or .example.com, or an IP address")
errNoHostnameLabels = errors.New("contains no hostname labels") errNotADuration = errors.New("not a duration such as 30s or 2m")
errNotADuration = errors.New("not a duration such as 30s or 2m") errMustBePositive = errors.New("must be positive")
errMustBePositive = errors.New("must be positive") errNotAnOrigin = errors.New(
errNotAnOrigin = errors.New(
`not "*" or an origin such as https://example.com`) `not "*" or an origin such as https://example.com`)
) )
@@ -130,6 +131,10 @@ type Config struct {
AllowHTTP bool // Allow non-TLS upstream (testing only) AllowHTTP bool // Allow non-TLS upstream (testing only)
UpstreamConnectionsPerHost int // Max concurrent connections per upstream host UpstreamConnectionsPerHost int // Max concurrent connections per upstream host
// RefererBlocklist holds host patterns, matched as AllowlistHosts is: the
// image routes refuse a request whose Referer names a matching host.
RefererBlocklist []string
// UpstreamConnections is the most concurrent connections to all // UpstreamConnections is the most concurrent connections to all
// upstream hosts together, on top of the per-host limit. // upstream hosts together, on top of the per-host limit.
// MaxConcurrentProcessing is the most images processed at once. // MaxConcurrentProcessing is the most images processed at once.
@@ -270,7 +275,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
} }
loader := &strictLoader{sc: sc} loader := &strictLoader{sc: sc}
c := &Config{ c := &Config{
Debug: loader.boolVal(keyDebug, false), Debug: loader.boolVal(keyDebug, false),
MaintenanceMode: loader.boolVal(keyMaintenanceMode, false), MaintenanceMode: loader.boolVal(keyMaintenanceMode, false),
@@ -298,9 +302,10 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
keyAccessControlAllowOrigin, DefaultAccessControlAllowOrigin), keyAccessControlAllowOrigin, DefaultAccessControlAllowOrigin),
DownstreamTimeout: loader.durationVal( DownstreamTimeout: loader.durationVal(
keyDownstreamTimeout, DefaultDownstreamTimeout), keyDownstreamTimeout, DefaultDownstreamTimeout),
CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0), CacheMaxBytes: loader.int64Val(keyCacheMaxBytes, 0),
BlockedNetworks: blockedNetworks, BlockedNetworks: blockedNetworks,
TrustedProxies: trustedProxies, TrustedProxies: trustedProxies,
RefererBlocklist: loader.hostListVal(keyRefererBlocklist),
} }
// The default for an omitted cache_max_bytes is worked out when // The default for an omitted cache_max_bytes is worked out when
@@ -420,7 +425,8 @@ func isKnownConfigKey(key string) bool {
keyUpstreamConnectionsPerHost, keyUpstreamConnections, keyUpstreamConnectionsPerHost, keyUpstreamConnections,
keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks, keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks,
keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout, keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout,
keyUpstreamMaxResponseSize, keyDownstreamTimeout, "env": keyUpstreamMaxResponseSize, keyDownstreamTimeout, keyRefererBlocklist,
"env":
return true return true
} }
@@ -443,6 +449,7 @@ func envVarNames() map[string]string {
keyMetricsPassword: "PIXA_METRICS_PASSWORD", keyMetricsPassword: "PIXA_METRICS_PASSWORD",
keySigningKey: "PIXA_SIGNING_KEY", keySigningKey: "PIXA_SIGNING_KEY",
keyAllowlistHosts: "PIXA_ALLOWLIST_HOSTS", keyAllowlistHosts: "PIXA_ALLOWLIST_HOSTS",
keyRefererBlocklist: "PIXA_REFERER_BLOCKLIST",
keyAllowHTTP: "PIXA_ALLOW_HTTP", keyAllowHTTP: "PIXA_ALLOW_HTTP",
keyUpstreamConnectionsPerHost: "PIXA_UPSTREAM_CONNECTIONS_PER_HOST", keyUpstreamConnectionsPerHost: "PIXA_UPSTREAM_CONNECTIONS_PER_HOST",
keyUpstreamConnections: "PIXA_UPSTREAM_CONNECTIONS", keyUpstreamConnections: "PIXA_UPSTREAM_CONNECTIONS",
@@ -612,7 +619,7 @@ func (c *Config) validate() error {
} }
for _, host := range c.AllowlistHosts { for _, host := range c.AllowlistHosts {
err := validateAllowlistHost(host) err := validateHostPattern(keyAllowlistHosts, host)
if err != nil { if err != nil {
return err return err
} }
@@ -735,25 +742,23 @@ func (c *Config) validateConcurrencyLimits() error {
return nil return nil
} }
// validateAllowlistHost checks that an allowlist_hosts entry is a bare // hostNamePattern matches a host name: letters, digits, hyphens and dots,
// hostname, optionally with a leading dot for suffix matching. URLs, // optionally after one leading dot.
// paths, and whitespace indicate a misconfigured entry. An entry with var hostNamePattern = regexp.MustCompile(`^\.?[A-Za-z0-9-][A-Za-z0-9.-]*$`)
// no hostname labels (such as ".") is rejected: the allowlist matcher
// treats a leading dot as a suffix pattern, so a bare "." would match // validateHostPattern checks that an entry of the named key, allowlist_hosts
// any upstream host written in FQDN trailing-dot form and effectively // or referer_blocklist, is an IP address or a host name, the host name
// disable URL signing. // optionally with one leading dot for suffix matching. Anything else, such as
func validateAllowlistHost(host string) error { // a URL, a port or a "*." wildcard, can never match a host, so it is refused.
if strings.Contains(host, "://") || strings.ContainsAny(host, "/ \t") { // So is "." alone: the allowlist matcher would match it against any host
return fmt.Errorf("%s: entry %q %w", // written with a trailing dot, which in allowlist_hosts disables URL signing.
settingName(keyAllowlistHosts), host, errNotBareHostname) func validateHostPattern(key, host string) error {
_, err := netip.ParseAddr(host)
if err == nil || hostNamePattern.MatchString(host) {
return nil
} }
if strings.Trim(host, ".") == "" { return fmt.Errorf("%s: entry %q %w", settingName(key), host, errNotAHost)
return fmt.Errorf("%s: entry %q %w",
settingName(keyAllowlistHosts), host, errNoHostnameLabels)
}
return nil
} }
// loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var // loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var
@@ -883,6 +888,19 @@ func (l *strictLoader) boolVal(key string, defaultVal bool) bool {
return val return val
} }
func (l *strictLoader) hostListVal(key string) []string {
if l.err != nil {
return nil
}
val, err := parseHostList(l.sc, key)
if err != nil {
l.err = err
}
return val
}
// getString returns the string value for key, or defaultVal if the key // getString returns the string value for key, or defaultVal if the key
// is omitted. A present value that is not a string, or is explicitly // is omitted. A present value that is not a string, or is explicitly
// null, is an error. // null, is an error.
@@ -1180,7 +1198,7 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
return nil, errNullConfigValue(key) return nil, errNullConfigValue(key)
} }
entries, err := cidrListEntries(raw, key) entries, err := listEntries(raw, key)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -1200,11 +1218,41 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
return prefixes, nil return prefixes, nil
} }
// cidrListEntries extracts the raw entries of the named CIDR-list key as // parseHostList parses the value of the named config key into host patterns,
// trimmed, non-empty strings, from either a YAML list of strings or a // or returns nil if the key is omitted. It accepts a YAML list of strings or a
// comma-separated string; an empty string is an empty list, as for // comma-separated string. An explicitly null value, a wrong type, an empty
// allowlist_hosts. Any other shape is a configuration error. // entry, a non-string entry, or an entry validateHostPattern rejects aborts
func cidrListEntries(raw any, key string) ([]string, error) { // startup naming the key and the offending value.
func parseHostList(sc *smartconfig.Config, key string) ([]string, error) {
raw, ok := lookupValue(sc, key)
if !ok {
return nil, nil
}
if raw == nil {
return nil, errNullConfigValue(key)
}
entries, err := listEntries(raw, key)
if err != nil {
return nil, err
}
for _, entry := range entries {
err := validateHostPattern(key, entry)
if err != nil {
return nil, err
}
}
return entries, nil
}
// listEntries extracts the raw entries of the named list key as trimmed,
// non-empty strings, from either a YAML list of strings or a comma-separated
// string; an empty string is an empty list, as for allowlist_hosts. Any other
// shape is a configuration error.
func listEntries(raw any, key string) ([]string, error) {
switch val := raw.(type) { switch val := raw.(type) {
case []any: case []any:
entries := make([]string, 0, len(val)) entries := make([]string, 0, len(val))
@@ -318,6 +318,20 @@ func invalidHostAndCredentialCases() []abortCase {
keyAllowlistHosts, "example.com/images", keyAllowlistHosts, "example.com/images",
}, },
}, },
{
name: "allowlist host with wildcard",
yaml: signingKeyLine + "allowlist_hosts:\n - \"*.example.com\"\n",
wantErrSubstrings: []string{
keyAllowlistHosts, "*.example.com",
},
},
{
name: "allowlist host with port",
yaml: signingKeyLine + "allowlist_hosts:\n - example.com:8443\n",
wantErrSubstrings: []string{
keyAllowlistHosts, "example.com:8443",
},
},
{ {
name: "allowlist host with whitespace", name: "allowlist host with whitespace",
yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n", yaml: signingKeyLine + "allowlist_hosts:\n - \"exa mple.com\"\n",
+2
View File
@@ -65,6 +65,7 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
t.Setenv("PIXA_METRICS_PASSWORD", "metricspass") t.Setenv("PIXA_METRICS_PASSWORD", "metricspass")
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey) t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
t.Setenv("PIXA_ALLOWLIST_HOSTS", "s3.sneak.cloud,.example.com") t.Setenv("PIXA_ALLOWLIST_HOSTS", "s3.sneak.cloud,.example.com")
t.Setenv("PIXA_REFERER_BLOCKLIST", "hotlinker.example,.leech.example")
t.Setenv("PIXA_ALLOW_HTTP", "true") t.Setenv("PIXA_ALLOW_HTTP", "true")
t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5") t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5")
t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10") t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10")
@@ -93,6 +94,7 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
MetricsPassword: "metricspass", MetricsPassword: "metricspass",
SigningKey: validTestSigningKey, SigningKey: validTestSigningKey,
AllowlistHosts: []string{testHostS3, ".example.com"}, AllowlistHosts: []string{testHostS3, ".example.com"},
RefererBlocklist: []string{"hotlinker.example", ".leech.example"},
AllowHTTP: true, AllowHTTP: true,
UpstreamConnectionsPerHost: 5, UpstreamConnectionsPerHost: 5,
UpstreamConnections: 10, UpstreamConnections: 10,
@@ -0,0 +1,147 @@
package config
import (
"slices"
"testing"
)
// TestRefererBlocklistParsed loads a referer_blocklist with a host and a
// pattern starting with "." and checks both are kept in order.
func TestRefererBlocklistParsed(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
- leech.example
- .hotlinker.example
`)
if err != nil {
t.Fatalf("valid referer_blocklist should load, got error: %v", err)
}
want := []string{"leech.example", ".hotlinker.example"}
if !slices.Equal(c.RefererBlocklist, want) {
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
}
}
// TestRefererBlocklistAcceptsIPAddresses checks that IPv4 and IPv6 addresses,
// the IPv6 one written without brackets, are accepted as entries.
func TestRefererBlocklistAcceptsIPAddresses(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
- 192.0.2.7
- "2001:db8::7"
`)
if err != nil {
t.Fatalf("IP address entries should load, got error: %v", err)
}
want := []string{"192.0.2.7", "2001:db8::7"}
if !slices.Equal(c.RefererBlocklist, want) {
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
}
}
// TestRefererBlocklistOmittedIsEmpty checks that an omitted key blocks no
// referer.
func TestRefererBlocklistOmittedIsEmpty(t *testing.T) {
t.Parallel()
c, err := configFromYAML(t, signingKeyLine)
if err != nil {
t.Fatalf("minimal config should be valid, got error: %v", err)
}
if len(c.RefererBlocklist) != 0 {
t.Errorf("RefererBlocklist = %v, want empty", c.RefererBlocklist)
}
}
// TestRefererBlocklistInvalidAbortsStartup checks that an entry that is not a
// host, or a value that is not a list of them, aborts startup with an error
// naming the key and the entry.
func TestRefererBlocklistInvalidAbortsStartup(t *testing.T) {
t.Parallel()
runAbortCases(t, []abortCase{
{
name: "entry with a scheme",
yaml: signingKeyLine + "referer_blocklist:\n - https://leech.example\n",
wantErrSubstrings: []string{
keyRefererBlocklist, "https://leech.example",
},
},
{
name: "entry with a path",
yaml: signingKeyLine + "referer_blocklist:\n - leech.example/page\n",
wantErrSubstrings: []string{
keyRefererBlocklist, "leech.example/page",
},
},
{
name: "wildcard entry",
yaml: signingKeyLine + "referer_blocklist:\n - \"*.leech.example\"\n",
wantErrSubstrings: []string{
keyRefererBlocklist, "*.leech.example",
},
},
{
name: "entry with a port",
yaml: signingKeyLine + "referer_blocklist:\n - leech.example:8080\n",
wantErrSubstrings: []string{
keyRefererBlocklist, "leech.example:8080",
},
},
{
name: "two leading dots",
yaml: signingKeyLine + "referer_blocklist:\n - ..leech.example\n",
wantErrSubstrings: []string{
keyRefererBlocklist, "..leech.example",
},
},
{
name: "dot only",
yaml: signingKeyLine + "referer_blocklist:\n - \".\"\n",
wantErrSubstrings: []string{keyRefererBlocklist, `"."`},
},
{
name: "empty entry",
yaml: signingKeyLine + "referer_blocklist:\n - \"\"\n",
wantErrSubstrings: []string{keyRefererBlocklist},
},
{
name: "entry not a string",
yaml: signingKeyLine + "referer_blocklist:\n - 42\n",
wantErrSubstrings: []string{keyRefererBlocklist, "42"},
},
{
name: "null value",
yaml: signingKeyLine + "referer_blocklist:\n",
wantErrSubstrings: []string{keyRefererBlocklist, nullValueText},
},
})
}
// TestRefererBlocklistFromEnvironment checks that PIXA_REFERER_BLOCKLIST
// takes comma-separated entries, and that an entry in it that is not a host
// aborts startup naming the variable and the entry.
func TestRefererBlocklistFromEnvironment(t *testing.T) {
t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
t.Setenv("PIXA_REFERER_BLOCKLIST", " leech.example , .hotlinker.example ")
c, err := newFromSmartConfig(nil)
if err != nil {
t.Fatalf("valid PIXA_REFERER_BLOCKLIST should load, got error: %v", err)
}
want := []string{"leech.example", ".hotlinker.example"}
if !slices.Equal(c.RefererBlocklist, want) {
t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
}
t.Setenv("PIXA_REFERER_BLOCKLIST", "leech.example,https://hotlinker.example")
_, err = newFromSmartConfig(nil)
wantStartupError(t, err, "PIXA_REFERER_BLOCKLIST", "https://hotlinker.example")
}
+11 -5
View File
@@ -9,6 +9,7 @@ import (
"time" "time"
"go.uber.org/fx" "go.uber.org/fx"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/config" "sneak.berlin/go/pixa/internal/config"
"sneak.berlin/go/pixa/internal/database" "sneak.berlin/go/pixa/internal/database"
"sneak.berlin/go/pixa/internal/encurl" "sneak.berlin/go/pixa/internal/encurl"
@@ -40,6 +41,10 @@ type Handlers struct {
sessMgr *session.Manager sessMgr *session.Manager
encGen *encurl.Generator encGen *encurl.Generator
csrfProtect func(http.Handler) http.Handler csrfProtect func(http.Handler) http.Handler
// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
// reports whether a URL's host is on that list.
refererBlocklist *allowlist.HostAllowList
} }
// New creates a new Handlers instance. // New creates a new Handlers instance.
@@ -50,11 +55,12 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
} }
s := &Handlers{ s := &Handlers{
log: params.Logger.Get(), log: params.Logger.Get(),
hc: params.Healthcheck, hc: params.Healthcheck,
db: params.Database, db: params.Database,
config: params.Config, config: params.Config,
csrfProtect: csrfProtect, csrfProtect: csrfProtect,
refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
} }
lc.Append(fx.Hook{ lc.Append(fx.Hook{
+21
View File
@@ -21,6 +21,10 @@ import (
// /v1/image/<host>/<path>/<width>x<height>.<format> // /v1/image/<host>/<path>/<width>x<height>.<format>
func (s *Handlers) HandleImage() http.HandlerFunc { func (s *Handlers) HandleImage() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
if s.refuseBlockedReferer(w, r) {
return
}
req, ok := s.parseImageRequest(w, r) req, ok := s.parseImageRequest(w, r)
if !ok { if !ok {
return return
@@ -248,6 +252,23 @@ func cacheControl(expires time.Time) string {
return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second)) return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second))
} }
// refuseBlockedReferer answers 403 with a JSON error when the request's Referer
// names a host on referer_blocklist, and reports whether it answered. A request
// with no Referer, or one that does not parse as a URL with a host, is not
// refused.
func (s *Handlers) refuseBlockedReferer(
w http.ResponseWriter, r *http.Request,
) bool {
referer, err := url.Parse(r.Referer())
if err != nil || !s.refererBlocklist.IsAllowed(referer) {
return false
}
s.respondError(w, "referer blocked", http.StatusForbidden)
return true
}
// notModified sets the ETag header to etag and, when the request's // notModified sets the ETag header to etag and, when the request's
// If-None-Match is that ETag, answers 304 Not Modified. It reports whether it // If-None-Match is that ETag, answers 304 Not Modified. It reports whether it
// answered. An empty etag sets no header and never answers. // answered. An empty etag sets no header and never answers.
+4
View File
@@ -22,6 +22,10 @@ import (
// browsers identify the content type. // browsers identify the content type.
func (s *Handlers) HandleImageEnc() http.HandlerFunc { func (s *Handlers) HandleImageEnc() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) {
if s.refuseBlockedReferer(w, r) {
return
}
ctx := r.Context() ctx := r.Context()
start := time.Now() start := time.Now()
@@ -0,0 +1,185 @@
package handlers
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/pixa/internal/allowlist"
"sneak.berlin/go/pixa/internal/encurl"
"sneak.berlin/go/pixa/internal/httpfetcher"
"sneak.berlin/go/pixa/internal/imgcache"
)
// blockedReferer is a page on leech.example, which newRefererRoutes puts on
// referer_blocklist.
const blockedReferer = "https://leech.example/page.html"
// countingFetcher passes each fetch on to the fetcher it holds and counts it.
type countingFetcher struct {
httpfetcher.Fetcher
fetches atomic.Int32
}
// Fetch counts the fetch and passes it on.
func (f *countingFetcher) Fetch(
ctx context.Context, url string,
) (*httpfetcher.FetchResult, error) {
f.fetches.Add(1)
return f.Fetcher.Fetch(ctx, url)
}
// newRefererRoutes returns both image routes of a Handlers whose
// referer_blocklist is "leech.example" and ".hotlinker.example", the
// Handlers, and the fetcher the routes fetch through. The JPEG at photoPath
// exists on allowlistedHost and on signedHost.
func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher) {
t.Helper()
fetcher := &countingFetcher{
Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
}
cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
StateDir: t.TempDir(),
CacheTTL: time.Hour,
NegativeTTL: 5 * time.Minute,
})
if err != nil {
t.Fatalf("imgcache.NewCache() error = %v", err)
}
svc, err := imgcache.NewService(&imgcache.ServiceConfig{
Cache: cache,
Fetcher: fetcher,
SigningKey: testSigningKey,
Allowlist: []string{allowlistedHost},
})
if err != nil {
t.Fatalf("imgcache.NewService() error = %v", err)
}
encGen, err := encurl.NewGenerator(testSigningKey)
if err != nil {
t.Fatalf("encurl.NewGenerator() error = %v", err)
}
h := &Handlers{
log: slog.New(slog.DiscardHandler),
imgSvc: svc,
encGen: encGen,
refererBlocklist: allowlist.New(
[]string{"leech.example", ".hotlinker.example"}),
}
r := chi.NewRouter()
r.Get("/v1/image/*", h.HandleImage())
r.Get("/v1/e/{token}/*", h.HandleImageEnc())
return r, h, fetcher
}
// getWithReferer sends a GET for target to routes with referer as its
// Referer header, or with none when referer is empty, and returns the
// response.
func getWithReferer(
t *testing.T, routes http.Handler, target, referer string,
) *httptest.ResponseRecorder {
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
if referer != "" {
req.Header.Set("Referer", referer)
}
rec := httptest.NewRecorder()
routes.ServeHTTP(rec, req)
t.Logf("GET %s with Referer %q: %d", target, referer, rec.Code)
return rec
}
// TestRefererBlocklist verifies that both image routes refuse a request whose
// Referer names a host on referer_blocklist with 403 and the JSON error,
// without fetching from the upstream host, and serve a request with no
// Referer, one that does not parse, or one naming any other host. Hosts are
// matched as allowlist_hosts matches them.
func TestRefererBlocklist(t *testing.T) {
t.Parallel()
cases := []struct {
name string
referer string
want int
}{
{"no referer", "", http.StatusOK},
{"unlisted host", "https://unlisted.example/page.html", http.StatusOK},
{"unparseable", "%zz", http.StatusOK},
{"listed host", blockedReferer, http.StatusForbidden},
{"subdomain of listed host", "https://www.leech.example/", http.StatusOK},
{"subdomain of dot pattern", "https://www.hotlinker.example/a.html",
http.StatusForbidden},
{"dot pattern without its dot", "https://hotlinker.example/",
http.StatusForbidden},
{"host continuing past dot pattern",
"https://hotlinker.example.evil.example/", http.StatusOK},
}
// The photo's URL on each image route.
photoURLs := map[string]func(t *testing.T, h *Handlers) string{
"plain URL": func(t *testing.T, _ *Handlers) string {
t.Helper()
return photoURL(allowlistedHost)
},
"encrypted URL": encPhotoURL,
}
for urlName, photoURLFor := range photoURLs {
for _, tc := range cases {
t.Run(urlName+", "+tc.name, func(t *testing.T) {
t.Parallel()
routes, h, fetcher := newRefererRoutes(t)
rec := getWithReferer(t, routes, photoURLFor(t, h), tc.referer)
if tc.want == http.StatusOK {
requireServedPhoto(t, rec)
return
}
checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
if n := fetcher.fetches.Load(); n != 0 {
t.Errorf("upstream fetched %d times, want 0", n)
}
})
}
}
}
// TestBlockedRefererRefusedWhenImageIsCached verifies that a request whose
// Referer is on referer_blocklist is refused even when the image it asks for
// is already cached, so the answer does not depend on the cache.
func TestBlockedRefererRefusedWhenImageIsCached(t *testing.T) {
t.Parallel()
routes, h, _ := newRefererRoutes(t)
for _, target := range []string{photoURL(allowlistedHost), encPhotoURL(t, h)} {
requireServedPhoto(t, getWithReferer(t, routes, target, ""))
rec := getWithReferer(t, routes, target, blockedReferer)
checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
}
}