Test the image proxy flow end to end (closes #80)

TestImageProxyFlow in internal/server starts the database, handlers and
middleware from the constructors pixad uses, in an fx app with a fresh
state directory, and replaces only the upstream origin with a local test
server, reached through the real fetcher by its new DialContext. For a
resize with a change to JPEG and for orig, the first request answers 200
with the right content type, decoded size and X-Pixa-Cache MISS; the
second answers HIT with the same image while the upstream has had one
request; the source and the converted image are then on disk with their
rows in SQLite. TODO.md records it and drops the item from Future Steps.

Model: opus-5-5

README.md

@@ -45,8 +45,7 @@ everything in this list without further settings. The reverse proxy must:
   Routes);
 - pass the `Host`, `Origin` and `Referer` headers on unchanged, as pixa refuses
   a form from those pages unless `Origin` or `Referer` names the host in `Host`,
-  builds encrypted URLs from `Host`, and checks `Referer` against
+  and builds encrypted URLs from `Host`;
-  `referer_blocklist`;
 - set `X-Forwarded-For` to the client's address, with `trusted_proxies` set to
   the address pixa sees the proxy's requests come from, so the login limit
   counts each client by its own address (see `trusted_proxies` under
@@ -176,13 +175,11 @@ path under `/v1/` answers 200, in maintenance mode too.
   allowlisted (see Source Hosts). Answers: 200; 304 when `If-None-Match` matches
   the image's `ETag`; 400 for a URL or parameter that is not valid; 401 for a
   missing or wrong signature, a missing `exp` or an `exp` in the past; 403 when
-  the request's `Referer` names a host in `referer_blocklist`, checked before
+  the upstream host, or a host it redirects to, is `localhost`, ends in
-  the signature, the cache and the upstream fetch; 403 when the upstream host,
+  `.localhost` or `.local`, or has an address in a blocked network (see
-  or a host it redirects to, is `localhost`, ends in `.localhost` or `.local`,
+  `blocked_networks`); 502 when the upstream answered with an error status, and
-  or has an address in a blocked network (see `blocked_networks`); 502 when the
+  for 5 minutes after that for the same source URL; 503 when pixa is busy or in
-  upstream answered with an error status, and for 5 minutes after that for the
+  maintenance mode; 500 for any other failure.
-  same source URL; 503 when pixa is busy or in maintenance mode; 500 for any
-  other failure.
 - `GET` or `HEAD` `/v1/e/<token>/<name>` — an image through an encrypted URL
   (see Encrypted URLs). Needs: nothing but the URL. Answers: 200; 304 when
   `If-None-Match` matches the image's `ETag`; 400 for a token that does not
@@ -395,11 +392,6 @@ and the URL is
 - **Suffix match**: `.example.com` — matches `cdn.example.com`,
   `images.example.com`, and `example.com`
 
-An IP address is matched exactly; write an IPv6 address without brackets. An
-entry that is neither a host name (letters, digits, hyphens, underscores and
-dots, with at most one leading dot) nor an IP address, such as one with a port
-or a `*.` wildcard, aborts startup.
-
 ### Configuration
 
 Every setting can be given as an environment variable, in a YAML config
@@ -436,7 +428,6 @@ tell. With no file, pixa uses the environment and the defaults.
 | `PIXA_DB_URL`                        | `db_url`                        | SQLite database URL; default `state.sqlite3` in the state directory          |
 | `PIXA_CACHE_MAX_BYTES`               | `cache_max_bytes`               | Disk cache limit in bytes; `0` disables it; default 75% of (free + cached)   |
 | `PIXA_ALLOWLIST_HOSTS`               | `allowlist_hosts`               | Upstream hosts served without a signature                                    |
-| `PIXA_REFERER_BLOCKLIST`             | `referer_blocklist`             | Hosts whose pages the image routes refuse with 403, by `Referer`             |
 | `PIXA_BLOCKED_NETWORKS`              | `blocked_networks`              | CIDR ranges never fetched from, on top of the built-in ones                  |
 | `PIXA_TRUSTED_PROXIES`               | `trusted_proxies`               | CIDR ranges of proxies whose `X-Forwarded-For` is believed; default RFC 1918 |
 | `PIXA_ALLOW_HTTP`                    | `allow_http`                    | Allow plain-HTTP upstreams, for testing only; default `false`                |
@@ -465,18 +456,6 @@ Key settings in more detail:
   that has no leading zero and is not the scheme's default. Any other value,
   including another scheme such as a browser extension's, aborts startup
 - `allowlist_hosts` — list of allowed upstream hosts
-- `referer_blocklist` — list of hosts whose pages may not show pixa's images, to
-  stop other sites hotlinking them. Entries are written and matched as for
-  `allowlist_hosts` (see Allowlist patterns), and an entry that is neither a
-  host name nor an IP address aborts startup. A request to `/v1/image/` or
-  `/v1/e/` whose `Referer` header names a listed host is refused with 403 before
-  its signature or token is checked and before the cache or the upstream host is
-  used, so it fetches nothing, and it is refused even when the image is cached.
-  A request with no `Referer`, or one that does not parse as a URL
-  with a host, is served, as many clients send none. So this is easily got
-  around: a site whose pages send no `Referer` (for example with
-  `Referrer-Policy: no-referrer`) is not stopped. It does not apply to the login
-  and generator pages. Default: empty
 - `blocked_networks` — list of CIDR ranges to refuse for SSRF protection,
   added to the always-enforced built-in ranges (loopback, private,
   link-local, CGNAT, benchmark, NAT64, and the like); an invalid CIDR

TODO.md

@@ -27,7 +27,7 @@ The disk cache is now size-bounded with LRU eviction
 
 # Next Step
 
-P2: security: per-IP rate limiting on the image routes
+P2: security: referer blacklist
 
 # Completed Steps
 
@@ -46,25 +46,6 @@ P2: security: per-IP rate limiting on the image routes
   `httpfetcher.Config.DialContext` connects in place of the dialer that refuses
   internal addresses, the URL and redirect checks still running, and
   `handlers.Params.Fetcher` replaces the fetcher the handlers build.
-- 2026-10-04 a URL made on the generator page with a `ttl` is tested to
-  expire (closes #199): a new test in `internal/handlers` makes a URL on the
-  generator page with a `ttl` of one second, checks that `/v1/e/` serves it at
-  once, waits two seconds and checks that it then answers 410. The test waits
-  for real, as pixa reads the clock directly when it makes and checks a URL; it
-  waits two seconds because the time a URL expires is kept in whole seconds.
-  Test only.
-- 2026-10-04 referer blocklist (closes #90): `referer_blocklist`
-  (`PIXA_REFERER_BLOCKLIST`) lists hosts, written and matched as for
-  `allowlist_hosts` with the same matcher; an entry of either list that is
-  neither a host name (letters, digits, hyphens, underscores and dots, with at
-  most one leading dot) nor an IP address, such as one with a port or a `*.`
-  wildcard, aborts startup naming the setting and the entry. Both image routes
-  refuse a request whose `Referer` names a listed host with 403 and a JSON error
-  before the signature, the cache and the upstream fetch, so it fetches nothing
-  and is refused whether or not the image is cached. A request with no
-  `Referer`, or one that does not parse as a URL with a host, is served, so the
-  list is easily got around; `README.md` and `configs/config.example.yml` say
-  so. It does not apply to the login and generator pages.
 - 2026-10-04 fewer files in the repository root (closes #97):
   `config.example.yml` moved unchanged to `configs/config.example.yml`, and
   `README.md`, the comments in `internal/config/config.go` and the startup error
@@ -595,6 +576,7 @@ P2: security: per-IP rate limiting on the image routes
 # Future Steps
 
 - P2: security
+    - per-IP rate limiting on the image routes
     - per-origin rate limiting
 - P2: HTTP response handling
     - Last-Modified headers

configs/config.example.yml

@@ -46,8 +46,6 @@ signing_key: "CHANGE_ME_generate_with_openssl_rand_base64_32"
 
 # Hosts that don't require signatures (default: none)
 # Use "." prefix for wildcard subdomain matching (e.g., ".example.com" matches "cdn.example.com")
-# An entry that is neither a host name nor an IP address (IPv6 without
-# brackets), such as one with a port or a "*." wildcard, aborts startup.
 allowlist_hosts:
   - s3.sneak.cloud
   - static.sneak.cloud
@@ -55,16 +53,6 @@ allowlist_hosts:
   - github.com
   - user-images.githubusercontent.com
 
-# Hosts whose pages may not show pixa's images, written as for
-# allowlist_hosts. A request to /v1/image/ or /v1/e/ whose Referer header
-# names one of them is answered 403 before anything is fetched, even when
-# the image is cached. A request with no Referer, or one that does not
-# parse, is served, so a site whose pages send no Referer is not stopped.
-# The login and generator pages are not covered. (default: none)
-# referer_blocklist:
-#   - leech.example
-#   - .hotlinker.example
-
 # Additional CIDR ranges to refuse when fetching upstream, extending the
 # SSRF protection. These are added to the always-enforced built-in ranges
 # (loopback, RFC 1918 private, link-local, CGNAT, benchmark, NAT64, and

internal/config/config.go

@@ -11,7 +11,6 @@ import (
 	"net/url"
 	"os"
 	"path/filepath"
-	"regexp"
 	"runtime"
 	"sort"
 	"strconv"
@@ -49,7 +48,6 @@ const (
 	keyMetricsPassword            = "metrics.password"
 	keySigningKey                 = "signing_key"
 	keyAllowlistHosts             = "allowlist_hosts"
-	keyRefererBlocklist           = "referer_blocklist"
 	keyAllowHTTP                  = "allow_http"
 	keyUpstreamConnectionsPerHost = "upstream_connections_per_host"
 	keyUpstreamConnections        = "upstream_connections"
@@ -99,8 +97,9 @@ var (
 		"value is null; omit the key entirely to use the default")
 	errValuesNull = errors.New(
 		"value is null; omit a key entirely to use its default")
-	errNotAHost = errors.New("must be a host name such as " +
+	errNotBareHostname = errors.New(
-		"cdn.example.com or .example.com, or an IP address")
+		"must be a bare hostname without scheme, path, or whitespace")
+	errNoHostnameLabels = errors.New("contains no hostname labels")
 	errNotADuration     = errors.New("not a duration such as 30s or 2m")
 	errMustBePositive   = errors.New("must be positive")
 	errNotAnOrigin      = errors.New(
@@ -131,10 +130,6 @@ type Config struct {
 	AllowHTTP                  bool     // Allow non-TLS upstream (testing only)
 	UpstreamConnectionsPerHost int      // Max concurrent connections per upstream host
 
-	// RefererBlocklist holds host patterns, matched as AllowlistHosts is: the
-	// image routes refuse a request whose Referer names a matching host.
-	RefererBlocklist []string
-
 	// UpstreamConnections is the most concurrent connections to all
 	// upstream hosts together, on top of the per-host limit.
 	// MaxConcurrentProcessing is the most images processed at once.
@@ -275,6 +270,7 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
 	}
 
 	loader := &strictLoader{sc: sc}
+
 	c := &Config{
 		Debug:           loader.boolVal(keyDebug, false),
 		MaintenanceMode: loader.boolVal(keyMaintenanceMode, false),
@@ -305,7 +301,6 @@ func newFromSmartConfig(sc *smartconfig.Config) (*Config, error) {
 		CacheMaxBytes:   loader.int64Val(keyCacheMaxBytes, 0),
 		BlockedNetworks: blockedNetworks,
 		TrustedProxies:  trustedProxies,
-		RefererBlocklist: loader.hostListVal(keyRefererBlocklist),
 	}
 
 	// The default for an omitted cache_max_bytes is worked out when
@@ -425,8 +420,7 @@ func isKnownConfigKey(key string) bool {
 		keyUpstreamConnectionsPerHost, keyUpstreamConnections,
 		keyMaxConcurrentProcessing, keyCacheMaxBytes, keyBlockedNetworks,
 		keyTrustedProxies, keyAccessControlAllowOrigin, keyUpstreamFetchTimeout,
-		keyUpstreamMaxResponseSize, keyDownstreamTimeout, keyRefererBlocklist,
+		keyUpstreamMaxResponseSize, keyDownstreamTimeout, "env":
-		"env":
 		return true
 	}
 
@@ -449,7 +443,6 @@ func envVarNames() map[string]string {
 		keyMetricsPassword:            "PIXA_METRICS_PASSWORD",
 		keySigningKey:                 "PIXA_SIGNING_KEY",
 		keyAllowlistHosts:             "PIXA_ALLOWLIST_HOSTS",
-		keyRefererBlocklist:           "PIXA_REFERER_BLOCKLIST",
 		keyAllowHTTP:                  "PIXA_ALLOW_HTTP",
 		keyUpstreamConnectionsPerHost: "PIXA_UPSTREAM_CONNECTIONS_PER_HOST",
 		keyUpstreamConnections:        "PIXA_UPSTREAM_CONNECTIONS",
@@ -619,7 +612,7 @@ func (c *Config) validate() error {
 	}
 
 	for _, host := range c.AllowlistHosts {
-		err := validateHostPattern(keyAllowlistHosts, host)
+		err := validateAllowlistHost(host)
 		if err != nil {
 			return err
 		}
@@ -742,24 +735,25 @@ func (c *Config) validateConcurrencyLimits() error {
 	return nil
 }
 
-// hostNamePattern matches a host name: letters, digits, hyphens, underscores
+// validateAllowlistHost checks that an allowlist_hosts entry is a bare
-// and dots, optionally after one leading dot.
+// hostname, optionally with a leading dot for suffix matching. URLs,
-var hostNamePattern = regexp.MustCompile(`^\.?[A-Za-z0-9_-][A-Za-z0-9_.-]*$`)
+// paths, and whitespace indicate a misconfigured entry. An entry with
-
+// no hostname labels (such as ".") is rejected: the allowlist matcher
-// validateHostPattern checks that an entry of the named key, allowlist_hosts
+// treats a leading dot as a suffix pattern, so a bare "." would match
-// or referer_blocklist, is an IP address or a host name, the host name
+// any upstream host written in FQDN trailing-dot form and effectively
-// optionally with one leading dot for suffix matching. Anything else, such as
+// disable URL signing.
-// a URL, a port or a "*." wildcard, can never match a host name that resolves,
+func validateAllowlistHost(host string) error {
-// so it is refused.
+	if strings.Contains(host, "://") || strings.ContainsAny(host, "/ \t") {
-// So is "." alone: the allowlist matcher would match it against any host
+		return fmt.Errorf("%s: entry %q %w",
-// written with a trailing dot, which in allowlist_hosts disables URL signing.
+			settingName(keyAllowlistHosts), host, errNotBareHostname)
-func validateHostPattern(key, host string) error {
-	_, err := netip.ParseAddr(host)
-	if err == nil || hostNamePattern.MatchString(host) {
-		return nil
 	}
 
-	return fmt.Errorf("%s: entry %q %w", settingName(key), host, errNotAHost)
+	if strings.Trim(host, ".") == "" {
+		return fmt.Errorf("%s: entry %q %w",
+			settingName(keyAllowlistHosts), host, errNoHostnameLabels)
+	}
+
+	return nil
 }
 
 // loadConfigFile loads configuration from the PIXA_CONFIG_PATH env var
@@ -889,19 +883,6 @@ func (l *strictLoader) boolVal(key string, defaultVal bool) bool {
 	return val
 }
 
-func (l *strictLoader) hostListVal(key string) []string {
-	if l.err != nil {
-		return nil
-	}
-
-	val, err := parseHostList(l.sc, key)
-	if err != nil {
-		l.err = err
-	}
-
-	return val
-}
-
 // getString returns the string value for key, or defaultVal if the key
 // is omitted. A present value that is not a string, or is explicitly
 // null, is an error.
@@ -1199,7 +1180,7 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
 		return nil, errNullConfigValue(key)
 	}
 
-	entries, err := listEntries(raw, key)
+	entries, err := cidrListEntries(raw, key)
 	if err != nil {
 		return nil, err
 	}
@@ -1219,41 +1200,11 @@ func parseCIDRList(sc *smartconfig.Config, key string) ([]netip.Prefix, error) {
 	return prefixes, nil
 }
 
-// parseHostList parses the value of the named config key into host patterns,
+// cidrListEntries extracts the raw entries of the named CIDR-list key as
-// or returns nil if the key is omitted. It accepts a YAML list of strings or a
+// trimmed, non-empty strings, from either a YAML list of strings or a
-// comma-separated string. An explicitly null value, a wrong type, an empty
+// comma-separated string; an empty string is an empty list, as for
-// entry, a non-string entry, or an entry validateHostPattern rejects aborts
+// allowlist_hosts. Any other shape is a configuration error.
-// startup naming the key and the offending value.
+func cidrListEntries(raw any, key string) ([]string, error) {
-func parseHostList(sc *smartconfig.Config, key string) ([]string, error) {
-	raw, ok := lookupValue(sc, key)
-	if !ok {
-		return nil, nil
-	}
-
-	if raw == nil {
-		return nil, errNullConfigValue(key)
-	}
-
-	entries, err := listEntries(raw, key)
-	if err != nil {
-		return nil, err
-	}
-
-	for _, entry := range entries {
-		err := validateHostPattern(key, entry)
-		if err != nil {
-			return nil, err
-		}
-	}
-
-	return entries, nil
-}
-
-// listEntries extracts the raw entries of the named list key as trimmed,
-// non-empty strings, from either a YAML list of strings or a comma-separated
-// string; an empty string is an empty list, as for allowlist_hosts. Any other
-// shape is a configuration error.
-func listEntries(raw any, key string) ([]string, error) {
 	switch val := raw.(type) {
 	case []any:
 		entries := make([]string, 0, len(val))

internal/config/config_validation_internal_test.go

@@ -5,7 +5,6 @@ import (
 	"log/slog"
 	"os"
 	"path/filepath"
-	"slices"
 	"strings"
 	"testing"
 	"time"
@@ -217,25 +216,6 @@ func TestCommaSeparatedAllowlistStillSupported(t *testing.T) {
 	}
 }
 
-// TestAllowlistHostsAcceptsUnderscore checks that an upstream host name with
-// an underscore, which pixa can fetch from, is accepted as an entry.
-func TestAllowlistHostsAcceptsUnderscore(t *testing.T) {
-	t.Parallel()
-
-	c, err := configFromYAML(t, signingKeyLine+`allowlist_hosts:
-  - my_bucket.example.com
-  - .my_bucket.example.org
-`)
-	if err != nil {
-		t.Fatalf("host names with an underscore should load, got error: %v", err)
-	}
-
-	want := []string{"my_bucket.example.com", ".my_bucket.example.org"}
-	if !slices.Equal(c.AllowlistHosts, want) {
-		t.Errorf("AllowlistHosts = %v, want %v", c.AllowlistHosts, want)
-	}
-}
-
 // runAbortCases asserts that each case's config aborts startup with an
 // error message mentioning every expected substring.
 func runAbortCases(t *testing.T, cases []abortCase) {
@@ -338,20 +318,6 @@ func invalidHostAndCredentialCases() []abortCase {
 				keyAllowlistHosts, "example.com/images",
 			},
 		},
-		{
-			name: "allowlist host with wildcard",
-			yaml: signingKeyLine + "allowlist_hosts:\n  - \"*.example.com\"\n",
-			wantErrSubstrings: []string{
-				keyAllowlistHosts, "*.example.com",
-			},
-		},
-		{
-			name: "allowlist host with port",
-			yaml: signingKeyLine + "allowlist_hosts:\n  - example.com:8443\n",
-			wantErrSubstrings: []string{
-				keyAllowlistHosts, "example.com:8443",
-			},
-		},
 		{
 			name:              "allowlist host with whitespace",
 			yaml:              signingKeyLine + "allowlist_hosts:\n  - \"exa mple.com\"\n",

internal/config/env_internal_test.go

@@ -65,7 +65,6 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
 	t.Setenv("PIXA_METRICS_PASSWORD", "metricspass")
 	t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
 	t.Setenv("PIXA_ALLOWLIST_HOSTS", "s3.sneak.cloud,.example.com")
-	t.Setenv("PIXA_REFERER_BLOCKLIST", "hotlinker.example,.leech.example")
 	t.Setenv("PIXA_ALLOW_HTTP", "true")
 	t.Setenv("PIXA_UPSTREAM_CONNECTIONS_PER_HOST", "5")
 	t.Setenv("PIXA_UPSTREAM_CONNECTIONS", "10")
@@ -94,7 +93,6 @@ func TestEnvironmentSetsEveryKey(t *testing.T) {
 		MetricsPassword:            "metricspass",
 		SigningKey:                 validTestSigningKey,
 		AllowlistHosts:             []string{testHostS3, ".example.com"},
-		RefererBlocklist:           []string{"hotlinker.example", ".leech.example"},
 		AllowHTTP:                  true,
 		UpstreamConnectionsPerHost: 5,
 		UpstreamConnections:        10,

internal/config/referer_blocklist_internal_test.go

@@ -1,166 +0,0 @@
-package config
-
-import (
-	"slices"
-	"testing"
-)
-
-// TestRefererBlocklistParsed loads a referer_blocklist with a host and a
-// pattern starting with "." and checks both are kept in order.
-func TestRefererBlocklistParsed(t *testing.T) {
-	t.Parallel()
-
-	c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
-  - leech.example
-  - .hotlinker.example
-`)
-	if err != nil {
-		t.Fatalf("valid referer_blocklist should load, got error: %v", err)
-	}
-
-	want := []string{"leech.example", ".hotlinker.example"}
-	if !slices.Equal(c.RefererBlocklist, want) {
-		t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
-	}
-}
-
-// TestRefererBlocklistAcceptsIPAddresses checks that IPv4 and IPv6 addresses,
-// the IPv6 one written without brackets, are accepted as entries.
-func TestRefererBlocklistAcceptsIPAddresses(t *testing.T) {
-	t.Parallel()
-
-	c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
-  - 192.0.2.7
-  - "2001:db8::7"
-`)
-	if err != nil {
-		t.Fatalf("IP address entries should load, got error: %v", err)
-	}
-
-	want := []string{"192.0.2.7", "2001:db8::7"}
-	if !slices.Equal(c.RefererBlocklist, want) {
-		t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
-	}
-}
-
-// TestRefererBlocklistAcceptsUnderscore checks that a host name with an
-// underscore, which a page can be served from, is accepted as an entry.
-func TestRefererBlocklistAcceptsUnderscore(t *testing.T) {
-	t.Parallel()
-
-	c, err := configFromYAML(t, signingKeyLine+`referer_blocklist:
-  - my_site.leech.example
-  - .my_site.hotlinker.example
-`)
-	if err != nil {
-		t.Fatalf("host names with an underscore should load, got error: %v", err)
-	}
-
-	want := []string{"my_site.leech.example", ".my_site.hotlinker.example"}
-	if !slices.Equal(c.RefererBlocklist, want) {
-		t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
-	}
-}
-
-// TestRefererBlocklistOmittedIsEmpty checks that an omitted key blocks no
-// referer.
-func TestRefererBlocklistOmittedIsEmpty(t *testing.T) {
-	t.Parallel()
-
-	c, err := configFromYAML(t, signingKeyLine)
-	if err != nil {
-		t.Fatalf("minimal config should be valid, got error: %v", err)
-	}
-
-	if len(c.RefererBlocklist) != 0 {
-		t.Errorf("RefererBlocklist = %v, want empty", c.RefererBlocklist)
-	}
-}
-
-// TestRefererBlocklistInvalidAbortsStartup checks that an entry that is not a
-// host, or a value that is not a list of them, aborts startup with an error
-// naming the key and the entry.
-func TestRefererBlocklistInvalidAbortsStartup(t *testing.T) {
-	t.Parallel()
-
-	runAbortCases(t, []abortCase{
-		{
-			name: "entry with a scheme",
-			yaml: signingKeyLine + "referer_blocklist:\n  - https://leech.example\n",
-			wantErrSubstrings: []string{
-				keyRefererBlocklist, "https://leech.example",
-			},
-		},
-		{
-			name: "entry with a path",
-			yaml: signingKeyLine + "referer_blocklist:\n  - leech.example/page\n",
-			wantErrSubstrings: []string{
-				keyRefererBlocklist, "leech.example/page",
-			},
-		},
-		{
-			name: "wildcard entry",
-			yaml: signingKeyLine + "referer_blocklist:\n  - \"*.leech.example\"\n",
-			wantErrSubstrings: []string{
-				keyRefererBlocklist, "*.leech.example",
-			},
-		},
-		{
-			name: "entry with a port",
-			yaml: signingKeyLine + "referer_blocklist:\n  - leech.example:8080\n",
-			wantErrSubstrings: []string{
-				keyRefererBlocklist, "leech.example:8080",
-			},
-		},
-		{
-			name: "two leading dots",
-			yaml: signingKeyLine + "referer_blocklist:\n  - ..leech.example\n",
-			wantErrSubstrings: []string{
-				keyRefererBlocklist, "..leech.example",
-			},
-		},
-		{
-			name:              "dot only",
-			yaml:              signingKeyLine + "referer_blocklist:\n  - \".\"\n",
-			wantErrSubstrings: []string{keyRefererBlocklist, `"."`},
-		},
-		{
-			name:              "empty entry",
-			yaml:              signingKeyLine + "referer_blocklist:\n  - \"\"\n",
-			wantErrSubstrings: []string{keyRefererBlocklist},
-		},
-		{
-			name:              "entry not a string",
-			yaml:              signingKeyLine + "referer_blocklist:\n  - 42\n",
-			wantErrSubstrings: []string{keyRefererBlocklist, "42"},
-		},
-		{
-			name:              "null value",
-			yaml:              signingKeyLine + "referer_blocklist:\n",
-			wantErrSubstrings: []string{keyRefererBlocklist, nullValueText},
-		},
-	})
-}
-
-// TestRefererBlocklistFromEnvironment checks that PIXA_REFERER_BLOCKLIST
-// takes comma-separated entries, and that an entry in it that is not a host
-// aborts startup naming the variable and the entry.
-func TestRefererBlocklistFromEnvironment(t *testing.T) {
-	t.Setenv("PIXA_SIGNING_KEY", validTestSigningKey)
-	t.Setenv("PIXA_REFERER_BLOCKLIST", " leech.example , .hotlinker.example ")
-
-	c, err := newFromSmartConfig(nil)
-	if err != nil {
-		t.Fatalf("valid PIXA_REFERER_BLOCKLIST should load, got error: %v", err)
-	}
-
-	want := []string{"leech.example", ".hotlinker.example"}
-	if !slices.Equal(c.RefererBlocklist, want) {
-		t.Errorf("RefererBlocklist = %v, want %v", c.RefererBlocklist, want)
-	}
-
-	t.Setenv("PIXA_REFERER_BLOCKLIST", "leech.example,https://hotlinker.example")
-
-	_, err = newFromSmartConfig(nil)
-	wantStartupError(t, err, "PIXA_REFERER_BLOCKLIST", "https://hotlinker.example")
-}

internal/handlers/auth_session_internal_test.go

@@ -8,7 +8,6 @@ import (
 	"regexp"
 	"strings"
 	"testing"
-	"time"
 
 	"sneak.berlin/go/pixa/internal/imgcache"
 	"sneak.berlin/go/pixa/internal/session"
@@ -242,51 +241,3 @@ func TestGeneratePost_URLServesImage(t *testing.T) {
 
 	requireServedPhoto(t, imageRec)
 }
-
-// TestGeneratePost_URLWithTTLExpires verifies that a URL the generator page
-// makes with a ttl of one second is served by /v1/e/ at once and answers 410
-// once the ttl has passed.
-func TestGeneratePost_URLWithTTLExpires(t *testing.T) {
-	t.Parallel()
-
-	_, imageSrv := newSignedHostServer(t, slog.New(slog.DiscardHandler))
-
-	rec := generatePost(t, url.Values{
-		sourceURLField: {"https://" + signedHost + photoPath},
-		widthField:     {"50"},
-		heightField:    {"50"},
-		formatField:    {string(imgcache.FormatJPEG)},
-		ttlField:       {"1"},
-	})
-
-	if rec.Code != http.StatusOK {
-		t.Fatalf("POST /generate status = %d, want %d", rec.Code, http.StatusOK)
-	}
-
-	match := generatedURLPattern.FindStringSubmatch(rec.Body.String())
-	if match == nil {
-		t.Fatalf("generator page shows no URL: %s", rec.Body.String())
-	}
-
-	imageRec := httptest.NewRecorder()
-	imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
-		t.Context(), http.MethodGet, match[1], nil))
-
-	requireServedPhoto(t, imageRec)
-
-	// The URL keeps the time it expires in whole seconds and is served
-	// through the whole of that second, so a ttl of one second has passed
-	// for certain two seconds after the URL was made.
-	time.Sleep(2 * time.Second)
-
-	imageRec = httptest.NewRecorder()
-	imageSrv.ServeHTTP(imageRec, httptest.NewRequestWithContext(
-		t.Context(), http.MethodGet, match[1], nil))
-
-	t.Logf("GET %s after the ttl: %d %q", match[1], imageRec.Code, imageRec.Body)
-
-	if imageRec.Code != http.StatusGone {
-		t.Errorf("status after the ttl = %d, want %d",
-			imageRec.Code, http.StatusGone)
-	}
-}

internal/handlers/handlers.go

@@ -9,7 +9,6 @@ import (
 	"time"
 
 	"go.uber.org/fx"
-	"sneak.berlin/go/pixa/internal/allowlist"
 	"sneak.berlin/go/pixa/internal/config"
 	"sneak.berlin/go/pixa/internal/database"
 	"sneak.berlin/go/pixa/internal/encurl"
@@ -47,10 +46,6 @@ type Handlers struct {
 	sessMgr     *session.Manager
 	encGen      *encurl.Generator
 	csrfProtect func(http.Handler) http.Handler
-
-	// refererBlocklist matches the hosts of referer_blocklist; its IsAllowed
-	// reports whether a URL's host is on that list.
-	refererBlocklist *allowlist.HostAllowList
 }
 
 // New creates a new Handlers instance.
@@ -67,7 +62,6 @@ func New(lc fx.Lifecycle, params Params) (*Handlers, error) {
 		config:      params.Config,
 		fetcher:     params.Fetcher,
 		csrfProtect: csrfProtect,
-		refererBlocklist: allowlist.New(params.Config.RefererBlocklist),
 	}
 
 	lc.Append(fx.Hook{

internal/handlers/image.go

@@ -21,10 +21,6 @@ import (
 // /v1/image/<host>/<path>/<width>x<height>.<format>
 func (s *Handlers) HandleImage() http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		if s.refuseBlockedReferer(w, r) {
-			return
-		}
-
 		req, ok := s.parseImageRequest(w, r)
 		if !ok {
 			return
@@ -252,23 +248,6 @@ func cacheControl(expires time.Time) string {
 	return fmt.Sprintf("public, max-age=%d, immutable", int64(maxAge/time.Second))
 }
 
-// refuseBlockedReferer answers 403 with a JSON error when the request's Referer
-// names a host on referer_blocklist, and reports whether it answered. A request
-// with no Referer, or one that does not parse as a URL with a host, is not
-// refused.
-func (s *Handlers) refuseBlockedReferer(
-	w http.ResponseWriter, r *http.Request,
-) bool {
-	referer, err := url.Parse(r.Referer())
-	if err != nil || !s.refererBlocklist.IsAllowed(referer) {
-		return false
-	}
-
-	s.respondError(w, "referer blocked", http.StatusForbidden)
-
-	return true
-}
-
 // notModified sets the ETag header to etag and, when the request's
 // If-None-Match is that ETag, answers 304 Not Modified. It reports whether it
 // answered. An empty etag sets no header and never answers.

internal/handlers/imageenc.go

@@ -22,10 +22,6 @@ import (
 // browsers identify the content type.
 func (s *Handlers) HandleImageEnc() http.HandlerFunc {
 	return func(w http.ResponseWriter, r *http.Request) {
-		if s.refuseBlockedReferer(w, r) {
-			return
-		}
-
 		ctx := r.Context()
 		start := time.Now()
 

internal/handlers/referer_blocklist_internal_test.go

@@ -1,185 +0,0 @@
-package handlers
-
-import (
-	"context"
-	"log/slog"
-	"net/http"
-	"net/http/httptest"
-	"sync/atomic"
-	"testing"
-	"time"
-
-	"github.com/go-chi/chi/v5"
-	"sneak.berlin/go/pixa/internal/allowlist"
-	"sneak.berlin/go/pixa/internal/encurl"
-	"sneak.berlin/go/pixa/internal/httpfetcher"
-	"sneak.berlin/go/pixa/internal/imgcache"
-)
-
-// blockedReferer is a page on leech.example, which newRefererRoutes puts on
-// referer_blocklist.
-const blockedReferer = "https://leech.example/page.html"
-
-// countingFetcher passes each fetch on to the fetcher it holds and counts it.
-type countingFetcher struct {
-	httpfetcher.Fetcher
-
-	fetches atomic.Int32
-}
-
-// Fetch counts the fetch and passes it on.
-func (f *countingFetcher) Fetch(
-	ctx context.Context, url string,
-) (*httpfetcher.FetchResult, error) {
-	f.fetches.Add(1)
-
-	return f.Fetcher.Fetch(ctx, url)
-}
-
-// newRefererRoutes returns both image routes of a Handlers whose
-// referer_blocklist is "leech.example" and ".hotlinker.example", the
-// Handlers, and the fetcher the routes fetch through. The JPEG at photoPath
-// exists on allowlistedHost and on signedHost.
-func newRefererRoutes(t *testing.T) (http.Handler, *Handlers, *countingFetcher) {
-	t.Helper()
-
-	fetcher := &countingFetcher{
-		Fetcher: newPhotoFetcher(t, allowlistedHost, signedHost),
-	}
-
-	cache, err := imgcache.NewCache(setupTestDB(t), imgcache.CacheConfig{
-		StateDir:    t.TempDir(),
-		CacheTTL:    time.Hour,
-		NegativeTTL: 5 * time.Minute,
-	})
-	if err != nil {
-		t.Fatalf("imgcache.NewCache() error = %v", err)
-	}
-
-	svc, err := imgcache.NewService(&imgcache.ServiceConfig{
-		Cache:      cache,
-		Fetcher:    fetcher,
-		SigningKey: testSigningKey,
-		Allowlist:  []string{allowlistedHost},
-	})
-	if err != nil {
-		t.Fatalf("imgcache.NewService() error = %v", err)
-	}
-
-	encGen, err := encurl.NewGenerator(testSigningKey)
-	if err != nil {
-		t.Fatalf("encurl.NewGenerator() error = %v", err)
-	}
-
-	h := &Handlers{
-		log:    slog.New(slog.DiscardHandler),
-		imgSvc: svc,
-		encGen: encGen,
-		refererBlocklist: allowlist.New(
-			[]string{"leech.example", ".hotlinker.example"}),
-	}
-
-	r := chi.NewRouter()
-	r.Get("/v1/image/*", h.HandleImage())
-	r.Get("/v1/e/{token}/*", h.HandleImageEnc())
-
-	return r, h, fetcher
-}
-
-// getWithReferer sends a GET for target to routes with referer as its
-// Referer header, or with none when referer is empty, and returns the
-// response.
-func getWithReferer(
-	t *testing.T, routes http.Handler, target, referer string,
-) *httptest.ResponseRecorder {
-	t.Helper()
-
-	req := httptest.NewRequestWithContext(t.Context(), http.MethodGet, target, nil)
-	if referer != "" {
-		req.Header.Set("Referer", referer)
-	}
-
-	rec := httptest.NewRecorder()
-
-	routes.ServeHTTP(rec, req)
-	t.Logf("GET %s with Referer %q: %d", target, referer, rec.Code)
-
-	return rec
-}
-
-// TestRefererBlocklist verifies that both image routes refuse a request whose
-// Referer names a host on referer_blocklist with 403 and the JSON error,
-// without fetching from the upstream host, and serve a request with no
-// Referer, one that does not parse, or one naming any other host. Hosts are
-// matched as allowlist_hosts matches them.
-func TestRefererBlocklist(t *testing.T) {
-	t.Parallel()
-
-	cases := []struct {
-		name    string
-		referer string
-		want    int
-	}{
-		{"no referer", "", http.StatusOK},
-		{"unlisted host", "https://unlisted.example/page.html", http.StatusOK},
-		{"unparseable", "%zz", http.StatusOK},
-		{"listed host", blockedReferer, http.StatusForbidden},
-		{"subdomain of listed host", "https://www.leech.example/", http.StatusOK},
-		{"subdomain of dot pattern", "https://www.hotlinker.example/a.html",
-			http.StatusForbidden},
-		{"dot pattern without its dot", "https://hotlinker.example/",
-			http.StatusForbidden},
-		{"host continuing past dot pattern",
-			"https://hotlinker.example.evil.example/", http.StatusOK},
-	}
-
-	// The photo's URL on each image route.
-	photoURLs := map[string]func(t *testing.T, h *Handlers) string{
-		"plain URL": func(t *testing.T, _ *Handlers) string {
-			t.Helper()
-
-			return photoURL(allowlistedHost)
-		},
-		"encrypted URL": encPhotoURL,
-	}
-
-	for urlName, photoURLFor := range photoURLs {
-		for _, tc := range cases {
-			t.Run(urlName+", "+tc.name, func(t *testing.T) {
-				t.Parallel()
-
-				routes, h, fetcher := newRefererRoutes(t)
-
-				rec := getWithReferer(t, routes, photoURLFor(t, h), tc.referer)
-
-				if tc.want == http.StatusOK {
-					requireServedPhoto(t, rec)
-
-					return
-				}
-
-				checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
-
-				if n := fetcher.fetches.Load(); n != 0 {
-					t.Errorf("upstream fetched %d times, want 0", n)
-				}
-			})
-		}
-	}
-}
-
-// TestBlockedRefererRefusedWhenImageIsCached verifies that a request whose
-// Referer is on referer_blocklist is refused even when the image it asks for
-// is already cached, so the answer does not depend on the cache.
-func TestBlockedRefererRefusedWhenImageIsCached(t *testing.T) {
-	t.Parallel()
-
-	routes, h, _ := newRefererRoutes(t)
-
-	for _, target := range []string{photoURL(allowlistedHost), encPhotoURL(t, h)} {
-		requireServedPhoto(t, getWithReferer(t, routes, target, ""))
-
-		rec := getWithReferer(t, routes, target, blockedReferer)
-		checkErrorBody(t, rec, http.StatusForbidden, "referer blocked")
-	}
-}