Implement referer blocklist to prevent hotlinking #90

New Issue

2026-08-09T03:51:22+02:00

clawbot commented

2026-08-09 03:51:22 +02:00

Verified against main at 61f42e6. This is the "referer blacklist" P2 item in TODO.md; confirmed missing — the Referer header is read only for logging (internal/middleware/middleware.go:97) and never used for any decision.

The use case is hotlink protection: a third-party site embedding pixa-served images makes pixa pay the bandwidth and the origin pay the fetches, on someone else's traffic. Since signed URLs are shareable by design and long-lived until exp, a leaked URL embedded elsewhere is a realistic drain — and with q/fit currently unsigned (filed separately), the amplification is worse.

Naming: TODO.md says "blacklist". Per the repo's inclusive-terminology convention (established when whitelist became allowlist in #41), this should be referer_blocklist or similar. Update the TODO.md wording in the same change.

Definition of done

A referer_blocklist config key (list of host patterns, supporting the same exact/suffix matching semantics the allowlist already uses — reuse internal/allowlist's matcher rather than writing a second one). Strict parsing: an invalid entry aborts startup naming the key and the value, per repo policy.
A request whose Referer matches is refused with 403 before any upstream fetch or cache write, so a blocked request costs nothing.
Absent or unparseable Referer is treated as not-blocked (a blocklist cannot be a de facto allowlist — many legitimate clients send no referer). Document this explicitly; it is the main limitation of the feature and operators should know it is trivially bypassed by omitting the header.
Decide and document whether a blocked request should still be served from cache if already present — recommend no, refuse uniformly, so behavior does not depend on cache state.
Failing tests first: a blocked referer returns 403 and triggers no fetch; an unlisted referer succeeds; an absent referer succeeds; suffix matching behaves like the allowlist's.
README and config.example.yml document the key and its limitations. make check green.

Note

If an allowlist model would suit better than a blocklist (refuse everything except named referers), say so before implementing — for hotlink protection an allowlist is usually the more effective shape, and the blocklist is the weaker default. Flagging the choice rather than assuming TODO.md's wording settles it.

Verified against `main` at `61f42e6`. This is the "referer blacklist" P2 item in `TODO.md`; confirmed missing — the `Referer` header is read only for logging (`internal/middleware/middleware.go:97`) and never used for any decision. The use case is hotlink protection: a third-party site embedding pixa-served images makes pixa pay the bandwidth and the origin pay the fetches, on someone else's traffic. Since signed URLs are shareable by design and long-lived until `exp`, a leaked URL embedded elsewhere is a realistic drain — and with `q`/`fit` currently unsigned (filed separately), the amplification is worse. Naming: `TODO.md` says "blacklist". Per the repo's inclusive-terminology convention (established when `whitelist` became `allowlist` in #41), this should be `referer_blocklist` or similar. Update the `TODO.md` wording in the same change. ## Definition of done 1. A `referer_blocklist` config key (list of host patterns, supporting the same exact/suffix matching semantics the allowlist already uses — reuse `internal/allowlist`'s matcher rather than writing a second one). Strict parsing: an invalid entry aborts startup naming the key and the value, per repo policy. 2. A request whose `Referer` matches is refused with 403 before any upstream fetch or cache write, so a blocked request costs nothing. 3. Absent or unparseable `Referer` is treated as not-blocked (a blocklist cannot be a de facto allowlist — many legitimate clients send no referer). Document this explicitly; it is the main limitation of the feature and operators should know it is trivially bypassed by omitting the header. 4. Decide and document whether a blocked request should still be served from cache if already present — recommend no, refuse uniformly, so behavior does not depend on cache state. 5. Failing tests first: a blocked referer returns 403 and triggers no fetch; an unlisted referer succeeds; an absent referer succeeds; suffix matching behaves like the allowlist's. 6. README and `config.example.yml` document the key and its limitations. `make check` green. ## Note If an allowlist model would suit better than a blocklist (refuse everything except named referers), say so before implementing — for hotlink protection an allowlist is usually the more effective shape, and the blocklist is the weaker default. Flagging the choice rather than assuming `TODO.md`'s wording settles it.

clawbot added this to the 1.0.0 milestone 2026-08-09 03:51:22 +02:00

clawbot referenced this issue

2026-08-09 16:10:39 +02:00

Tracking: 1.0.0 sequencing and critical path #103

clawbot commented

2026-10-04 20:00:46 +02:00

Plan, checked against next at 66e71b4. The Referer header is still read only for logging.

The issue's note asks whether an allowlist of referers would suit better. TODO.md (the owner's roadmap) names a referer blocklist, so this builds that, named referer_blocklist as the issue says; an allowlist of referers can be a later unit if wanted.

A referer_blocklist setting (PIXA_REFERER_BLOCKLIST), a list of host patterns matched exactly as allowlist_hosts is: a plain host matches only itself, a pattern starting with . matches any host under it. It reuses internal/allowlist's matcher (allowlist.New and IsAllowed on the referer's URL), not a second one. Strict parsing like the other lists: an entry that is not a host aborts startup naming the setting and the entry. Empty by default.
On both image routes, a request whose Referer host matches is refused with 403 and a JSON error before anything else is done for it: no upstream fetch, no cache read or write. So a blocked request is refused the same whether or not the image is cached.
A request with no Referer, or one that does not parse as a URL with a host, is not blocked. README.md says plainly that the blocklist is easily bypassed by sending no Referer, and that it does not touch the login or generator pages.
Tests first: a blocked referer answers 403 and the upstream receives nothing; an unlisted referer, no referer and an unparseable referer are served; a .example.com pattern blocks a subdomain but not example.com.evil.net.
README.md (setting table, Routes 403, a short paragraph on what it is for and its limit) and configs/config.example.yml (or config.example.yml, wherever it is on next when you start) document it; TODO.md says blocklist.

Model: opus-5-5

Plan, checked against `next` at `66e71b4`. The `Referer` header is still read only for logging. The issue's note asks whether an allowlist of referers would suit better. `TODO.md` (the owner's roadmap) names a referer blocklist, so this builds that, named `referer_blocklist` as the issue says; an allowlist of referers can be a later unit if wanted. - A `referer_blocklist` setting (`PIXA_REFERER_BLOCKLIST`), a list of host patterns matched exactly as `allowlist_hosts` is: a plain host matches only itself, a pattern starting with `.` matches any host under it. It reuses `internal/allowlist`'s matcher (`allowlist.New` and `IsAllowed` on the referer's URL), not a second one. Strict parsing like the other lists: an entry that is not a host aborts startup naming the setting and the entry. Empty by default. - On both image routes, a request whose `Referer` host matches is refused with 403 and a JSON error before anything else is done for it: no upstream fetch, no cache read or write. So a blocked request is refused the same whether or not the image is cached. - A request with no `Referer`, or one that does not parse as a URL with a host, is not blocked. `README.md` says plainly that the blocklist is easily bypassed by sending no `Referer`, and that it does not touch the login or generator pages. - Tests first: a blocked referer answers 403 and the upstream receives nothing; an unlisted referer, no referer and an unparseable referer are served; a `.example.com` pattern blocks a subdomain but not `example.com.evil.net`. - `README.md` (setting table, Routes 403, a short paragraph on what it is for and its limit) and `configs/config.example.yml` (or `config.example.yml`, wherever it is on `next` when you start) document it; `TODO.md` says blocklist. Model: opus-5-5

clawbot referenced this issue from a commit

2026-10-04 20:09:18 +02:00

Refuse image requests whose Referer is on referer_blocklist (closes #90)

clawbot referenced a pull request that will close this issue

2026-10-04 20:24:51 +02:00

Refuse image requests whose Referer is on referer_blocklist (closes #90) #197

clawbot commented

2026-10-04 20:25:10 +02:00