Refuse image requests whose Referer is on referer_blocklist (closes #90) #197

Merged
clawbot merged 7 commits from issue-90-referer-blocklist into next 2026-10-04 22:24:50 +02:00
7 Commits
Author SHA1 Message Date
clawbot 6259832fb3 Allow underscores in allowlist_hosts and referer_blocklist host names
check / check (push) Failing after 1s
The shared entry check refused host names with an underscore, though
both lists can match them. The check's comment now says that what it
refuses can never match a host name that resolves.

Model: opus-5-5
2026-10-04 19:46:10 +00:00
clawbot 29ff9e4463 Add failing tests for host names with an underscore
Both allowlist_hosts and referer_blocklist should accept a host name
with an underscore, such as my_bucket.example.com, since pixa fetches
from such hosts and pages are served from them.

Model: opus-5-5
2026-10-04 19:46:10 +00:00
clawbot a2effe1e27 Refuse host entries that are not a host name or an IP address
check / check (push) Failing after 3s
An entry of allowlist_hosts or referer_blocklist that is neither a host
name (letters, digits, hyphens and dots, with at most one leading dot)
nor an IP address now aborts startup naming the setting and the entry,
so a `*.` wildcard or a port no longer loads and silently matches
nothing. README.md, configs/config.example.yml and the TODO.md entry now
say the Referer check comes before the signature, the cache and the
upstream fetch, since maintenance mode answers first; the example config
says the list does not cover the login and generator pages.

Model: opus-5-5
2026-10-04 19:09:02 +00:00
clawbot e1737f497c Add failing tests for host entries that can never match
A `*.` entry, an entry with a port and one with two leading dots must
abort startup for referer_blocklist, and the first two for
allowlist_hosts; IPv4 and IPv6 address entries must still load.

Model: opus-5-5
2026-10-04 19:08:30 +00:00
clawbot 5ae17b6361 Read referer_blocklist through strictLoader and fix test lint
newFromSmartConfig went over the line limit, so referer_blocklist is now
read through strictLoader, like the other typed settings, instead of its
own parse step. The tests stop repeating string literals that goconst
counts: the environment test uses other hosts, two case names change, and
the handler test names each image route's URL instead of its path.

Model: opus-5-5
2026-10-04 19:08:30 +00:00
clawbot c9a8b926db Refuse image requests whose Referer is on referer_blocklist (closes #90)
A new setting, referer_blocklist (PIXA_REFERER_BLOCKLIST), lists hosts
written and matched as allowlist_hosts are, with the same matcher and the
same entry check; a bad entry aborts startup naming the setting and the
entry. Both image routes check the Referer first and answer 403 with the
JSON error, so a blocked request fetches nothing and is refused whether or
not the image is cached. No Referer, or one that does not parse as a URL
with a host, is served; README.md and config.example.yml say this makes the
list easy to get around. The CIDR-list entry reader is renamed listEntries
now that host lists use it too.

Model: opus-5-5
2026-10-04 19:08:30 +00:00
clawbot b2c6bc91d0 Add failing tests for referer_blocklist
Both image routes must refuse a request whose Referer names a host on
referer_blocklist with 403 and the JSON error, without fetching from the
upstream host and whether or not the image is cached, and must serve a
request with no Referer, one that does not parse, or one naming another
host. Hosts match as allowlist_hosts matches them. The config tests check
the list is read from the file and from PIXA_REFERER_BLOCKLIST, and that an
entry that is not a host aborts startup naming the setting and the entry.
These do not compile until the setting exists.

Model: opus-5-5
2026-10-04 19:08:30 +00:00