Compare commits

...
Author SHA1 Message Date
sneak b157359910 rproxy: a page of its own for a refused visitor
check / check (push) Failing after 1m48s
A deny of 403 from waf is answered with refused.html, a deny of 429
with ratelimited.html, each as text/html with waf's other headers kept.
Both pages are made once at start and held in memory. With
SWW_RPROXY_CONTACT_EMAIL set, their "contact the site's operator" is a
mailto link to it; the value is taken as given.
2026-11-19 21:12:31 -08:00
sneak 5dfa47c97b Tests: rproxy gives a refused visitor a page of its own
A deny of 403 from waf gets rproxy's refused page, a deny of 429 its
rate-limited page, each as text/html with waf's other headers kept. With
SWW_RPROXY_CONTACT_EMAIL set, the pages' sentence about the operator is a
mailto link. Every other answer from waf still goes to the visitor as waf
gave it. The two pages are beside the code, as the tests read them.
2026-11-19 21:10:10 -08:00
sneak 8219664ad1 Add a docker-compose.yml template to the example app
check / check (push) Successful in 7m2s
A three-container setup to copy into an app's repository: the app built
from its own Dockerfile, rproxy in front of it behind traefik, and waf
with a volume for its state files. The settings most often changed are
commented out, each with a value that differs from its default and a
note on what it does.
2026-11-19 20:45:06 -08:00
sneak 58b9985b6d docker cache should be enabled 2026-11-19 19:55:24 -08:00
sneak 677df6a235 prefix docker image name
check / check (push) Failing after 17m53s
2026-11-19 19:48:57 -08:00
sneak 8188602a09 unnecessary llm crap
check / check (push) Failing after 1m40s
2026-11-19 19:47:11 -08:00
sneak ee550be738 Merge branch 'main' into next 2026-11-19 19:41:49 -08:00
clawbot 493aac02a1 waf: the server's types and constructor in waf.go, active_ban in its own file (closes #435)
check / check (push) Failing after 4m6s
Moves only. waf.go now holds the package comment, Params, Server, New,
newHandler, handler and ServeHTTP. The code that answers questions and
takes reports moves to questionsandreports.go, and activeBan to
activeban.go. refusedWhateverTheAnswer moves to lookup.go, beside its
only caller. proxy.go is gone. The only changed lines are imports and
the file comment of bans.go, split between it and activeban.go.

Model: opus-5-5
2026-11-20 03:26:58 +01:00
sneak 51245804d0 tag image
check / check (push) Successful in 4m12s
2026-11-18 18:52:48 -08:00
20 changed files with 1208 additions and 673 deletions

No files matched your search

+1
View File
@@ -308,6 +308,7 @@ Every setting the code reads, with its default:
| `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. | | `SWW_RPROXY_MAX_RESPONSE_SIZE` | `1G` | The largest response body, a size or `off`. The visitor gets `502` in place of a larger one, or it is cut off if it has already started. |
| `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. | | `SWW_RPROXY_MAX_REQUEST_TIME_MS` | `60000` | Two time limits, each this long, in milliseconds or `off`: one for the visitor to send its request line and headers, then one for the whole request to reach the app. |
| `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. | | `SWW_RPROXY_MAX_RESPONSE_TIME_MS` | `1800000` | How long the answer may take, from the end of the request to its last byte reaching the visitor, in milliseconds or `off`. |
| `SWW_RPROXY_CONTACT_EMAIL` | unset | The email address the refusal pages link to. |
| `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. | | `SWW_WAF_LISTEN_ADDR` | `:8080` | Where `waf` listens; leave the host part empty. |
| `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. | | `SWW_WAF_INSTANCE_NAME` | the container's host name | The name in every line `waf` logs, and in every alert and metric, such as `host1/waf`. |
| `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. | | `SWW_WAF_MODE` | `enforce` | `observe` refuses nothing and makes no ban; each log line says what `enforce` would have done. |
+3 -2
View File
@@ -2431,8 +2431,9 @@ but the last below carries `X-Sww-Disposition`.
carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the carries `X-Client-ASN` and `X-Client-Country`, which `rproxy` adds to the
request it passes on. A header whose value is unknown is left out. request it passes on. A header whose value is unknown is left out.
- `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to - `X-Sww-Disposition: deny`, with the status, headers and body the visitor is to
get. `rproxy` gives them to the visitor, less this header. The status is one get. `rproxy` gives them to the visitor, less this header; a `403` or `429`
of: gets `rproxy`'s own page as its body, linking `SWW_RPROXY_CONTACT_EMAIL` when
set. The status is one of:
- `SWW_WAF_BAN_RESPONSE`, for a refusal; - `SWW_WAF_BAN_RESPONSE`, for a refusal;
- `403`, for a `block` rule or the Core Rule Set; - `403`, for a `block` rule or the Core Rule Set;
- `414` or `431`, for a request over the header limits; - `414` or `431`, for a request over the header limits;
+217
View File
@@ -0,0 +1,217 @@
services:
exampleapp:
# Your app, built from the Dockerfile in this directory, listening
# on port 8081. Publish no port of it: it is reached through rproxy.
build: .
rproxy:
image: git.eeqj.de/sneak/smallwebwaf:latest
environment:
SWW_RPROXY_WAF_BACKEND: http://waf:8080
SWW_RPROXY_UPSTREAM_URL: http://exampleapp:8081
# The address the refusal pages link to.
# SWW_RPROXY_CONTACT_EMAIL: ops@example.com
# The netblocks whose X-Forwarded-For rproxy believes. By default
# the private ranges, which cover a TLS-terminating proxy on the
# Docker network. A proxy elsewhere needs its address added; a
# list you give replaces the default, so repeat the private
# ranges if they are still wanted. Set but empty, rproxy believes
# no forwarded header.
# SWW_RPROXY_TRUSTED_PROXIES: 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,203.0.113.5,2001:db8:1::/64
# By default rproxy passes a request to the app while waf gives
# no answer. Uncomment this to answer 503 instead.
# SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION: deny
# The largest request body (default 100M) and response body
# (default 1G): a size with K, M or G, or `off`. Raise them for
# uploads and downloads larger than that, such as git pushes and
# clones or image layers.
# SWW_RPROXY_MAX_REQUEST_SIZE: 2G
# SWW_RPROXY_MAX_RESPONSE_SIZE: 4G
# How long a request may take to reach the app (default 60000,
# one minute) and how long its answer may take to reach the
# visitor (default 1800000, 30 minutes), in milliseconds or
# `off`. Raise the first for slow uploads, the second for long
# downloads.
# SWW_RPROXY_MAX_REQUEST_TIME_MS: "600000"
# SWW_RPROXY_MAX_RESPONSE_TIME_MS: "7200000"
# traefik terminates TLS and forwards to rproxy:8080 over its own
# network, which the `networks` block at the end of this file names.
# No port is published.
networks:
- default
- traefik
labels:
traefik.enable: "true"
# rproxy is on two networks; traefik reaches it on this one.
traefik.docker.network: traefik
traefik.http.routers.exampleapp.rule: Host(`app.example.com`)
traefik.http.routers.exampleapp.entrypoints: websecure
traefik.http.routers.exampleapp.tls.certresolver: letsencrypt
traefik.http.services.exampleapp.loadbalancer.server.port: "8080"
waf:
image: git.eeqj.de/sneak/smallwebwaf:latest
environment:
SWW_COMMAND: waf
# `observe` refuses nothing and makes no ban; each log line says
# what `enforce` (the default) would have done. Start here to
# see what the defaults would refuse on your site.
# SWW_WAF_MODE: observe
# waf's own messages below `info` (the default), while finding
# out why something is refused.
# SWW_WAF_LOG_LEVEL: debug
# Netblocks whose clients skip every check, such as monitoring.
# SWW_WAF_ALLOW_NETS: 192.0.2.10,2001:db8:2::/48
# Netblocks whose clients are refused.
# SWW_WAF_DENY_NETS: 198.51.100.128/25,2001:db8:3::/48
# Netblocks exempt from rate limiting and the byte limits, such
# as the office and an automation host that calls an API. Their
# requests are still checked for attacks.
# SWW_WAF_RATE_LIMIT_EXEMPT_NETS: 203.0.113.0/24,198.51.100.7
# Refuse every request from a country, by its two-letter code.
# This refuses North Korea, Cuba, Russia and Iran. A client the
# lookup cannot place gets through; see SWW_WAF_LOOKUP_SOURCE.
# SWW_WAF_TAG_DENY: country=KP,country=CU,country=RU,country=IR
# Or from a network, by its AS number: here two hosting networks
# that send only scrapers, M247 (AS9009) and WorldStream
# (AS49981). Countries and networks go in the one list together.
# SWW_WAF_TAG_DENY: asn=9009,asn=49981,country=KP,country=RU
# Or let only these countries through; a client the lookup
# cannot place is refused too.
# SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES: us,de
# Address lists that each tag their clients' requests, as
# name=URL pairs; this one tags Tor exit nodes. A tag is refused
# by adding its name to SWW_WAF_TAG_DENY, such as
# `torexit,country=KP`, or given a share of the limits in
# SWW_WAF_TAG_LIMIT_PERCENT, such as `torexit:25`.
# SWW_WAF_TAG_LISTS: torexit=https://check.torproject.org/torbulkexitlist
# A percentage, from 1 to 99, of every limit for a tag: here
# half for DigitalOcean (AS14061), a quarter for OVH (AS16276)
# and a quarter for China. The lowest of a client's tags applies.
# SWW_WAF_TAG_LIMIT_PERCENT: asn=14061:50,asn=16276:25,country=CN:25
# The most requests a client may make in a minute, an hour and a
# day (defaults 120, 1500 and 6000), a count or `off`; the next
# one bans it. Lower them for a small site, raise them for a
# busy API.
# SWW_WAF_RATE_LIMIT_PER_MINUTE: "60"
# SWW_WAF_RATE_LIMIT_PER_HOUR: "600"
# SWW_WAF_RATE_LIMIT_PER_DAY: "3000"
# Path prefixes exempt from rate limiting and the byte limits,
# such as static assets that one page loads many of.
# SWW_WAF_RATE_LIMIT_EXEMPT_PATHS: /assets/,/static/
# Paths only scanners ask for; one request bans the client. Pick
# paths your app never serves.
# SWW_WAF_TRAP_PATHS: /wp-login.php,/xmlrpc.php,/wp-admin/
# How a refused client is answered: 403 (the default), 429 or
# `close`, which drops the connection.
# SWW_WAF_BAN_RESPONSE: "429"
# The ban for a first broken limit (default 1h) and for a first
# clear sign of attack (default 7d): a duration with m, h or d.
# A repeat within SWW_WAF_LIMIT_BAN_REPEAT_WINDOW (default 24h)
# gets a ban three times as long.
# SWW_WAF_LIMIT_BAN_DURATION: 30m
# SWW_WAF_ATTACK_BAN_DURATION: 30d
# The Core Rule Set, which finds attacks in requests. `block`
# (the default) refuses a match, `detect` only logs and alerts
# it, `off` runs no rule. Use `detect` while finding out what it
# would refuse of your app's ordinary requests.
# SWW_WAF_CORE_RULE_SET_MODE: detect
# Path prefixes the Core Rule Set does not inspect, for a part of
# the app that it keeps matching, such as an editor that posts
# HTML or code.
# SWW_WAF_CORE_RULE_SET_EXEMPT_PATHS: /api/editor/,/admin/
# The apps each hostname serves, as hostname:app pairs separated
# by semicolons, so that each hostname gets its own apps' fixes
# to the attack detection and no other's. "Attack detection" in
# the README lists the apps.
# SWW_WAF_APPS: git.example.com:gitea;blog.example.com:wordpress
# Blocklists of addresses and netblocks, text files fetched once
# a day, whose clients are refused.
# SWW_WAF_BLOCKLIST_URLS: https://www.spamhaus.org/drop/drop.txt,https://www.spamhaus.org/drop/dropv6.txt
# With the key of an AbuseIPDB account, a client that has
# committed an offence is checked there in the background; one
# scored 75 or more (SWW_WAF_ABUSEIPDB_MIN_SCORE) gets what
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION says: a quarter of every limit
# by default, or `deny`. At most 900 checks a day
# (SWW_WAF_ABUSEIPDB_DAILY_BUDGET), under the free plan's 1000.
# SWW_WAF_ABUSEIPDB_KEY: api-key-.....
# SWW_WAF_DNSBL_ABUSEIPDB_ACTION: deny
# By default waf looks each client's address up at GeoJS, a web
# service, for its country and AS number. Set `off`, it makes no
# external requests for lookups. Every setting at its default
# works without them; one that needs a lookup, such as a country
# or ASN setting above, then stops the start. Set `file`, it
# reads an IPinfo Lite database instead, mounted by its
# directory as under `volumes` below, so that a replaced file is
# seen.
# SWW_WAF_LOOKUP_SOURCE: "off"
# SWW_WAF_LOOKUP_SOURCE: file
# SWW_WAF_LOOKUP_DB_PATH: /var/lib/ipinfo/ipinfo_lite.mmdb
# Alerts about bans, attacks and failures, to an ntfy topic, a
# Slack incoming webhook or any webhook. The ntfy token is the
# access token ntfy issued for the topic.
# SWW_WAF_ALERT_NTFY_URL: https://ntfy.sh/my-alerts
# SWW_WAF_ALERT_NTFY_TOKEN: hunter2
# SWW_WAF_ALERT_SLACK_WEBHOOK_URL: https://hooks.slack.com/services/...
# Bearer tokens you make up, at least 32 characters each. The
# admin token switches on the ban and client endpoints, through
# which a ban is lifted, your own included, as
# docs/admin-endpoints.md describes. The metrics token switches
# on /_sww/metrics.
# SWW_WAF_ADMIN_TOKEN: hunter2
# SWW_WAF_METRICS_TOKEN: hunter2
# Any setting can be read from a file instead, by adding _FILE
# to its name: the file's contents are the value, less one
# newline at the end. That keeps a secret out of this file and
# out of `docker inspect`. Mount the files as under `volumes`
# below, and set each setting in one form only.
# SWW_WAF_ALERT_NTFY_TOKEN_FILE: /etc/sww/tokens/ntfy
# SWW_WAF_ADMIN_TOKEN_FILE: /etc/sww/tokens/admin
# SWW_WAF_METRICS_TOKEN_FILE: /etc/sww/tokens/metrics
# SWW_WAF_ABUSEIPDB_KEY_FILE: /etc/sww/tokens/abuseipdb
volumes:
# waf's state files: its clients and bans, kept across restarts.
- waf-state:/var/lib/sww
# Token files, each owned by uid 65532 (the sww user) with mode
# 0400, for the _FILE settings above.
# - /srv/sww/tokens:/etc/sww/tokens:ro
# The IPinfo Lite database, for SWW_WAF_LOOKUP_SOURCE=file.
# - /srv/ipinfo:/var/lib/ipinfo:ro
volumes:
waf-state:
# traefik's network, made outside this file.
networks:
traefik:
external: true
+2
View File
@@ -483,6 +483,8 @@ start, and an invalid value stops the start. Any can be given as a file, as a
answer `rproxy` gives itself always has at least 10 seconds to be sent, and so answer `rproxy` gives itself always has at least 10 seconds to be sent, and so
do the status and headers of the app's answer. A time in milliseconds or do the status and headers of the app's answer. A time in milliseconds or
`off`. `off`.
- `SWW_RPROXY_CONTACT_EMAIL` (default unset): the email address the refusal
pages link to.
- `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as - `SWW_RPROXY_LISTEN_ADDR` (default `:8080`): where `rproxy` listens, such as
`:9000`. `:9000`.
- `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An - `SWW_RPROXY_UPSTREAM_URL` (default `http://127.0.0.1:8081`): the app. An
+4
View File
@@ -46,6 +46,9 @@ type RProxy struct {
// (SWW_RPROXY_MAX_RESPONSE_TIME_MS). // (SWW_RPROXY_MAX_RESPONSE_TIME_MS).
MaxRequestTime time.Duration MaxRequestTime time.Duration
MaxResponseTime time.Duration MaxResponseTime time.Duration
// ContactEmail is the address the pages a refused visitor gets link to,
// or "" for no link (SWW_RPROXY_CONTACT_EMAIL). It is taken as given.
ContactEmail string
// settings are the values read, as given or by default, for the log line // settings are the values read, as given or by default, for the log line
// at start. // at start.
@@ -108,6 +111,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
"SWW_RPROXY_MAX_RESPONSE_TIME_MS", "SWW_RPROXY_MAX_RESPONSE_TIME_MS",
"1800000", "1800000",
), ),
ContactEmail: env.value("SWW_RPROXY_CONTACT_EMAIL", ""),
} }
if env.err != nil { if env.err != nil {
+2
View File
@@ -25,6 +25,7 @@ const (
rproxyMaxResponse = "SWW_RPROXY_MAX_RESPONSE_SIZE" rproxyMaxResponse = "SWW_RPROXY_MAX_RESPONSE_SIZE"
rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS" rproxyRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS" rproxyAnswerTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
rproxyContactEmail = "SWW_RPROXY_CONTACT_EMAIL"
backendURL = "http://waf:8080" backendURL = "http://waf:8080"
rproxyNoAnswerAddr = ":8080" rproxyNoAnswerAddr = ":8080"
rproxyNoAnswerApp = "http://127.0.0.1:8081" rproxyNoAnswerApp = "http://127.0.0.1:8081"
@@ -250,6 +251,7 @@ func TestRProxySettingsAreLoggedByName(t *testing.T) {
rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, wafBackendTimeout, rproxyListen, rproxyUpstream, rproxyTrusted, wafBackend, wafBackendTimeout,
rproxyNoAnswer, rproxyNoAnswer,
rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime, rproxyMaxRequest, rproxyMaxResponse, rproxyRequestTime, rproxyAnswerTime,
rproxyContactEmail,
} { } {
if !slices.Contains(names, name) { if !slices.Contains(names, name) {
t.Errorf("%s is not logged, only %v", name, names) t.Errorf("%s is not logged, only %v", name, names)
+71
View File
@@ -0,0 +1,71 @@
package rproxy
import (
"embed"
"html"
"net/http"
"strings"
)
// pageFiles are the pages a refused visitor gets, as HTML with nothing
// external: refused.html for a 403, ratelimited.html for a 429.
//
//go:embed refused.html ratelimited.html
var pageFiles embed.FS
// contactSentence is the sentence of each page that becomes a link to
// SWW_RPROXY_CONTACT_EMAIL when that is set.
const contactSentence = "contact the site's operator"
// htmlContentType is the Content-Type of the pages.
const htmlContentType = "text/html; charset=utf-8"
// blockPages are the pages a refused visitor gets, ready to send, made once
// at start.
type blockPages struct {
refused []byte
rateLimited []byte
}
// newBlockPages makes the pages. With contactEmail set, each page's
// contactSentence is a mailto link to it.
func newBlockPages(contactEmail string) blockPages {
return blockPages{
refused: readPage("refused.html", contactEmail),
rateLimited: readPage("ratelimited.html", contactEmail),
}
}
// readPage reads the page in name from pageFiles, linking its
// contactSentence to contactEmail unless that is "". The files are part of
// the binary, so a missing one is a bug, and a panic.
func readPage(name, contactEmail string) []byte {
contents, err := pageFiles.ReadFile(name)
if err != nil {
panic(err)
}
if contactEmail == "" {
return contents
}
link := `<a href="mailto:` + html.EscapeString(contactEmail) + `">` +
contactSentence + `</a>`
return []byte(
strings.Replace(string(contents), contactSentence, link, 1),
)
}
// forStatus returns the page for a refusal with status, and whether there is
// one: the rate-limited page for a 429, the refused page for a 403.
func (p blockPages) forStatus(status int) ([]byte, bool) {
switch status {
case http.StatusForbidden:
return p.refused, true
case http.StatusTooManyRequests:
return p.rateLimited, true
default:
return nil, false
}
}
+27
View File
@@ -0,0 +1,27 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="noindex">
<title>Too many requests</title>
<style>
:root{color-scheme:light dark;--fg:#1a1a1a;--muted:#5c5c5c;--bg:#f6f6f4}
@media(prefers-color-scheme:dark){:root{--fg:#ebebe8;--muted:#9a9a96;--bg:#161616}}
html{background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:1.5rem}
main{max-width:34rem;width:100%}
p.code{margin:0 0 .5rem;color:var(--muted);font-size:.8rem;letter-spacing:.08em}
h1{margin:0 0 1rem;font-size:1.6rem;font-weight:600;line-height:1.25}
p{margin:0 0 1rem}
</style>
</head>
<body>
<main>
<p class="code">429</p>
<h1>Too many requests</h1>
<p>Your address has sent more requests than this site allows in a short time. Wait a while, then try again.</p>
<p>If you think this is a mistake, contact the site's operator.</p>
</main>
</body>
</html>
+27
View File
@@ -0,0 +1,27 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="noindex">
<title>Request refused</title>
<style>
:root{color-scheme:light dark;--fg:#1a1a1a;--muted:#5c5c5c;--bg:#f6f6f4}
@media(prefers-color-scheme:dark){:root{--fg:#ebebe8;--muted:#9a9a96;--bg:#161616}}
html{background:var(--bg);color:var(--fg);font:16px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
body{margin:0;min-height:100vh;display:flex;align-items:center;justify-content:center;padding:1.5rem}
main{max-width:34rem;width:100%}
p.code{margin:0 0 .5rem;color:var(--muted);font-size:.8rem;letter-spacing:.08em}
h1{margin:0 0 1rem;font-size:1.6rem;font-weight:600;line-height:1.25}
p{margin:0 0 1rem}
</style>
</head>
<body>
<main>
<p class="code">403</p>
<h1>Request refused</h1>
<p>This site did not accept your request.</p>
<p>If you think this is a mistake, contact the site's operator.</p>
</main>
</body>
</html>
+3
View File
@@ -92,6 +92,7 @@ func New(params Params) *Server {
Path: wafprotocol.ReportPath, Path: wafprotocol.ReportPath,
}).String(), }).String(),
breakerClock: time.Now, breakerClock: time.Now,
pages: newBlockPages(params.Config.ContactEmail),
} }
s := &Server{h: h} s := &Server{h: h}
@@ -156,6 +157,8 @@ type handler struct {
breakerClock func() time.Time breakerClock func() time.Time
// reports are the goroutines sending reports to waf. // reports are the goroutines sending reports to waf.
reports reportGoroutines reports reportGoroutines
// pages are the pages a refused visitor gets.
pages blockPages
} }
// newTransport returns what carries requests to the app. It never goes through // newTransport returns what carries requests to the app. It never goes through
+181 -30
View File
@@ -11,6 +11,7 @@ import (
"net" "net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"os"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -52,8 +53,56 @@ const (
maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE" maxResponseSize = "SWW_RPROXY_MAX_RESPONSE_SIZE"
maxRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS" maxRequestTime = "SWW_RPROXY_MAX_REQUEST_TIME_MS"
maxResponseTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS" maxResponseTime = "SWW_RPROXY_MAX_RESPONSE_TIME_MS"
contactEmail = "SWW_RPROXY_CONTACT_EMAIL"
) )
// The pages rproxy gives a refused visitor in place of waf's body.
const (
// refusedPageFile and rateLimitedPageFile are the pages' files, beside
// the code.
refusedPageFile = "refused.html"
rateLimitedPageFile = "ratelimited.html"
// contactSentence is the sentence of each page that becomes a link to
// SWW_RPROXY_CONTACT_EMAIL.
contactSentence = "contact the site's operator"
htmlContentType = "text/html; charset=utf-8"
)
// page is the page in file, as rproxy serves it without a contact email.
func page(t *testing.T, file string) string {
t.Helper()
//nolint:gosec // a page beside the code
contents, err := os.ReadFile(file)
if err != nil {
t.Fatalf("read the page: %v", err)
}
return string(contents)
}
// wantPage checks that got is status with the page in file as its body, and
// the headers of a page.
func wantPage(t *testing.T, got answered, status int, file string) {
t.Helper()
if got.status != status || got.body != page(t, file) {
t.Errorf(
"answered %d %q, want %d and %s",
got.status,
got.body,
status,
file,
)
}
if got.header.Get("Content-Type") != htmlContentType ||
got.header.Get("X-Content-Type-Options") != "nosniff" ||
got.header.Get("Cache-Control") != "no-store" {
t.Errorf("the page's headers are %v", got.header)
}
}
// The headers the tests set and look at. // The headers the tests set and look at.
const ( const (
dispositionHeader = wafprotocol.DispositionHeader dispositionHeader = wafprotocol.DispositionHeader
@@ -306,6 +355,20 @@ func deny(w http.ResponseWriter, _ *http.Request) {
_, _ = io.WriteString(w, deniedBody) _, _ = io.WriteString(w, deniedBody)
} }
// answerAs answers a question with status and body, with disposition unless
// it is "", and a header of its own.
func answerAs(disposition string, status int, body string) http.HandlerFunc {
return func(w http.ResponseWriter, _ *http.Request) {
if disposition != "" {
w.Header().Set(dispositionHeader, disposition)
}
w.Header().Set(fromWAFHeader, addedHeaderValue)
w.WriteHeader(status)
_, _ = io.WriteString(w, body)
}
}
// startApp starts app as the app rproxy passes requests to. // startApp starts app as the app rproxy passes requests to.
func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server { func startApp(t *testing.T, app http.HandlerFunc) *httptest.Server {
t.Helper() t.Helper()
@@ -767,7 +830,7 @@ func TestRemovesTheVisitorsLookupHeadersWhenWAFGivesNone(t *testing.T) {
} }
} }
func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) { func TestDenyGivesTheVisitorTheRefusedPageWithWAFsHeaders(t *testing.T) {
t.Parallel() t.Parallel()
standIn := startWAF(t, deny) standIn := startWAF(t, deny)
@@ -777,9 +840,10 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
addr, out := startRProxy(t, standIn.URL, app.URL, nil) addr, out := startRProxy(t, standIn.URL, app.URL, nil)
got := get(t, addr, "/") got := get(t, addr, "/")
if got.status != http.StatusForbidden || got.body != deniedBody || wantPage(t, got, http.StatusForbidden, refusedPageFile)
got.header.Get(fromWAFHeader) != addedHeaderValue {
t.Errorf("answered %+v, want waf's answer", got) if got.header.Get(fromWAFHeader) != addedHeaderValue {
t.Errorf("answered %+v, want waf's headers", got)
} }
if len(got.header.Values(dispositionHeader)) != 0 { if len(got.header.Values(dispositionHeader)) != 0 {
@@ -788,7 +852,7 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
report := standIn.report(t) report := standIn.report(t)
if report.Status != http.StatusForbidden || if report.Status != http.StatusForbidden ||
report.ResponseBytes != int64(len(deniedBody)) || report.ResponseBytes != int64(len(page(t, refusedPageFile))) ||
report.DurationUpstreamTotalMS != 0 { report.DurationUpstreamTotalMS != 0 {
t.Errorf("report %+v, of a request denied", report) t.Errorf("report %+v, of a request denied", report)
} }
@@ -800,7 +864,7 @@ func TestDenyGivesTheVisitorWAFsAnswer(t *testing.T) {
} }
} }
func TestADeniedHeadKeepsWAFsContentLength(t *testing.T) { func TestADeniedHeadGivesThePagesContentLength(t *testing.T) {
t.Parallel() t.Parallel()
standIn := startWAF(t, deny) standIn := startWAF(t, deny)
@@ -808,13 +872,13 @@ func TestADeniedHeadKeepsWAFsContentLength(t *testing.T) {
addr, _ := startRProxy(t, standIn.URL, app.URL, nil) addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody)) got := visit(t, newRequest(t, http.MethodHead, addr, "/", http.NoBody))
if got.status != http.StatusForbidden || if got.status != http.StatusForbidden || got.body != "" ||
got.header.Get( got.header.Get(
contentLengthHeader, contentLengthHeader,
) != strconv.Itoa( ) != strconv.Itoa(
len(deniedBody), len(page(t, refusedPageFile)),
) { ) {
t.Errorf("answered %+v, want waf's 403 and its length", got) t.Errorf("answered %+v, want waf's 403 and the page's length", got)
} }
} }
@@ -823,11 +887,13 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
const host = " HTTP/1.1\r\nHost: site.example\r\n\r\n" const host = " HTTP/1.1\r\nHost: site.example\r\n\r\n"
refusedPage := page(t, refusedPageFile)
for _, tc := range []struct { for _, tc := range []struct {
name, request, body string name, request, body string
status int status int
}{ }{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden}, {"a deny", "GET /" + host, refusedPage, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest}, {"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK}, {"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
} { } {
@@ -899,6 +965,83 @@ func TestRProxysOwnAnswerIsPlainText(t *testing.T) {
} }
} }
func TestADenyOf429GivesTheVisitorTheRateLimitedPage(t *testing.T) {
t.Parallel()
standIn := startWAF(t, func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(dispositionHeader, dispositionDeny)
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
w.WriteHeader(http.StatusTooManyRequests)
_, _ = io.WriteString(w, deniedBody)
})
app := startApp(t, greet)
addr, _ := startRProxy(t, standIn.URL, app.URL, nil)
wantPage(
t,
get(t, addr, "/"),
http.StatusTooManyRequests,
rateLimitedPageFile,
)
}
func TestThePagesLinkTheContactEmail(t *testing.T) {
t.Parallel()
const email = "ops@example.com"
link := `<a href="mailto:` + email + `">` + contactSentence + `</a>`
for _, tc := range []struct {
file string
status int
}{
{refusedPageFile, http.StatusForbidden},
{rateLimitedPageFile, http.StatusTooManyRequests},
} {
t.Run(tc.file, func(t *testing.T) {
t.Parallel()
standIn := startWAF(
t,
func(w http.ResponseWriter, _ *http.Request) {
w.Header().Set(dispositionHeader, dispositionDeny)
w.WriteHeader(tc.status)
},
)
app := startApp(t, greet)
addr, _ := startRProxy(
t,
standIn.URL,
app.URL,
map[string]string{contactEmail: email},
)
got := get(t, addr, "/")
want := strings.Replace(
page(t, tc.file),
contactSentence,
link,
1,
)
if got.status != tc.status || got.body != want {
t.Errorf(
"answered %d %q, want %d %q",
got.status,
got.body,
tc.status,
want,
)
}
if !strings.Contains(page(t, tc.file), contactSentence) {
t.Errorf("%s has no %q to link", tc.file, contactSentence)
}
})
}
}
func TestCloseClosesTheConnectionWithoutAnAnswer(t *testing.T) { func TestCloseClosesTheConnectionWithoutAnAnswer(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1017,17 +1160,28 @@ func noAnswers() map[string]func(t *testing.T) string {
func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) { func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
t.Parallel() t.Parallel()
const unknownBody = "who knows\n"
refusedPage := page(t, refusedPageFile)
rateLimitedPage := page(t, rateLimitedPageFile)
// want is the body the visitor gets: waf's, but for a refusal, which gets
// rproxy's page for its status.
for _, tc := range []struct { for _, tc := range []struct {
name, disposition, body string name, disposition, body, want string
status int status int
}{ }{
{"a refusal", dispositionDeny, deniedBody, http.StatusForbidden}, {"a refusal", dispositionDeny, deniedBody, refusedPage, http.StatusForbidden},
{"a refusal, 429", dispositionDeny, deniedBody, http.StatusTooManyRequests}, {
{"waf's 400", "", "refused\n", http.StatusBadRequest}, "a refusal, 429", dispositionDeny, deniedBody, rateLimitedPage,
{"a 500", "", "broken\n", http.StatusInternalServerError}, http.StatusTooManyRequests,
{"a 503 denied", dispositionDeny, "busy\n", http.StatusServiceUnavailable}, },
{"no header", "", "who knows\n", http.StatusOK}, {"waf's 400", "", "refused\n", "refused\n", http.StatusBadRequest},
{"an unknown disposition", "maybe", "who knows\n", http.StatusOK}, {"a 500", "", "broken\n", "broken\n", http.StatusInternalServerError},
{"a 503 denied", dispositionDeny, "busy\n", "busy\n", http.StatusServiceUnavailable},
{"a 403 without a disposition", "", "nope\n", "nope\n", http.StatusForbidden},
{"no header", "", unknownBody, unknownBody, http.StatusOK},
{"an unknown disposition", "maybe", unknownBody, unknownBody, http.StatusOK},
} { } {
for _, setting := range []string{dispositionAllow, dispositionDeny} { for _, setting := range []string{dispositionAllow, dispositionDeny} {
subtest := tc.name + " with " + noAnswerSetting + "=" + setting subtest := tc.name + " with " + noAnswerSetting + "=" + setting
@@ -1036,15 +1190,7 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
standIn := startWAF( standIn := startWAF(
t, t,
func(w http.ResponseWriter, _ *http.Request) { answerAs(tc.disposition, tc.status, tc.body),
if tc.disposition != "" {
w.Header().Set(dispositionHeader, tc.disposition)
}
w.Header().Set(fromWAFHeader, addedHeaderValue)
w.WriteHeader(tc.status)
_, _ = io.WriteString(w, tc.body)
},
) )
app := startApp(t, func(http.ResponseWriter, *http.Request) { app := startApp(t, func(http.ResponseWriter, *http.Request) {
t.Error("the app was asked") t.Error("the app was asked")
@@ -1059,9 +1205,14 @@ func TestAnAnswerOtherThanAllowGoesToTheVisitorAsWAFGaveIt(t *testing.T) {
) )
got := get(t, addr, "/") got := get(t, addr, "/")
if got.status != tc.status || got.body != tc.body || if got.status != tc.status || got.body != tc.want ||
got.header.Get(fromWAFHeader) != addedHeaderValue { got.header.Get(fromWAFHeader) != addedHeaderValue {
t.Errorf("answered %+v, want waf's answer", got) t.Errorf(
"answered %+v, want %d %q",
got,
tc.status,
tc.want,
)
} }
if len(got.header.Values(dispositionHeader)) != 0 { if len(got.header.Values(dispositionHeader)) != 0 {
+24 -1
View File
@@ -7,6 +7,7 @@ import (
"io" "io"
"maps" "maps"
"net/http" "net/http"
"strconv"
"strings" "strings"
"sync" "sync"
@@ -187,7 +188,9 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1 len(r.Header.Values("User-Agent")) <= 1
} }
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition. A deny of
// 403 or 429 gets rproxy's page for that status as its body, in place of
// waf's.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.limits.requestDeadline()) rq.prepareOwnAnswer(rq.limits.requestDeadline())
@@ -195,6 +198,15 @@ func (rq *request) answerWith(answer *wafAnswer) {
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
header.Del(wafprotocol.DispositionHeader) header.Del(wafprotocol.DispositionHeader)
if rq.disposition == wafprotocol.DispositionDeny {
page, found := rq.h.pages.forStatus(answer.status)
if found {
rq.sendPage(answer.status, page)
return
}
}
// Go's server adds a Content-Type it guesses from the body to a response whose // Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none. // headers have no Content-Type key. The key with no value sends none.
if header.Get("Content-Type") == "" { if header.Get("Content-Type") == "" {
@@ -204,6 +216,17 @@ func (rq *request) answerWith(answer *wafAnswer) {
rq.send(answer.status, answer.body) rq.send(answer.status, answer.body)
} }
// sendPage sends page as the body of a refusal with status. The length is set
// here too, since send leaves a HEAD's as waf gave it, which is of waf's body.
func (rq *request) sendPage(status int, page []byte) {
header := rq.out.Header()
header.Set("Content-Type", htmlContentType)
header.Set("X-Content-Type-Options", "nosniff")
header.Set("Cache-Control", "no-store")
header.Set("Content-Length", strconv.Itoa(len(page)))
rq.send(status, page)
}
// sendReport sends waf report on a goroutine of its own, so that the // sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own // visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended. // context, has ended.
+58
View File
@@ -0,0 +1,58 @@
package waf
// The heuristic active_ban. It reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the
// bans are the ban ledger's.
import (
"context"
"time"
)
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
+2 -53
View File
@@ -1,11 +1,9 @@
package waf package waf
// The heuristic active_ban, and banMaker, which makes the bans of several // banMaker, which makes the bans of several heuristics. The bans are the ban
// heuristics. active_ban reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the bans // ledger's.
// are the ban ledger's.
import ( import (
"context"
"net/netip" "net/netip"
"time" "time"
@@ -17,55 +15,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
// banAction is the action of a request refused under a ban or for making one: // banAction is the action of a request refused under a ban or for making one:
// permanent says the ban never runs out. // permanent says the ban never runs out.
func banAction(permanent bool) requestlog.Action { func banAction(permanent bool) requestlog.Action {
+20
View File
@@ -80,6 +80,26 @@ func (rq *request) startLookUp(
return result return result
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// noteLookup notes result for the log line and as the request's asn= and // noteLookup notes result for the log line and as the request's asn= and
// country= tags. // country= tags.
func (rq *request) noteLookup(result lookupResult) { func (rq *request) noteLookup(result lookupResult) {
-329
View File
@@ -1,329 +0,0 @@
// Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf
import (
"cmp"
"io"
"log/slog"
"net/http"
"net/netip"
"time"
lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
// BansPath is where an admin lists and adds bans, and, followed by / and
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
const BansPath = "/_sww/bans"
// ClientsPath is where an admin asks what waf knows of a client, by
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
const ClientsPath = "/_sww/clients/"
// Params are what New needs.
type Params struct {
Config *config.WAF
// RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own
// endpoints.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// Metrics count what waf does. Whoever builds them adds the metrics
// read from the other parts.
Metrics *metrics.Metrics
// Limiter is the table of clients the rate limits count, with each
// client's history.
Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
// request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
}
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
type CoreRuleSetInspector interface {
Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (coreruleset.Result, []byte, error)
}
// Server is the server waf runs.
type Server struct {
httpServer *http.Server
h *handler
}
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
metrics: params.Metrics,
awaitingReport: awaitingReport,
limiter: params.Limiter,
ledger: params.Ledger,
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets, banResponse: cfg.BanResponse},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{
allowed: cfg.ExclusivelyAllowedCountries,
banResponse: cfg.BanResponse,
},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.ownEndpoints = h.routes()
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
// besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
+235
View File
@@ -0,0 +1,235 @@
package waf
import (
"context"
"errors"
"net/http"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
if !ok {
return
}
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
err.Error(),
)
// This panic has Go's server close the connection without an answer,
// and log nothing.
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return question, false
}
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
h.processLog.Warn("a question is refused", "reason", reason)
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.lookupAnswer.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
// holdForReport holds a copy of what the report's steps need, by the request's
// id. A report only ever reaches that copy, never the request being handled.
func (rq *request) holdForReport() {
held := rq.decision
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
}
// deny answers a question with what the visitor is to get for r.
func (rq *request) deny(r refusal) {
rq.refused = &r
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
// request to its client's history, the anomaly counters and the metrics, with
// the status and bytes the report gives. A denied request's counting was
// finished at its decision. ctx is the report's own context.
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
h: h,
out: &responseWriter{
status: report.Status,
bytes: report.ResponseBytes,
},
requestBytes: report.RequestBytes,
}
rq.line.WAFLine = requestlog.WAFLineReport
rq.line.Status = report.Status
rq.line.RequestBytes = report.RequestBytes
rq.line.ResponseBytes = report.ResponseBytes
rq.line.DurationTotalMS = report.DurationTotalMS
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
time.Duration(
report.DurationUpstreamTotalMS*float64(time.Millisecond),
),
)
}
rq.writeLine()
}
+321 -246
View File
@@ -1,18 +1,48 @@
// Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf package waf
import ( import (
"cmp"
"context" "context"
"errors" "io"
"log/slog" "log/slog"
"net" "net"
"net/http" "net/http"
"strings" "net/netip"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
// BansPath is where an admin lists and adds bans, and, followed by / and
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
const BansPath = "/_sww/bans"
// ClientsPath is where an admin asks what waf knows of a client, by
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
const ClientsPath = "/_sww/clients/"
const ( const (
// maxAwaitingReport is the most answered requests held until their // maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped. // reports come. Past it, the request answered longest ago is dropped.
@@ -24,6 +54,79 @@ const (
wafIdleTimeout = 120 * time.Second wafIdleTimeout = 120 * time.Second
) )
// Params are what New needs.
type Params struct {
Config *config.WAF
// RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own
// endpoints.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// Metrics count what waf does. Whoever builds them adds the metrics
// read from the other parts.
Metrics *metrics.Metrics
// Limiter is the table of clients the rate limits count, with each
// client's history.
Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
// request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
}
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
type CoreRuleSetInspector interface {
Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (coreruleset.Result, []byte, error)
}
// Server is the server waf runs.
type Server struct {
httpServer *http.Server
h *handler
}
// New returns the server `sww waf` runs. It answers // New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It // rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the // passes nothing to an app. rproxy applies the size and time limits to the
@@ -70,6 +173,221 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r) s.h.ServeHTTP(w, r)
} }
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
metrics: params.Metrics,
awaitingReport: awaitingReport,
limiter: params.Limiter,
ledger: params.Ledger,
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets, banResponse: cfg.BanResponse},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{
allowed: cfg.ExclusivelyAllowedCountries,
banResponse: cfg.BanResponse,
},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.ownEndpoints = h.routes()
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
// besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
// ServeHTTP handles one request to waf: a request carrying // ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for one of // wafprotocol.ClientHeader is rproxy's question, and any other is for one of
// waf's own endpoints. // waf's own endpoints.
@@ -82,246 +400,3 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
h.ownEndpoints.ServeHTTP(w, r) h.ownEndpoints.ServeHTTP(w, r)
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
if !ok {
return
}
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
err.Error(),
)
// This panic has Go's server close the connection without an answer,
// and log nothing.
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return question, false
}
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
h.processLog.Warn("a question is refused", "reason", reason)
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.lookupAnswer.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
// holdForReport holds a copy of what the report's steps need, by the request's
// id. A report only ever reaches that copy, never the request being handled.
func (rq *request) holdForReport() {
held := rq.decision
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
}
// deny answers a question with what the visitor is to get for r.
func (rq *request) deny(r refusal) {
rq.refused = &r
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
// request to its client's history, the anomaly counters and the metrics, with
// the status and bytes the report gives. A denied request's counting was
// finished at its decision. ctx is the report's own context.
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
h: h,
out: &responseWriter{
status: report.Status,
bytes: report.ResponseBytes,
},
requestBytes: report.RequestBytes,
}
rq.line.WAFLine = requestlog.WAFLineReport
rq.line.Status = report.Status
rq.line.RequestBytes = report.RequestBytes
rq.line.ResponseBytes = report.ResponseBytes
rq.line.DurationTotalMS = report.DurationTotalMS
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
time.Duration(
report.DurationUpstreamTotalMS*float64(time.Millisecond),
),
)
}
rq.writeLine()
}
+8 -3
View File
@@ -8,6 +8,8 @@ set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
set -x
main() { main() {
cd "$ROOT" cd "$ROOT"
# Own line: a failing command substitution inside an argument does # Own line: a failing command substitution inside an argument does
@@ -18,12 +20,15 @@ main() {
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
revision="$(git rev-parse HEAD 2>/dev/null || true)" revision="$(git rev-parse HEAD 2>/dev/null || true)"
[ -n "$revision" ] || revision="unknown" [ -n "$revision" ] || revision="unknown"
created="$(date -u +%Y-%m-%dT%H:%M:%SZ)" created="$(date -u +%Y-%m-%d)"
docker build --no-cache \ docker build \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
--build-arg REVISION="$revision" \ --build-arg REVISION="$revision" \
--build-arg CREATED="$created" \ --build-arg CREATED="$created" \
-t "$("$SCRIPT_DIR/projectname")" . -t "git.eeqj.de/sneak/$("$SCRIPT_DIR/projectname")" \
-t "git.eeqj.de/sneak/$("$SCRIPT_DIR/projectname"):$version" \
-t "git.eeqj.de/sneak/$("$SCRIPT_DIR/projectname"):$(date -u +%Y-%m-%d)" \
.
} }
main "$@" main "$@"
+2 -9
View File
@@ -14,16 +14,9 @@ set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { PUBLISH_IMAGE="git.eeqj.de/sneak/smallwebwaf"
if [ -z "${DOCKER_HOST:-}" ]; then
echo "script/publish: DOCKER_HOST is not set; set it to the Docker host that builds the image, such as ssh://user@build.example.com" >&2
exit 1
fi
if [ -z "${PUBLISH_IMAGE:-}" ]; then
echo "script/publish: PUBLISH_IMAGE is not set; set it to the image name in the registry, such as registry.example.com/smallwebwaf" >&2
exit 1
fi
main() {
cd "$ROOT" cd "$ROOT"
# Own line: a failing command substitution inside an argument does # Own line: a failing command substitution inside an argument does