Compare commits

..
Author SHA1 Message Date
sneak ad7ea4608b Hold Go's server privately in waf.Server and rproxy.Server (closes #437)
check / check (push) Failing after 1m42s
Neither Server embeds *http.Server any more. Each holds it as a private
field, is its handler, and has Serve, Shutdown, Close and ServeHTTP.
sww's serve hook takes the new daemonServer interface of those first
three, and fx provides each Server as it.

Model: opus-5-5
2026-11-20 00:33:14 +00:00
clawbot cc77435f2a rproxy: one type for the goroutines sending reports (closes #433)
check / check (push) Successful in 4m31s
handler held a WaitGroup, a mutex and a stopped flag for the reports,
and the rule that no report starts once waiting has begun was split
between sendReport and Server.WaitForReports. reportGoroutines now
holds all three, and its two methods are the only code that touches
the lock and the flag.

Model: opus-5-5
2026-11-20 01:21:06 +01:00
clawbot dcfdd12c51 waf: Server exports no parts (closes #436)
check / check (push) Successful in 4m9s
waf.Server held its parts as public fields only for the tests to read.
The tests now keep the Params the server was built from in a test-only
wafAndParts, which embeds both, so every assertion reads as before.
newWAFWith takes a change to the Params instead of a build function, so
the Params kept are the ones the server was built from.

Model: opus-5-5
2026-11-20 00:57:04 +01:00
clawbot 2301a76029 Give ban causes, limit kinds, alert events, actions and dispositions named types (closes #428)
check / check (push) Failing after 1m41s
Each fixed set of string constants becomes a named string type in the
package that owns it, and the fields and parameters that carry it use
the type: bans.Cause, ratelimit.LimitKind, alerts.Event,
alerts.DestinationName, rules.Action, requestlog.Action,
requestlog.Offence, requestlog.WAFLineKind and wafprotocol.Disposition.
Every value keeps its string, so the logs, state files, alerts and
metrics do not change.

The switches exhaustive now checks list their missing cases.

Model: opus-5-5
2026-11-20 00:43:20 +01:00
clawbot abf6c8d517 rproxy: the two time limits get a type and a file of their own (closes #432)
check / check (push) Failing after 3m36s
timeLimits in internal/rproxy/timelimits.go owns the mutex, the two
timers and the end of the request; request holds one field of it. It
is given refuse, whether the visitor's body is being waited on, and a
function that ends that read. Tests of the type alone run in a
synctest bubble.

Model: opus-5-5
2026-11-20 00:28:55 +01:00
clawbot 17400f68b8 waf: heuristics pass facts through typed fields of decision (closes #442)
check / check (push) Failing after 1m35s
The tags, what observe mode would have done, the broken limit and the
three reputation flags are fields of decision, each naming what sets it.
The log line is written from them and from the lookup answer, and no
heuristic, ban note or alert reads it back. A ban's notes take the
request's method, Host and User-Agent from decision, since the report's
steps have no request to read them from.

Model: opus-5-5
2026-11-19 23:43:00 +01:00
clawbot 9d008ed31a waf: a heuristic says itself that it refuses whatever the lookup answers (closes #446)
check / check (push) Failing after 1m37s
refusedWhateverTheAnswer asks each heuristic of afterLookup that
implements the new refusesWithoutLookup interface, instead of naming
blocklists and rate_limits. The comment on adding a heuristic names the
places outside its file that may change with it.

Model: opus-5-5
2026-11-19 23:28:52 +01:00
clawbot 86bb33bf07 waf: list its own endpoints on a chi router in routes.go (closes #426)
check / check (push) Successful in 7m7s
routes.go lists every own endpoint. One middleware checks a token and
another writes the log line. answerOwnEndpoint, answerAdmin and
endpoint() are gone; the handlers return http.HandlerFunc. The router
matches the decoded path, as the old code did. A test checks the
request_bytes of an own endpoint's log line.

Model: opus-5-5
2026-11-19 23:13:24 +01:00
clawbot 91444a37b3 metrics: build from one Params in place of seven Add calls (closes #438)
check / check (push) Failing after 1m48s
metrics.New takes a Params holding every part at once, each behind a
small interface declared in metrics beside the metrics it reads. A part
that is off is left nil and adds no metric. The rule-match and
reputation-hit counters are made in New, so no metric field is nil and
no call order is needed. metrics no longer imports remotelog or rules.

Model: opus-5-5
2026-11-19 22:59:44 +01:00
76 changed files with 1915 additions and 1058 deletions

No files matched your search

+4 -3
View File
@@ -74,9 +74,9 @@ never as a time.
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts `go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work. and stops their background work.
- `internal/config`: reads the settings, the one place they are read. - `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers - `internal/waf`: `waf`'s server. It runs the checks in `check`. Its
`waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals `routes.go` lists `waf`'s own endpoints. It counts bytes, refusals and
and anomalies, and writes the log lines. anomalies, and writes the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf` - `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits. library's `httputil.ReverseProxy`, within the time and size limits.
@@ -128,6 +128,7 @@ never as a time.
default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that default `SWW_WAF_CORE_RULE_SET_DISABLED_RULES`, are built for that
version. They are checked again before the version changes, since rule version. They are checked again before the version changes, since rule
ids, and what each rule matches, change between versions; ids, and what each rule matches, change between versions;
- `github.com/go-chi/chi/v5` to route the requests to `waf`'s own endpoints;
- `github.com/hashicorp/golang-lru/v2` for the bounded tables; - `github.com/hashicorp/golang-lru/v2` for the bounded tables;
- `github.com/oschwald/maxminddb-golang/v2` to read the lookup database; - `github.com/oschwald/maxminddb-golang/v2` to read the lookup database;
- `github.com/fsnotify/fsnotify` to notice edited files; - `github.com/fsnotify/fsnotify` to notice edited files;
+1
View File
@@ -7,6 +7,7 @@ require (
github.com/corazawaf/coraza-coreruleset/v4 v4.25.0 github.com/corazawaf/coraza-coreruleset/v4 v4.25.0
github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68 github.com/corazawaf/coraza/v3 v3.8.2-0.20261022102553-d028be399c68
github.com/fsnotify/fsnotify v1.10.1 github.com/fsnotify/fsnotify v1.10.1
github.com/go-chi/chi/v5 v5.5.0
github.com/hashicorp/golang-lru/v2 v2.0.7 github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/maxmind/mmdbwriter v1.2.0 github.com/maxmind/mmdbwriter v1.2.0
github.com/oschwald/maxminddb-golang/v2 v2.7.0 github.com/oschwald/maxminddb-golang/v2 v2.7.0
+2
View File
@@ -21,6 +21,8 @@ github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHk
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0= github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho= github.com/fsnotify/fsnotify v1.10.1 h1:b0/UzAf9yR5rhf3RPm9gf3ehBPpf0oZKIjtpKrx59Ho=
github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo= github.com/fsnotify/fsnotify v1.10.1/go.mod h1:TLheqan6HD6GBK6PrDWyDPBaEV8LspOxvPSjC+bVfgo=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs= github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM= github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4= github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
+51 -38
View File
@@ -34,65 +34,76 @@ import (
"time" "time"
) )
// Event is what an alert is for, such as EventBan.
type Event string
// The events an alert is for, as SWW_WAF_ALERT_EVENTS names them. // The events an alert is for, as SWW_WAF_ALERT_EVENTS names them.
const ( const (
// EventBan is a ban sww made. // EventBan is a ban sww made.
EventBan = "ban" EventBan Event = "ban"
// EventPermanentBan is a permanent ban sww made, or a ban it // EventPermanentBan is a permanent ban sww made, or a ban it
// made permanent. // made permanent.
EventPermanentBan = "permanent_ban" EventPermanentBan Event = "permanent_ban"
// EventAnomaly is a count of requests or bytes over an anomaly // EventAnomaly is a count of requests or bytes over an anomaly
// threshold. // threshold.
EventAnomaly = "anomaly" EventAnomaly Event = "anomaly"
// EventCoreRuleSetMatch is a request the Core Rule Set scored at or over // EventCoreRuleSetMatch is a request the Core Rule Set scored at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in // SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, refused in block mode, let through in
// detect mode. // detect mode.
EventCoreRuleSetMatch = "core_rule_set_match" EventCoreRuleSetMatch Event = "core_rule_set_match"
// EventReputationHit is a request whose client a blocklist or a DNSBL // EventReputationHit is a request whose client a blocklist or a DNSBL
// zone lists, or whose AbuseIPDB score is a hit. // zone lists, or whose AbuseIPDB score is a hit.
EventReputationHit = "reputation_hit" EventReputationHit Event = "reputation_hit"
// EventSourceFailure is GeoJS failing or refusing sww, a fetch // EventSourceFailure is GeoJS failing or refusing sww, a fetch
// of a list failing, a query to a DNSBL zone or a check with AbuseIPDB // of a list failing, a query to a DNSBL zone or a check with AbuseIPDB
// failing or refused, or the day's AbuseIPDB checks used up. // failing or refused, or the day's AbuseIPDB checks used up.
EventSourceFailure = "source_failure" EventSourceFailure Event = "source_failure"
// EventFileError is one of: // EventFileError is one of:
// - a rule file or state file edited while sww runs that does not parse; // - a rule file or state file edited while sww runs that does not parse;
// - a cache file that does not parse, at start or after an edit; // - a cache file that does not parse, at start or after an edit;
// - a state file or cache file that cannot be written; // - a state file or cache file that cannot be written;
// - a replacement of the lookup database that cannot be read. // - a replacement of the lookup database that cannot be read.
EventFileError = "file_error" EventFileError Event = "file_error"
// EventSummary is the summary sent as an hour ends: of the alerts held // EventSummary is the summary sent as an hour ends: of the alerts held
// back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held // back in it past SWW_WAF_ALERT_MAX_PER_HOUR, and of the repeats held
// back by the cooldowns dropped as it ends, which no alert let through // back by the cooldowns dropped as it ends, which no alert let through
// has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for // has given. It is sent with SWW_WAF_ALERT_MAX_PER_HOUR off too, for
// those repeats. SWW_WAF_ALERT_EVENTS does not name it. // those repeats. SWW_WAF_ALERT_EVENTS does not name it.
EventSummary = "summary" EventSummary Event = "summary"
) )
// Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its // Events returns every event SWW_WAF_ALERT_EVENTS can name, which is its
// default. // default.
func Events() []string { func Events() []Event {
return []string{ return []Event{
EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly, EventBan, EventPermanentBan, EventCoreRuleSetMatch, EventAnomaly,
EventReputationHit, EventSourceFailure, EventFileError, EventReputationHit, EventSourceFailure, EventFileError,
} }
} }
// DestinationName is the name of a destination alerts are sent to:
// DestinationWebhook, DestinationSlack or DestinationNtfy.
type DestinationName string
// The destinations alerts are sent to, as the metrics and alerts.json // The destinations alerts are sent to, as the metrics and alerts.json
// name them. // name them.
const ( const (
// DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names. // DestinationWebhook is the webhook SWW_WAF_ALERT_WEBHOOK_URL names.
DestinationWebhook = "webhook" DestinationWebhook DestinationName = "webhook"
// DestinationSlack is the Slack incoming webhook // DestinationSlack is the Slack incoming webhook
// SWW_WAF_ALERT_SLACK_WEBHOOK_URL names. // SWW_WAF_ALERT_SLACK_WEBHOOK_URL names.
DestinationSlack = "slack" DestinationSlack DestinationName = "slack"
// DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names. // DestinationNtfy is the ntfy topic SWW_WAF_ALERT_NTFY_URL names.
DestinationNtfy = "ntfy" DestinationNtfy DestinationName = "ntfy"
) )
// Destinations returns every destination alerts can be sent to. // Destinations returns every destination alerts can be sent to.
func Destinations() []string { func Destinations() []DestinationName {
return []string{DestinationWebhook, DestinationSlack, DestinationNtfy} return []DestinationName{
DestinationWebhook,
DestinationSlack,
DestinationNtfy,
}
} }
const ( const (
@@ -135,7 +146,7 @@ type Params struct {
NtfyURL *url.URL NtfyURL *url.URL
NtfyToken string NtfyToken string
// Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS). // Events are the events alerts are sent for (SWW_WAF_ALERT_EVENTS).
Events []string Events []Event
// Cooldown is how long a repeat of an alert is held back // Cooldown is how long a repeat of an alert is held back
// (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts // (SWW_WAF_ALERT_COOLDOWN), 0 for no time. MaxPerHour is the most alerts
// sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit. // sent in an hour (SWW_WAF_ALERT_MAX_PER_HOUR), 0 for no limit.
@@ -159,7 +170,7 @@ type Params struct {
type Alert struct { type Alert struct {
Instance string `json:"instance"` Instance string `json:"instance"`
Time time.Time `json:"time"` Time time.Time `json:"time"`
Event string `json:"event"` Event Event `json:"event"`
Host string `json:"host,omitempty"` Host string `json:"host,omitempty"`
Client netip.Addr `json:"client"` Client netip.Addr `json:"client"`
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
@@ -186,7 +197,7 @@ type Alert struct {
// //
//nolint:tagliatelle // the state files use snake_case, as the request log does //nolint:tagliatelle // the state files use snake_case, as the request log does
type Cooldown struct { type Cooldown struct {
Event string `json:"event"` Event Event `json:"event"`
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
File string `json:"file,omitempty"` File string `json:"file,omitempty"`
Source string `json:"source,omitempty"` Source string `json:"source,omitempty"`
@@ -205,7 +216,7 @@ type Cooldown struct {
type Hour struct { type Hour struct {
Start time.Time `json:"start"` Start time.Time `json:"start"`
Sent int `json:"sent"` Sent int `json:"sent"`
HeldBack map[string]int `json:"held_back"` HeldBack map[Event]int `json:"held_back"`
} }
// State is what alerts.json holds: the cooldowns, the hour under way, and // State is what alerts.json holds: the cooldowns, the hour under way, and
@@ -214,7 +225,7 @@ type Hour struct {
type State struct { type State struct {
Cooldowns []Cooldown `json:"cooldowns"` Cooldowns []Cooldown `json:"cooldowns"`
Hour Hour `json:"hour"` Hour Hour `json:"hour"`
Waiting map[string][]Alert `json:"waiting"` Waiting map[DestinationName][]Alert `json:"waiting"`
} }
// Counts are, for a destination, how many alerts it took, how many // Counts are, for a destination, how many alerts it took, how many
@@ -248,7 +259,7 @@ type destination struct {
// name is how the metrics and alerts.json name the destination, and // name is how the metrics and alerts.json name the destination, and
// setting the setting that is its URL, which the log names in place // setting the setting that is its URL, which the log names in place
// of the URL. // of the URL.
name string name DestinationName
setting string setting string
url *url.URL url *url.URL
// message returns the body an alert is posted with, and the headers // message returns the body an alert is posted with, and the headers
@@ -273,7 +284,7 @@ type destination struct {
// scope with the same AS number or name, as its detail names them. Each // scope with the same AS number or name, as its detail names them. Each
// is empty for an alert without one. // is empty for an alert without one.
type cooldownKey struct { type cooldownKey struct {
event string event Event
netblock netip.Prefix netblock netip.Prefix
file string file string
source string source string
@@ -306,7 +317,7 @@ func New(params Params) *Queue {
q := &Queue{ q := &Queue{
params: params, params: params,
cooldowns: map[cooldownKey]*Cooldown{}, cooldowns: map[cooldownKey]*Cooldown{},
hour: Hour{HeldBack: map[string]int{}}, hour: Hour{HeldBack: map[Event]int{}},
} }
if params.WebhookURL != nil { if params.WebhookURL != nil {
@@ -377,7 +388,7 @@ func (q *Queue) Raise(alert Alert) {
// event, no alert for event on netblock was let through less than // event, no alert for event on netblock was let through less than
// Cooldown before, and fewer than MaxPerHour alerts have been let through // Cooldown before, and fewer than MaxPerHour alerts have been let through
// in the hour under way. Unlike Raise, it counts nothing. // in the hour under way. Unlike Raise, it counts nothing.
func (q *Queue) WouldSend(event string, netblock netip.Prefix) bool { func (q *Queue) WouldSend(event Event, netblock netip.Prefix) bool {
if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) { if len(q.destinations) == 0 || !slices.Contains(q.params.Events, event) {
return false return false
} }
@@ -433,7 +444,7 @@ func (q *Queue) Run(ctx context.Context) {
// Counts returns the counts of the destination name, all 0 for one not // Counts returns the counts of the destination name, all 0 for one not
// set. // set.
func (q *Queue) Counts(name string) Counts { func (q *Queue) Counts(name DestinationName) Counts {
for _, d := range q.destinations { for _, d := range q.destinations {
if d.name == name { if d.name == name {
return Counts{ return Counts{
@@ -449,8 +460,8 @@ func (q *Queue) Counts(name string) Counts {
// DestinationsSet returns the destinations set, in the order of // DestinationsSet returns the destinations set, in the order of
// Destinations. // Destinations.
func (q *Queue) DestinationsSet() []string { func (q *Queue) DestinationsSet() []DestinationName {
names := make([]string, 0, len(q.destinations)) names := make([]DestinationName, 0, len(q.destinations))
for _, d := range q.destinations { for _, d := range q.destinations {
names = append(names, d.name) names = append(names, d.name)
} }
@@ -474,7 +485,7 @@ func (q *Queue) Snapshot() State {
state := State{ state := State{
Cooldowns: make([]Cooldown, 0, len(q.cooldowns)), Cooldowns: make([]Cooldown, 0, len(q.cooldowns)),
Hour: q.hour, Hour: q.hour,
Waiting: map[string][]Alert{}, Waiting: map[DestinationName][]Alert{},
} }
state.Hour.HeldBack = maps.Clone(q.hour.HeldBack) state.Hour.HeldBack = maps.Clone(q.hour.HeldBack)
@@ -525,7 +536,7 @@ func (q *Queue) Load(state State) {
q.hour.HeldBack = maps.Clone(state.Hour.HeldBack) q.hour.HeldBack = maps.Clone(state.Hour.HeldBack)
if q.hour.HeldBack == nil { if q.hour.HeldBack == nil {
q.hour.HeldBack = map[string]int{} q.hour.HeldBack = map[Event]int{}
} }
for _, d := range q.destinations { for _, d := range q.destinations {
@@ -537,7 +548,7 @@ func (q *Queue) Load(state State) {
// its URL, that URL, target, and message, which makes the messages sent // its URL, that URL, target, and message, which makes the messages sent
// to it. // to it.
func (q *Queue) addDestination( func (q *Queue) addDestination(
name, setting string, target *url.URL, name DestinationName, setting string, target *url.URL,
message func(alert *Alert) ([]byte, http.Header, error), message func(alert *Alert) ([]byte, http.Header, error),
) { ) {
q.destinations = append(q.destinations, &destination{ q.destinations = append(q.destinations, &destination{
@@ -649,7 +660,7 @@ func (q *Queue) endHour(now time.Time) {
}) })
} }
q.hour = Hour{Start: start, HeldBack: map[string]int{}} q.hour = Hour{Start: start, HeldBack: map[Event]int{}}
} }
// queue adds alert to the alerts waiting for each destination. // queue adds alert to the alerts waiting for each destination.
@@ -904,20 +915,21 @@ func (q *Queue) ntfyMessage(alert *Alert) ([]byte, http.Header, error) {
// ntfyPriority returns the priority an alert for event is published to // ntfyPriority returns the priority an alert for event is published to
// ntfy with: high for an event the admin needs to look at. // ntfy with: high for an event the admin needs to look at.
func ntfyPriority(event string) string { func ntfyPriority(event Event) string {
switch event { switch event {
case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError: case EventPermanentBan, EventAnomaly, EventSourceFailure, EventFileError:
return "high" return "high"
case EventReputationHit: case EventReputationHit:
return "low" return "low"
default: // ban, core_rule_set_match and summary case EventBan, EventCoreRuleSetMatch, EventSummary:
return "default"
} }
return "default"
} }
// ntfyTag returns the tag an alert for event is published to ntfy with, // ntfyTag returns the tag an alert for event is published to ntfy with,
// which ntfy shows as an emoji. // which ntfy shows as an emoji.
func ntfyTag(event string) string { func ntfyTag(event Event) string {
switch event { switch event {
case EventBan, EventPermanentBan: case EventBan, EventPermanentBan:
return "no_entry" return "no_entry"
@@ -929,15 +941,16 @@ func ntfyTag(event string) string {
return "label" return "label"
case EventSourceFailure, EventFileError: case EventSourceFailure, EventFileError:
return "warning" return "warning"
default: // summary case EventSummary:
return "bar_chart"
} }
return "bar_chart"
} }
// title returns the title of alert in Slack and ntfy: the instance and // title returns the title of alert in Slack and ntfy: the instance and
// the event. // the event.
func title(alert *Alert) string { func title(alert *Alert) string {
return alert.Instance + ": " + alert.Event return alert.Instance + ": " + string(alert.Event)
} }
// text returns the text of alert in Slack and ntfy: its reason, then a // text returns the text of alert in Slack and ntfy: its reason, then a
+29 -18
View File
@@ -126,7 +126,7 @@ func TestOnlyTheChosenEventsAreSent(t *testing.T) {
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
params := newParams() params := newParams()
params.Events = []string{ params.Events = []alerts.Event{
alerts.EventSourceFailure, alerts.EventSourceFailure,
alerts.EventFileError, alerts.EventFileError,
} }
@@ -156,7 +156,7 @@ func TestWouldSendOnlyForTheChosenEvents(t *testing.T) {
t.Parallel() t.Parallel()
params := newParams() params := newParams()
params.Events = []string{ params.Events = []alerts.Event{
alerts.EventSourceFailure, alerts.EventSourceFailure,
alerts.EventFileError, alerts.EventFileError,
} }
@@ -186,8 +186,8 @@ func TestRaiseDoesNothingWithoutADestination(t *testing.T) {
// No alert waits, no cooldown has started, and the hour counts none. // No alert waits, no cooldown has started, and the hour counts none.
want := alerts.State{ want := alerts.State{
Cooldowns: []alerts.Cooldown{}, Cooldowns: []alerts.Cooldown{},
Hour: alerts.Hour{HeldBack: map[string]int{}}, Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[string][]alerts.Alert{}, Waiting: map[alerts.DestinationName][]alerts.Alert{},
} }
if got := q.Snapshot(); !reflect.DeepEqual(got, want) { if got := q.Snapshot(); !reflect.DeepEqual(got, want) {
t.Errorf("state %+v, want %+v", got, want) t.Errorf("state %+v, want %+v", got, want)
@@ -235,7 +235,7 @@ func TestRepeatWithinTheCooldownIsHeldBackAndCountedInTheNext(t *testing.T) {
synctest.Test(t, func(t *testing.T) { synctest.Test(t, func(t *testing.T) {
webhook, q := start(t, newParams()) webhook, q := start(t, newParams())
raise := func(event string, n int) { raise := func(event alerts.Event, n int) {
q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)}) q.Raise(alerts.Alert{Event: event, Netblock: netblock(n)})
} }
@@ -1014,7 +1014,7 @@ func TestNtfyTokenIsSentToNtfyAlone(t *testing.T) {
) )
synctest.Wait() synctest.Wait()
for destination, want := range map[string][]string{ for destination, want := range map[alerts.DestinationName][]string{
alerts.DestinationWebhook: nil, alerts.DestinationWebhook: nil,
alerts.DestinationSlack: nil, alerts.DestinationSlack: nil,
alerts.DestinationNtfy: tc.want, alerts.DestinationNtfy: tc.want,
@@ -1063,7 +1063,7 @@ func TestDestinationThatDoesNotAnswerHoldsUpNeitherOther(t *testing.T) {
// With no time passed, the webhook and ntfy have taken every alert, // With no time passed, the webhook and ntfy have taken every alert,
// while Slack has not answered the first. // while Slack has not answered the first.
for destination, want := range map[string]alerts.Counts{ for destination, want := range map[alerts.DestinationName]alerts.Counts{
alerts.DestinationWebhook: {Sent: 3}, alerts.DestinationWebhook: {Sent: 3},
alerts.DestinationSlack: {}, alerts.DestinationSlack: {},
alerts.DestinationNtfy: {Sent: 3}, alerts.DestinationNtfy: {Sent: 3},
@@ -1126,7 +1126,7 @@ func TestDropsAreCountedForEachDestination(t *testing.T) {
// the webhook takes every one. // the webhook takes every one.
const all = alerts.QueueSize + 1 const all = alerts.QueueSize + 1
for destination, want := range map[string]alerts.Counts{ for destination, want := range map[alerts.DestinationName]alerts.Counts{
alerts.DestinationWebhook: {Sent: all}, alerts.DestinationWebhook: {Sent: all},
alerts.DestinationSlack: {Dropped: 1}, alerts.DestinationSlack: {Dropped: 1},
alerts.DestinationNtfy: {SendFailures: all, Dropped: all}, alerts.DestinationNtfy: {SendFailures: all, Dropped: all},
@@ -1152,11 +1152,21 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
// As read from alerts.json. The alert waiting for the webhook, which // As read from alerts.json. The alert waiting for the webhook, which
// is not set, is dropped. // is not set, is dropped.
q.Load(roundTrip(t, alerts.State{Waiting: map[string][]alerts.Alert{ q.Load(
roundTrip(
t,
alerts.State{
Waiting: map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationWebhook: {waiting("first")}, alerts.DestinationWebhook: {waiting("first")},
alerts.DestinationSlack: {waiting("second"), waiting("third")}, alerts.DestinationSlack: {
waiting("second"),
waiting("third"),
},
alerts.DestinationNtfy: {waiting("fourth")}, alerts.DestinationNtfy: {waiting("fourth")},
}})) },
},
),
)
synctest.Wait() synctest.Wait()
wantBodies(t, standIns[alerts.DestinationSlack], wantBodies(t, standIns[alerts.DestinationSlack],
@@ -1165,7 +1175,7 @@ func TestEachDestinationIsSentOnlyTheAlertsWaitingForIt(t *testing.T) {
wantBodies(t, standIns[alerts.DestinationNtfy], "fourth") wantBodies(t, standIns[alerts.DestinationNtfy], "fourth")
// alerts.json then lists Slack and ntfy alone, with no alert waiting. // alerts.json then lists Slack and ntfy alone, with no alert waiting.
want := map[string][]alerts.Alert{ want := map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationSlack: {}, alerts.DestinationNtfy: {}, alerts.DestinationSlack: {}, alerts.DestinationNtfy: {},
} }
if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) { if got := q.Snapshot().Waiting; !reflect.DeepEqual(got, want) {
@@ -1334,11 +1344,11 @@ func start(t *testing.T, params alerts.Params) (*standIn, *alerts.Queue) {
func startAll( func startAll(
t *testing.T, t *testing.T,
params alerts.Params, params alerts.Params,
) (map[string]*standIn, *alerts.Queue) { ) (map[alerts.DestinationName]*standIn, *alerts.Queue) {
t.Helper() t.Helper()
q := alerts.New(params) q := alerts.New(params)
standIns := map[string]*standIn{} standIns := map[alerts.DestinationName]*standIn{}
for _, destination := range q.DestinationsSet() { for _, destination := range q.DestinationsSet() {
standIns[destination] = &standIn{answers: answering} standIns[destination] = &standIn{answers: answering}
@@ -1404,14 +1414,14 @@ func wantAlert(t *testing.T, got, want map[string]any) {
// wantEvents checks the events of the alerts the webhook was sent, in // wantEvents checks the events of the alerts the webhook was sent, in
// order. // order.
func wantEvents(t *testing.T, webhook *standIn, want ...string) { func wantEvents(t *testing.T, webhook *standIn, want ...alerts.Event) {
t.Helper() t.Helper()
got := make([]string, 0, len(webhook.received())) got := make([]alerts.Event, 0, len(webhook.received()))
for _, request := range webhook.received() { for _, request := range webhook.received() {
event, _ := request.alert["event"].(string) event, _ := request.alert["event"].(string)
got = append(got, event) got = append(got, alerts.Event(event))
} }
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
@@ -1447,7 +1457,8 @@ func wantCounts(
// wantDestinationCounts checks q's counts for destination. // wantDestinationCounts checks q's counts for destination.
func wantDestinationCounts( func wantDestinationCounts(
t *testing.T, q *alerts.Queue, destination string, want alerts.Counts, t *testing.T, q *alerts.Queue, destination alerts.DestinationName,
want alerts.Counts,
) { ) {
t.Helper() t.Helper()
+3 -1
View File
@@ -7,7 +7,9 @@ const QueueSize = queueSize
// SetTransport has q's requests to the destination name go through // SetTransport has q's requests to the destination name go through
// transport instead of the network. // transport instead of the network.
func (q *Queue) SetTransport(name string, transport http.RoundTripper) { func (q *Queue) SetTransport(
name DestinationName, transport http.RoundTripper,
) {
for _, d := range q.destinations { for _, d := range q.destinations {
if d.name == name { if d.name == name {
d.httpClient.Transport = transport d.httpClient.Transport = transport
+6 -4
View File
@@ -295,7 +295,8 @@ func (c *Counters) netAround(client netip.Addr) netip.Prefix {
// overThreshold is a count over its threshold: what it counts, requests or // overThreshold is a count over its threshold: what it counts, requests or
// bytes, its window, the count and the threshold. // bytes, its window, the count and the threshold.
type overThreshold struct { type overThreshold struct {
kind, window string kind ratelimit.LimitKind
window string
count float64 count float64
threshold int64 threshold int64
} }
@@ -346,7 +347,8 @@ func (c *Counter) add(
// bucketCount is one of a counter's four counts: requests or bytes, in a // bucketCount is one of a counter's four counts: requests or bytes, in a
// window of length, and the buckets they are counted in. // window of length, and the buckets they are counted in.
type bucketCount struct { type bucketCount struct {
kind, window string kind ratelimit.LimitKind
window string
length time.Duration length time.Duration
buckets *ratelimit.Buckets buckets *ratelimit.Buckets
} }
@@ -404,8 +406,8 @@ func (k key) counter() *Counter {
// counted, the count and the threshold. // counted, the count and the threshold.
func alertFor(r Request, k key, o overThreshold) alerts.Alert { func alertFor(r Request, k key, o overThreshold) alerts.Alert {
detail := map[string]any{ detail := map[string]any{
"scope": k.scope, "window": o.window, "kind": o.kind, "count": o.count, "scope": k.scope, "window": o.window, "kind": string(o.kind),
"threshold": o.threshold, "count": o.count, "threshold": o.threshold,
} }
var counted string var counted string
+21 -14
View File
@@ -16,20 +16,26 @@ import (
"time" "time"
"github.com/hashicorp/golang-lru/v2/simplelru" "github.com/hashicorp/golang-lru/v2/simplelru"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
) )
// Cause is why a ban was made: CauseLimit, CauseAttack, CauseAdmin or
// CauseAbuseIPDB.
type Cause string
// The causes of bans. // The causes of bans.
const ( const (
// CauseLimit is a ban sww made for a broken limit. // CauseLimit is a ban sww made for a broken limit.
CauseLimit = "limit" CauseLimit Cause = "limit"
// CauseAttack is a ban sww made for a clear sign of attack. // CauseAttack is a ban sww made for a clear sign of attack.
CauseAttack = "attack" CauseAttack Cause = "attack"
// CauseAdmin is a ban an admin made, or one sww made that an // CauseAdmin is a ban an admin made, or one sww made that an
// admin keeps. It is never dropped. // admin keeps. It is never dropped.
CauseAdmin = "admin" CauseAdmin Cause = "admin"
// CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's // CauseAbuseIPDB is a ban sww made for a client on AbuseIPDB's
// blacklist. It is forgotten once it has ended. // blacklist. It is forgotten once it has ended.
CauseAbuseIPDB = "abuseipdb" CauseAbuseIPDB Cause = "abuseipdb"
) )
// repeatFactor is how many times as long as the netblock's last ban a ban // repeatFactor is how many times as long as the netblock's last ban a ban
@@ -68,8 +74,7 @@ type Ban struct {
Start time.Time Start time.Time
// Expires is when the ban ends, zero for a permanent ban. // Expires is when the ban ends, zero for a permanent ban.
Expires time.Time Expires time.Time
// Cause is CauseLimit, CauseAttack, CauseAdmin or CauseAbuseIPDB. Cause Cause
Cause string
// Reason is a short text: for a ban sww made, the limit broken, the // Reason is a short text: for a ban sww made, the limit broken, the
// rule that matched or AbuseIPDB's score; for an admin's, what the // rule that matched or AbuseIPDB's score; for an admin's, what the
// admin wrote. // admin wrote.
@@ -111,7 +116,7 @@ type Notes struct {
// request that broke the limit included. // request that broke the limit included.
// These are what counted toward the ban, and the window is the time // These are what counted toward the ban, and the window is the time
// over which they came. // over which they came.
Kind string `json:"kind,omitempty"` Kind ratelimit.LimitKind `json:"kind,omitempty"`
Limit int64 `json:"limit,omitempty"` Limit int64 `json:"limit,omitempty"`
Window string `json:"window,omitempty"` Window string `json:"window,omitempty"`
Count float64 `json:"count,omitempty"` Count float64 `json:"count,omitempty"`
@@ -194,7 +199,7 @@ type Ledger struct {
// made is how many bans have been made since the start, by cause: by // made is how many bans have been made since the start, by cause: by
// the ledger, and by an admin, through BanForAdmin or in an edit of // the ledger, and by an admin, through BanForAdmin or in an edit of
// bans.json. // bans.json.
made map[string]int made map[Cause]int
// v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6 // v4Lengths and v6Lengths are the lengths of the IPv4 and IPv6
// netblocks that have been banned. Check looks for a ban at each of // netblocks that have been banned. Check looks for a ban at each of
// them, so that a ban read from bans.json refuses every client in its // them, so that a ban read from bans.json refuses every client in its
@@ -221,7 +226,7 @@ func New(params Params) *Ledger {
params: params, params: params,
changed: make(chan struct{}, 1), changed: make(chan struct{}, 1),
netblocks: netblocks, netblocks: netblocks,
made: map[string]int{}, made: map[Cause]int{},
} }
} }
@@ -303,7 +308,7 @@ func activeBan(bans []Ban, now time.Time) *Ban {
// Grounds are why a request makes a ban: its cause, CauseLimit or // Grounds are why a request makes a ban: its cause, CauseLimit or
// CauseAttack, and its notes. // CauseAttack, and its notes.
type Grounds struct { type Grounds struct {
Cause string Cause Cause
Notes Notes Notes Notes
} }
@@ -345,7 +350,7 @@ func (l *Ledger) WouldBan(
// or CauseAttack, made at now would be permanent, as Ban would make it. It // or CauseAttack, made at now would be permanent, as Ban would make it. It
// works out nothing else of the ban. // works out nothing else of the ban.
func (l *Ledger) WouldBePermanent( func (l *Ledger) WouldBePermanent(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause Cause,
) bool { ) bool {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -439,7 +444,7 @@ func (l *Ledger) BanForAbuseIPDB(
// The source is named as the request log names AbuseIPDB. // The source is named as the request log names AbuseIPDB.
notes.Reputation = []ReputationHit{ notes.Reputation = []ReputationHit{
{Source: CauseAbuseIPDB, Score: &score}, {Source: string(CauseAbuseIPDB), Score: &score},
} }
l.add(Ban{ l.add(Ban{
@@ -577,7 +582,7 @@ func (l *Ledger) AddLookup(
// for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for // for CauseLimit, CauseAttack and CauseAbuseIPDB, by the ledger; for
// CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as // CauseAdmin, by an admin, with BanForAdmin or in an edit of bans.json, as
// LoadEdit counts them. The bans read from bans.json at the start are not among them. // LoadEdit counts them. The bans read from bans.json at the start are not among them.
func (l *Ledger) Made(cause string) int { func (l *Ledger) Made(cause Cause) int {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
@@ -770,8 +775,10 @@ func (l *Ledger) ban(
case CauseAttack: case CauseAttack:
ban.Reason = attackReason(notes) ban.Reason = attackReason(notes)
ban.Expires = l.attackExpiry(held, now) ban.Expires = l.attackExpiry(held, now)
case CauseAdmin, CauseAbuseIPDB:
fallthrough
default: default:
panic("bans: a request's ban has the cause " + grounds.Cause) panic("bans: a request's ban has the cause " + string(grounds.Cause))
} }
if !keep { if !keep {
+10 -5
View File
@@ -10,6 +10,8 @@ import (
"time" "time"
"github.com/spf13/viper" "github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// RProxy is the settings of `sww rproxy`. A size or time limit of zero // RProxy is the settings of `sww rproxy`. A size or time limit of zero
@@ -32,7 +34,7 @@ type RProxy struct {
// cannot be connected to, does not answer at all, or does not answer // cannot be connected to, does not answer at all, or does not answer
// within WAFBackendTimeout: allow or deny // within WAFBackendTimeout: allow or deny
// (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION). // (SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION).
WAFBackendNoAnswerDisposition string WAFBackendNoAnswerDisposition wafprotocol.Disposition
// MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and // MaxRequestSize is the largest request body (SWW_RPROXY_MAX_REQUEST_SIZE), and
// MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE). // MaxResponseSize the largest response body (SWW_RPROXY_MAX_RESPONSE_SIZE).
MaxRequestSize int64 MaxRequestSize int64
@@ -94,7 +96,7 @@ func RProxyFromEnvironment(v *viper.Viper) (*RProxy, error) {
), ),
WAFBackendNoAnswerDisposition: env.disposition( WAFBackendNoAnswerDisposition: env.disposition(
"SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION", "SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION",
"allow", wafprotocol.DispositionAllow,
), ),
MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"), MaxRequestSize: env.size("SWW_RPROXY_MAX_REQUEST_SIZE", "100M"),
MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"), MaxResponseSize: env.size("SWW_RPROXY_MAX_RESPONSE_SIZE", "1G"),
@@ -166,9 +168,12 @@ func (e *environment) wafBackend(name string) *url.URL {
// disposition reads the setting that is what is done with a request when waf // disposition reads the setting that is what is done with a request when waf
// cannot be connected to, does not answer at all, or does not answer in time: // cannot be connected to, does not answer at all, or does not answer in time:
// allow or deny. // allow or deny.
func (e *environment) disposition(name, defaultValue string) string { func (e *environment) disposition(
value := e.value(name, defaultValue) name string, defaultValue wafprotocol.Disposition,
if value != "allow" && value != "deny" { ) wafprotocol.Disposition {
value := wafprotocol.Disposition(e.value(name, string(defaultValue)))
if value != wafprotocol.DispositionAllow &&
value != wafprotocol.DispositionDeny {
e.check(name, fmt.Errorf("%q %w", value, errNotDisposition)) e.check(name, fmt.Errorf("%q %w", value, errNotDisposition))
} }
+19 -10
View File
@@ -305,7 +305,7 @@ type WAF struct {
AlertSlackWebhookURL *url.URL AlertSlackWebhookURL *url.URL
AlertNtfyURL *url.URL AlertNtfyURL *url.URL
AlertNtfyToken string AlertNtfyToken string
AlertEvents []string AlertEvents []alerts.Event
AlertCooldown time.Duration AlertCooldown time.Duration
AlertMaxPerHour int AlertMaxPerHour int
// The anomaly thresholds, which only raise alerts: the most requests // The anomaly thresholds, which only raise alerts: the most requests
@@ -789,8 +789,7 @@ func WAFFromEnvironment(v *viper.Viper) (*WAF, error) {
), ),
AlertNtfyURL: env.webhookURL("SWW_WAF_ALERT_NTFY_URL"), AlertNtfyURL: env.webhookURL("SWW_WAF_ALERT_NTFY_URL"),
AlertNtfyToken: env.secret("SWW_WAF_ALERT_NTFY_TOKEN"), AlertNtfyToken: env.secret("SWW_WAF_ALERT_NTFY_TOKEN"),
AlertEvents: env.alertEvents("SWW_WAF_ALERT_EVENTS", AlertEvents: env.alertEvents("SWW_WAF_ALERT_EVENTS"),
strings.Join(alerts.Events(), ",")),
AlertCooldown: env.duration("SWW_WAF_ALERT_COOLDOWN", "15m"), AlertCooldown: env.duration("SWW_WAF_ALERT_COOLDOWN", "15m"),
AlertMaxPerHour: env.numberOrOff( AlertMaxPerHour: env.numberOrOff(
"SWW_WAF_ALERT_MAX_PER_HOUR", "SWW_WAF_ALERT_MAX_PER_HOUR",
@@ -1686,9 +1685,15 @@ func (e *environment) secret(name string) string {
return value return value
} }
// alertEvents reads the setting that is the events alerts are sent for. // alertEvents reads the setting that is the events alerts are sent for,
func (e *environment) alertEvents(name, defaultValue string) []string { // every event by default.
events, err := parseAlertEvents(e.value(name, defaultValue)) func (e *environment) alertEvents(name string) []alerts.Event {
every := make([]string, 0, len(alerts.Events()))
for _, event := range alerts.Events() {
every = append(every, string(event))
}
events, err := parseAlertEvents(e.value(name, strings.Join(every, ",")))
e.check(name, err) e.check(name, err)
return events return events
@@ -2548,16 +2553,20 @@ func parseWebhookHeaders(value string) (http.Header, string, error) {
// parseAlertEvents reads a comma-separated list of the events alerts can // parseAlertEvents reads a comma-separated list of the events alerts can
// be sent for. // be sent for.
func parseAlertEvents(value string) ([]string, error) { func parseAlertEvents(value string) ([]alerts.Event, error) {
events, err := parseList(value) names, err := parseList(value)
if err != nil { if err != nil {
return nil, err return nil, err
} }
for _, event := range events { events := make([]alerts.Event, 0, len(names))
for _, name := range names {
event := alerts.Event(name)
if !slices.Contains(alerts.Events(), event) { if !slices.Contains(alerts.Events(), event) {
return nil, fmt.Errorf("%q %w", event, errNotAlertEvent) return nil, fmt.Errorf("%q %w", name, errNotAlertEvent)
} }
events = append(events, event)
} }
return events, nil return events, nil
+16 -2
View File
@@ -20,6 +20,7 @@ import (
"github.com/spf13/viper" "github.com/spf13/viper"
"golang.org/x/net/http/httpguts" "golang.org/x/net/http/httpguts"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/config/configtest" "sneak.berlin/go/smallwebwaf/internal/config/configtest"
@@ -188,6 +189,16 @@ const (
defaultAlertCooldown = "15m" defaultAlertCooldown = "15m"
) )
// joinEvents returns events as SWW_WAF_ALERT_EVENTS lists them.
func joinEvents(events []alerts.Event) string {
names := make([]string, 0, len(events))
for _, event := range events {
names = append(names, string(event))
}
return strings.Join(names, ",")
}
// defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL. // defaultIPReputationCacheTTL is the default of SWW_WAF_IP_REPUTATION_CACHE_TTL.
const defaultIPReputationCacheTTL = "24h" const defaultIPReputationCacheTTL = "24h"
@@ -1110,7 +1121,7 @@ func TestAlertSettingsDefaults(t *testing.T) {
if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 || if cfg.AlertWebhookURL != nil || len(cfg.AlertWebhookHeaders) != 0 ||
cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil || cfg.AlertSlackWebhookURL != nil || cfg.AlertNtfyURL != nil ||
cfg.AlertNtfyToken != "" || cfg.AlertNtfyToken != "" ||
strings.Join(cfg.AlertEvents, ",") != defaultAlertEvents || joinEvents(cfg.AlertEvents) != defaultAlertEvents ||
cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 { cfg.AlertCooldown != 15*time.Minute || cfg.AlertMaxPerHour != 60 {
t.Errorf( t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+ "alert settings %v, %v, %v, %v, %q, %v, %s and %d, want no URLs, "+
@@ -1157,7 +1168,10 @@ func TestAlertSettingsAsSet(t *testing.T) {
!reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) || !reflect.DeepEqual(cfg.AlertWebhookHeaders, headers) ||
cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy || cfg.AlertSlackWebhookURL.String() != slack || cfg.AlertNtfyURL.String() != ntfy ||
cfg.AlertNtfyToken != token || cfg.AlertNtfyToken != token ||
!slices.Equal(cfg.AlertEvents, []string{"ban", "file_error"}) || !slices.Equal(
cfg.AlertEvents,
[]alerts.Event{"ban", "file_error"},
) ||
cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 { cfg.AlertCooldown != time.Hour || cfg.AlertMaxPerHour != 10 {
t.Errorf( t.Errorf(
"alert settings %v, %v, %v, %v, %q, %v, %s and %d", "alert settings %v, %v, %v, %v, %q, %v, %s and %d",
+396 -269
View File
@@ -15,12 +15,34 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/remotelog"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules" "sneak.berlin/go/smallwebwaf/internal/rules"
) )
// Params are what New builds the metrics from. Each part is nil while it is
// off, and a part that is nil adds no metric of its own.
type Params struct {
// BusiestCount is how many countries and how many AS numbers get series
// of their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS).
BusiestCount int
// InstanceName is the label instance every metric carries
// (SWW_WAF_INSTANCE_NAME).
InstanceName string
// Now returns the time at which Bans counts the bans active.
Now func() time.Time
Bans banLedger
Clients sized
Rules sized
Lists fetchedLists
DNSBL dnsblZones
AbuseIPDB abuseIPDBChecker
Alerts alertQueue
LookupFile lookupDatabase
RemoteLog remoteLogSender
}
// Metrics are sww's metrics. They are safe for concurrent use. // Metrics are sww's metrics. They are safe for concurrent use.
type Metrics struct { type Metrics struct {
// registry gives every metric registered with it the label instance. // registry gives every metric registered with it the label instance.
@@ -36,8 +58,6 @@ type Metrics struct {
rateLimitHits *prometheus.CounterVec rateLimitHits *prometheus.CounterVec
offences *prometheus.CounterVec offences *prometheus.CounterVec
coreRuleSetMatches *prometheus.CounterVec coreRuleSetMatches *prometheus.CounterVec
// ruleMatches are made by AddRules, and reputationHits by
// AddReputation.
ruleMatches *prometheus.CounterVec ruleMatches *prometheus.CounterVec
reputationHits *prometheus.CounterVec reputationHits *prometheus.CounterVec
countries *busiest countries *busiest
@@ -59,20 +79,18 @@ type Metrics struct {
stateFileEditsSetAside *prometheus.CounterVec stateFileEditsSetAside *prometheus.CounterVec
} }
// New returns the metrics, with the Go runtime's and the process's own. // New returns the metrics, with the Go runtime's and the process's own, and
// busiestCount is how many countries and how many AS numbers get series of // those read from each part in params as the metrics are asked for.
// their own (SWW_WAF_METRICS_BUSIEST_COUNTRIES_AND_ASNS). Every metric
// carries instanceName (SWW_WAF_INSTANCE_NAME) as its label instance.
// //
//nolint:funlen // a few lines for each metric, a list that grows with them //nolint:funlen // a few lines for each metric, a list that grows with them
func New(busiestCount int, instanceName string) *Metrics { func New(params Params) *Metrics {
byStatus := []string{"status_class", "action"} byStatus := []string{"status_class", "action"}
byFile := []string{"file"} byFile := []string{"file"}
registry := prometheus.NewRegistry() registry := prometheus.NewRegistry()
m := &Metrics{ m := &Metrics{
registry: prometheus.WrapRegistererWith( registry: prometheus.WrapRegistererWith(
prometheus.Labels{"instance": instanceName}, registry), prometheus.Labels{"instance": params.InstanceName}, registry),
handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}), handler: promhttp.HandlerFor(registry, promhttp.HandlerOpts{}),
inFlight: prometheus.NewGauge(prometheus.GaugeOpts{ inFlight: prometheus.NewGauge(prometheus.GaugeOpts{
Name: "sww_requests_in_flight", Name: "sww_requests_in_flight",
@@ -113,8 +131,19 @@ func New(busiestCount int, instanceName string) *Metrics {
"and the rule's id.", "and the rule's id.",
[]string{"mode", "rule_id"}, []string{"mode", "rule_id"},
), ),
countries: newCountries(busiestCount), ruleMatches: counterVec(
asns: newASNs(busiestCount), "sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
),
reputationHits: counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
),
countries: newCountries(params.BusiestCount),
asns: newASNs(params.BusiestCount),
GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{ GeoJSRequests: prometheus.NewCounter(prometheus.CounterOpts{
Name: "sww_geojs_requests_total", Name: "sww_geojs_requests_total",
Help: "Requests to GeoJS.", Help: "Requests to GeoJS.",
@@ -166,6 +195,7 @@ func New(busiestCount int, instanceName string) *Metrics {
m.inFlight, m.requests, m.requestBytes, m.responseBytes, m.inFlight, m.requests, m.requestBytes, m.responseBytes,
m.requestDuration, m.upstreamDuration, m.requestDuration, m.upstreamDuration,
m.rateLimitHits, m.offences, m.coreRuleSetMatches, m.rateLimitHits, m.offences, m.coreRuleSetMatches,
m.ruleMatches, m.reputationHits,
m.countries, m.asns, m.countries, m.asns,
m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer, m.GeoJSRequests, m.GeoJSFailures, m.RequestsWithoutGeoJSAnswer,
m.stateFileWrites, m.stateFileWriteFailures, m.stateFileWrites, m.stateFileWriteFailures,
@@ -173,261 +203,45 @@ func New(busiestCount int, instanceName string) *Metrics {
m.stateFileEditsTakenIn, m.stateFileEditsSetAside, m.stateFileEditsTakenIn, m.stateFileEditsSetAside,
) )
if params.Bans != nil {
m.addBans(params.Bans, params.Now)
}
if params.Clients != nil {
m.addClients(params.Clients)
}
if params.Rules != nil {
m.addRules(params.Rules)
}
if params.Lists != nil {
m.addLists(params.Lists)
}
if params.DNSBL != nil {
m.addDNSBL(params.DNSBL)
}
if params.AbuseIPDB != nil {
m.addAbuseIPDB(params.AbuseIPDB)
}
if params.Alerts != nil {
m.addAlerts(params.Alerts)
}
if params.LookupFile != nil {
m.addLookupFile(params.LookupFile)
}
if params.RemoteLog != nil {
m.addRemoteLog(params.RemoteLog)
}
return m return m
} }
// AddBansAndClients adds the metrics read from the ledger and the table
// of clients as the metrics are asked for: the bans made since the start,
// by cause, the bans active and permanent at now, and the clients in the
// table.
func (m *Metrics) AddBansAndClients(
ledger *bans.Ledger, limiter *ratelimit.Limiter, now func() time.Time,
) {
for _, cause := range []string{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": cause},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(limiter.Len())
}),
)
}
// AddRules adds the metrics of the rule files: the requests that matched
// each rule, which RuleMatched counts, and the rules loaded from
// ruleFiles, read as the metrics are asked for. It is called once, before
// RuleMatched.
func (m *Metrics) AddRules(ruleFiles *rules.Files) {
m.ruleMatches = counterVec(
"sww_rule_matches_total",
"Requests that matched a rule of the rule files, by its id and action.",
[]string{"rule_id", "action"},
)
m.registry.MustRegister(m.ruleMatches,
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(ruleFiles.Len())
}))
}
// AddRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) AddRemoteLog(remote *remotelog.Sender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// AddLookupFile adds the metrics of the lookup database, read as the
// metrics are asked for: when the file in use was read, which lastRead
// returns, and the replacements of it that could not be read, which
// readFailures returns.
func (m *Metrics) AddLookupFile(
lastRead func() time.Time,
readFailures func() int,
) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(lastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(readFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// AddReputation adds the metrics of the lists fetched from URLs and of the
// DNSBL zones, by source, each list's URL or each zone, its key masked as
// reputation.MaskZoneKey masks it: the requests whose client a blocklist, a
// zone's verdict or AbuseIPDB's score lists, which ReputationHit counts,
// and, read from lists and dnsbl as the metrics are asked for, for a list,
// the fetches that failed and when the copy in use was fetched, and for a
// zone, the queries made and those that failed. It is called once, before
// ReputationHit.
func (m *Metrics) AddReputation(
lists *reputation.Lists,
dnsbl *reputation.DNSBL,
) {
m.reputationHits = counterVec(
"sww_reputation_hits_total",
"Requests whose client a blocklist, a DNSBL zone or AbuseIPDB lists, "+
"by the list's URL, the zone, or abuseipdb.",
[]string{sourceLabel},
)
m.registry.MustRegister(m.reputationHits)
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// AddAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left. It is
// called once, after AddReputation, while SWW_WAF_ABUSEIPDB_KEY is set.
func (m *Metrics) AddAbuseIPDB(abuseIPDB *reputation.AbuseIPDB) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// AddAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) AddAlerts(queue *alerts.Queue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": name}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// ServeHTTP answers with the metrics in the Prometheus text format. // ServeHTTP answers with the metrics in the Prometheus text format.
func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (m *Metrics) ServeHTTP(w http.ResponseWriter, r *http.Request) {
m.handler.ServeHTTP(w, r) m.handler.ServeHTTP(w, r)
@@ -466,7 +280,7 @@ func (m *Metrics) RequestReported(
// LimitHit counts a request that broke a rate limit, a byte limit or the // LimitHit counts a request that broke a rate limit, a byte limit or the
// refusal limit, by the window and the kind of hit. // refusal limit, by the window and the kind of hit.
func (m *Metrics) LimitHit(hit ratelimit.Hit) { func (m *Metrics) LimitHit(hit ratelimit.Hit) {
m.rateLimitHits.WithLabelValues(hit.Window, hit.Kind).Inc() m.rateLimitHits.WithLabelValues(hit.Window, string(hit.Kind)).Inc()
} }
// Offences counts the offences of r, a request that has ended, as its // Offences counts the offences of r, a request that has ended, as its
@@ -487,8 +301,8 @@ func (m *Metrics) Offences(r ratelimit.Request) {
// RuleMatched counts a request that matched the rule id, whose action is // RuleMatched counts a request that matched the rule id, whose action is
// action. // action.
func (m *Metrics) RuleMatched(id, action string) { func (m *Metrics) RuleMatched(id string, action rules.Action) {
m.ruleMatches.WithLabelValues(id, action).Inc() m.ruleMatches.WithLabelValues(id, string(action)).Inc()
} }
// CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id, // CoreRuleSetMatched counts a request that matched the Core Rule Set's rule id,
@@ -497,6 +311,12 @@ func (m *Metrics) CoreRuleSetMatched(mode string, id int) {
m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc() m.coreRuleSetMatches.WithLabelValues(mode, strconv.Itoa(id)).Inc()
} }
// ReputationHit counts a request whose client source lists: a blocklist,
// by its URL, a DNSBL zone, its key masked, or AbuseIPDB, abuseipdb.
func (m *Metrics) ReputationHit(source string) {
m.reputationHits.WithLabelValues(source).Inc()
}
// StateFileWritten counts a write of the state or cache file name, of size // StateFileWritten counts a write of the state or cache file name, of size
// bytes, that ended with err. // bytes, that ended with err.
func (m *Metrics) StateFileWritten(name string, size int, err error) { func (m *Metrics) StateFileWritten(name string, size int, err error) {
@@ -528,16 +348,323 @@ func (m *Metrics) StateFileEditSetAside(name string) {
m.stateFileEditsSetAside.WithLabelValues(name).Inc() m.stateFileEditsSetAside.WithLabelValues(name).Inc()
} }
// banLedger is what the metrics read from the ledger of bans.
type banLedger interface {
// Made returns the bans made since the start with cause.
Made(cause bans.Cause) int
// Count returns the bans active at now, the permanent ones included,
// and the permanent ones.
Count(now time.Time) (active, permanent int)
}
// addBans adds the metrics read from ledger as the metrics are asked for:
// the bans made since the start, by cause, and the bans active and
// permanent at now.
func (m *Metrics) addBans(ledger banLedger, now func() time.Time) {
for _, cause := range []bans.Cause{
bans.CauseLimit, bans.CauseAttack, bans.CauseAdmin, bans.CauseAbuseIPDB,
} {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_bans_made_total",
Help: "Bans made, by cause.",
ConstLabels: prometheus.Labels{"cause": string(cause)},
}, func() float64 {
return float64(ledger.Made(cause))
}),
)
}
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_active_bans",
Help: "Bans active now, the permanent ones included.",
}, func() float64 {
active, _ := ledger.Count(now())
return float64(active)
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_permanent_bans",
Help: "Permanent bans not lifted.",
}, func() float64 {
_, permanent := ledger.Count(now())
return float64(permanent)
}),
)
}
// sized is what the metrics read from the table of clients and from the
// rule files.
type sized interface {
// Len returns how many clients the table holds, or how many rules were
// loaded from the rule files.
Len() int
}
// addClients adds the metric of the clients in table, read as the metrics
// are asked for.
func (m *Metrics) addClients(table sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_tracked_clients",
Help: "Clients in the table of clients.",
}, func() float64 {
return float64(table.Len())
}),
)
}
// addRules adds the metric of the rules loaded from files, read as the
// metrics are asked for.
func (m *Metrics) addRules(files sized) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_rules_loaded",
Help: "Rules loaded from the rule files.",
}, func() float64 {
return float64(files.Len())
}))
}
// remoteLogSender is what the metrics read from what sends the log lines to
// SWW_WAF_LOG_REMOTE_URL.
type remoteLogSender interface {
// Sent returns the lines sent.
Sent() int64
// Dropped returns the lines dropped.
Dropped() int64
// Depth returns the lines waiting in the buffer.
Depth() int
}
// addRemoteLog adds the metrics of sending the log lines to
// SWW_WAF_LOG_REMOTE_URL, read from remote as the metrics are asked for: the
// lines sent, those dropped, and those waiting in the buffer.
func (m *Metrics) addRemoteLog(remote remoteLogSender) {
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_sent_total",
Help: "Log lines sent to SWW_WAF_LOG_REMOTE_URL.",
}, func() float64 {
return float64(remote.Sent())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_remote_log_lines_dropped_total",
Help: "Log lines dropped: the oldest in a full buffer, and those " +
"whose sending failed.",
}, func() float64 {
return float64(remote.Dropped())
}),
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_remote_log_lines_waiting",
Help: "Log lines in the buffer, waiting to be sent.",
}, func() float64 {
return float64(remote.Depth())
}),
)
}
// lookupDatabase is what the metrics read from the lookup database.
type lookupDatabase interface {
// LastRead returns when the file in use was read.
LastRead() time.Time
// ReadFailures returns the replacements of the file that could not be
// read.
ReadFailures() int
}
// addLookupFile adds the metrics of the lookup database, read from file as
// the metrics are asked for: when the file in use was read, and the
// replacements of it that could not be read.
func (m *Metrics) addLookupFile(file lookupDatabase) {
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_lookup_database_last_read_timestamp_seconds",
Help: "When the lookup database in use was read, in seconds since 1970.",
}, func() float64 {
return float64(file.LastRead().Unix())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_lookup_database_read_failures_total",
Help: "Replacements of the lookup database that could not be read.",
}, func() float64 {
return float64(file.ReadFailures())
}),
)
}
// sourceLabel is the label of the reputation metrics: a list's URL, a
// DNSBL zone, its key masked, or abuseipdb.
const sourceLabel = "source"
// dnsblZones is what the metrics read from what asks the DNSBL zones.
type dnsblZones interface {
// Zones returns the zones asked, their keys unmasked.
Zones() []string
// Queries returns the queries made to zone.
Queries(zone string) int
// Failures returns the queries to zone that failed.
Failures(zone string) int
}
// addDNSBL adds the metrics of each zone dnsbl asks, by source, the zone,
// its key masked as reputation.MaskZoneKey masks it, read as the metrics are
// asked for: the queries made and those that failed.
func (m *Metrics) addDNSBL(dnsbl dnsblZones) {
for _, zone := range dnsbl.Zones() {
source := prometheus.Labels{
sourceLabel: reputation.MaskZoneKey(zone),
}
m.addReputationQueries(
source,
func() int { return dnsbl.Queries(zone) },
)
m.addReputationFailures(
source,
func() int { return dnsbl.Failures(zone) },
)
}
}
// fetchedLists is what the metrics read from the lists fetched from URLs.
type fetchedLists interface {
// URLs returns the lists' URLs.
URLs() []string
// Failures returns the fetches of the list at listURL that failed.
Failures(listURL string) int
// Fetched returns when the copy in use of the list at listURL was
// fetched, or the zero time while there is none.
Fetched(listURL string) time.Time
}
// addLists adds the metrics of each list fetched from a URL, by source, the
// list's URL, read from lists as the metrics are asked for: the fetches
// that failed and when the copy in use was fetched.
func (m *Metrics) addLists(lists fetchedLists) {
for _, listURL := range lists.URLs() {
source := prometheus.Labels{sourceLabel: listURL}
m.addReputationFailures(
source,
func() int { return lists.Failures(listURL) },
)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_last_fetch_timestamp_seconds",
Help: "When the copy of the list in use was fetched, in seconds since " +
"1970, or 0 while there is none.",
ConstLabels: source,
}, func() float64 {
fetched := lists.Fetched(listURL)
if fetched.IsZero() {
return 0
}
return float64(fetched.Unix())
}),
)
}
}
// abuseIPDBChecker is what the metrics read from what checks clients with
// AbuseIPDB.
type abuseIPDBChecker interface {
// Checked returns the checks made.
Checked() int
// Failures returns the checks that failed.
Failures() int
// BudgetLeft returns how many checks the day's budget has left.
BudgetLeft() int
}
// addAbuseIPDB adds the metrics of AbuseIPDB, with the source abuseipdb,
// read from abuseIPDB as the metrics are asked for: the checks made, those
// that failed, and how many checks the day's budget has left.
func (m *Metrics) addAbuseIPDB(abuseIPDB abuseIPDBChecker) {
source := prometheus.Labels{sourceLabel: reputation.AbuseIPDBSource}
m.addReputationQueries(source, abuseIPDB.Checked)
m.addReputationFailures(source, abuseIPDB.Failures)
m.registry.MustRegister(
prometheus.NewGaugeFunc(prometheus.GaugeOpts{
Name: "sww_reputation_daily_budget_remaining",
Help: "Checks of the day's SWW_WAF_ABUSEIPDB_DAILY_BUDGET not yet spent.",
ConstLabels: source,
}, func() float64 {
return float64(abuseIPDB.BudgetLeft())
}),
)
}
// alertQueue is what the metrics read from the queue of alerts.
type alertQueue interface {
// DestinationsSet returns the names of the destinations set.
DestinationsSet() []alerts.DestinationName
// Counts returns the alerts sent to the destination name, the requests
// to it that failed, and the alerts dropped.
Counts(name alerts.DestinationName) alerts.Counts
// Suppressed returns the alerts held back.
Suppressed() int64
}
// addAlerts adds the metrics of the alerts sent to each destination set,
// read from queue as the metrics are asked for, by destination: the
// alerts sent, the requests to the destination that failed, the alerts
// held back, which are the same for every destination, and those
// dropped. With no destination set, it adds none.
func (m *Metrics) addAlerts(queue alertQueue) {
for _, name := range queue.DestinationsSet() {
destination := prometheus.Labels{"destination": string(name)}
m.registry.MustRegister(
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_sent_total",
Help: "Alerts the destination took.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Sent)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alert_send_failures_total",
Help: "Requests to the destination that failed.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).SendFailures)
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_suppressed_total",
Help: "Alerts held back: repeats within SWW_WAF_ALERT_COOLDOWN, and " +
"alerts past SWW_WAF_ALERT_MAX_PER_HOUR, for the hour's summary.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Suppressed())
}),
prometheus.NewCounterFunc(prometheus.CounterOpts{
Name: "sww_alerts_dropped_total",
Help: "Alerts dropped, the oldest first, from a full queue, and " +
"alerts given up as the destination refused them.",
ConstLabels: destination,
}, func() float64 {
return float64(queue.Counts(name).Dropped)
}),
)
}
}
// count counts a request that has ended, as RequestEnded and RequestReported // count counts a request that has ended, as RequestEnded and RequestReported
// describe. // describe.
func (m *Metrics) count( func (m *Metrics) count(
line *requestlog.Line, duration, upstreamDuration time.Duration, line *requestlog.Line, duration, upstreamDuration time.Duration,
) { ) {
class := statusClass(line.Status) class := statusClass(line.Status)
m.requests.WithLabelValues(class, line.Action).Inc() action := string(line.Action)
m.requestBytes.WithLabelValues(class, line.Action). m.requests.WithLabelValues(class, action).Inc()
m.requestBytes.WithLabelValues(class, action).
Add(float64(line.RequestBytes)) Add(float64(line.RequestBytes))
m.responseBytes.WithLabelValues(class, line.Action). m.responseBytes.WithLabelValues(class, action).
Add(float64(line.ResponseBytes)) Add(float64(line.ResponseBytes))
m.requestDuration.Observe(duration.Seconds()) m.requestDuration.Observe(duration.Seconds())
+161
View File
@@ -0,0 +1,161 @@
package metrics_test
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/metrics"
)
const (
listURL = "https://lists.example/block.txt"
zone = "dnsbl.example"
destination = "webhook"
)
// Each stand-in answers with a number of its own, so that each metric
// shows which part it was read from.
type ledgerStandIn struct{}
func (ledgerStandIn) Made(bans.Cause) int { return 1 }
func (ledgerStandIn) Count(time.Time) (int, int) { return 2, 1 }
type clientTableStandIn struct{}
func (clientTableStandIn) Len() int { return 3 }
type ruleFilesStandIn struct{}
func (ruleFilesStandIn) Len() int { return 4 }
type listsStandIn struct{}
func (listsStandIn) URLs() []string { return []string{listURL} }
func (listsStandIn) Failures(string) int { return 5 }
func (listsStandIn) Fetched(string) time.Time { return time.Time{} }
type dnsblStandIn struct{}
func (dnsblStandIn) Zones() []string { return []string{zone} }
func (dnsblStandIn) Queries(string) int { return 6 }
func (dnsblStandIn) Failures(string) int { return 0 }
type abuseIPDBStandIn struct{}
func (abuseIPDBStandIn) Checked() int { return 0 }
func (abuseIPDBStandIn) Failures() int { return 0 }
func (abuseIPDBStandIn) BudgetLeft() int { return 7 }
type alertQueueStandIn struct{}
func (alertQueueStandIn) DestinationsSet() []alerts.DestinationName {
return []alerts.DestinationName{destination}
}
func (alertQueueStandIn) Counts(alerts.DestinationName) alerts.Counts {
return alerts.Counts{Sent: 8}
}
func (alertQueueStandIn) Suppressed() int64 { return 0 }
type lookupDatabaseStandIn struct{}
func (lookupDatabaseStandIn) LastRead() time.Time { return time.Time{} }
func (lookupDatabaseStandIn) ReadFailures() int { return 9 }
type remoteLogStandIn struct{}
func (remoteLogStandIn) Sent() int64 { return 10 }
func (remoteLogStandIn) Dropped() int64 { return 0 }
func (remoteLogStandIn) Depth() int { return 0 }
func TestMetricsAreReadFromEachPart(t *testing.T) {
t.Parallel()
m := metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
Now: func() time.Time { return time.Time{} },
Bans: ledgerStandIn{},
Clients: clientTableStandIn{},
Rules: ruleFilesStandIn{},
Lists: listsStandIn{},
DNSBL: dnsblStandIn{},
AbuseIPDB: abuseIPDBStandIn{},
Alerts: alertQueueStandIn{},
LookupFile: lookupDatabaseStandIn{},
RemoteLog: remoteLogStandIn{},
})
scraped := scrape(t, m)
for _, line := range []string{
`sww_bans_made_total{cause="limit",instance="app"} 1`,
`sww_active_bans{instance="app"} 2`,
`sww_tracked_clients{instance="app"} 3`,
`sww_rules_loaded{instance="app"} 4`,
`sww_reputation_failures_total{instance="app",source="` + listURL + `"} 5`,
`sww_reputation_queries_total{instance="app",source="` + zone + `"} 6`,
`sww_reputation_daily_budget_remaining{instance="app",source="abuseipdb"} 7`,
`sww_alerts_sent_total{destination="` + destination + `",instance="app"} 8`,
`sww_lookup_database_read_failures_total{instance="app"} 9`,
`sww_remote_log_lines_sent_total{instance="app"} 10`,
} {
if !strings.Contains(scraped, "\n"+line+"\n") {
t.Errorf("metrics\n%s\nwant\n%s", scraped, line)
}
}
}
func TestPartThatIsOffAddsNoMetric(t *testing.T) {
t.Parallel()
scraped := scrape(t, metrics.New(metrics.Params{
BusiestCount: 1,
InstanceName: "app",
}))
for _, name := range []string{
"sww_bans_made_total",
"sww_tracked_clients",
"sww_rules_loaded",
"sww_reputation_",
"sww_alert",
"sww_lookup_database_",
"sww_remote_log_",
} {
if strings.Contains(scraped, name) {
t.Errorf("metrics\n%s\nshow %s", scraped, name)
}
}
}
// scrape returns the metrics m serves.
func scrape(t *testing.T, m *metrics.Metrics) string {
t.Helper()
scraped := httptest.NewRecorder()
m.ServeHTTP(
scraped,
httptest.NewRequestWithContext(
t.Context(), http.MethodGet, "/", http.NoBody,
),
)
return scraped.Body.String()
}
+8 -6
View File
@@ -19,15 +19,19 @@ import (
const day = 24 * time.Hour const day = 24 * time.Hour
// LimitKind is what a limit counts: KindRequests, KindBytes or
// KindRefusals.
type LimitKind string
// The kinds of limits, as the metrics name them. // The kinds of limits, as the metrics name them.
const ( const (
// KindRequests is a rate limit, on a client's requests. // KindRequests is a rate limit, on a client's requests.
KindRequests = "requests" KindRequests LimitKind = "requests"
// KindBytes is a byte limit, on a client's bytes. // KindBytes is a byte limit, on a client's bytes.
KindBytes = "bytes" KindBytes LimitKind = "bytes"
// KindRefusals is the refusal limit, on a client's requests sww // KindRefusals is the refusal limit, on a client's requests sww
// refused after a rule file match or for a missing or wrong token. // refused after a rule file match or for a missing or wrong token.
KindRefusals = "refusals" KindRefusals LimitKind = "refusals"
) )
// Limits are the most requests a client may make in a minute, an hour and // Limits are the most requests a client may make in a minute, an hour and
@@ -200,9 +204,7 @@ func New(params Params) *Limiter {
// Hit is a request that takes a client over a rate limit or the refusal // Hit is a request that takes a client over a rate limit or the refusal
// limit, or whose bytes take it over a byte limit. // limit, or whose bytes take it over a byte limit.
type Hit struct { type Hit struct {
// Kind is KindRequests for a rate limit, KindBytes for a byte limit, Kind LimitKind
// KindRefusals for the refusal limit.
Kind string
// Window is "minute", "hour" or "day". // Window is "minute", "hour" or "day".
Window string Window string
// Limit is the window's limit, as the client's percentage of it. // Limit is the window's limit, as the client's percentage of it.
+1 -1
View File
@@ -652,7 +652,7 @@ func wantBytesCount(
client netip.Prefix, client netip.Prefix,
now time.Time, now time.Time,
bytes int64, bytes int64,
want string, want ratelimit.LimitKind,
) { ) {
t.Helper() t.Helper()
+9 -12
View File
@@ -445,12 +445,9 @@ func TestKeyIsSentInTheKeyHeaderAndNeverShown(t *testing.T) {
scores: map[string]int64{suspect: 100}, scores: map[string]int64{suspect: 100},
} }
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation( BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
reputation.New(params()), })
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one it refuses with an answer // One check that AbuseIPDB answers, and one it refuses with an answer
// that names the key. // that names the key.
@@ -503,12 +500,9 @@ func TestMetricsCountTheChecksTheFailuresAndTheBudgetLeft(t *testing.T) {
p := abuseIPDBParams() p := abuseIPDBParams()
p.DailyBudget = 5 p.DailyBudget = 5
checker := newAbuseIPDB(abuseIPDB, p) checker := newAbuseIPDB(abuseIPDB, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation( BusiestCount: 1, InstanceName: instanceName, AbuseIPDB: checker,
reputation.New(params()), })
reputation.NewDNSBL(dnsblParams()),
)
m.AddAbuseIPDB(checker)
// One check that AbuseIPDB answers, and one that fails. // One check that AbuseIPDB answers, and one that fails.
hitFrom(t, checker, suspect, true) hitFrom(t, checker, suspect, true)
@@ -811,6 +805,9 @@ func wantBudgetLeft(t *testing.T, checker *reputation.AbuseIPDB, want int) {
} }
} }
// instanceName is the label instance of the metrics the tests build.
const instanceName = "app"
// scrapeMetrics returns the metrics m serves. // scrapeMetrics returns the metrics m serves.
func scrapeMetrics(t *testing.T, m *metrics.Metrics) string { func scrapeMetrics(t *testing.T, m *metrics.Metrics) string {
t.Helper() t.Helper()
+6 -4
View File
@@ -358,8 +358,9 @@ func TestMetricsCountEachZonesQueriesAndThoseThatFailed(t *testing.T) {
"99.2.0.192." + otherZone + ".": {rcode: serverFailure}, "99.2.0.192." + otherZone + ".": {rcode: serverFailure},
}} }}
dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone)) dnsbl := newDNSBL(resolver, dnsblParams(zone, otherZone))
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation(reputation.New(params()), dnsbl) BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
wantZones(t, dnsbl, listed) wantZones(t, dnsbl, listed)
synctest.Wait() synctest.Wait()
@@ -404,8 +405,9 @@ func TestZoneKeyIsMaskedInTheVerdictsTheFailuresAndTheMetrics(t *testing.T) {
"99.2.0.192." + keyed + ".": {addrs: []string{listing}}, "99.2.0.192." + keyed + ".": {addrs: []string{listing}},
"100.2.0.192." + keyed + ".": {rcode: serverFailure}, "100.2.0.192." + keyed + ".": {rcode: serverFailure},
}}, p) }}, p)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{
m.AddReputation(reputation.New(params()), dnsbl) BusiestCount: 1, InstanceName: instanceName, DNSBL: dnsbl,
})
// Both clients are asked about before either answer comes, so that // Both clients are asked about before either answer comes, so that
// the failure does not keep the zone from the other query. // the failure does not keep the zone from the other query.
+33 -23
View File
@@ -12,82 +12,94 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// Action is what waf did with a request, as its request line names it,
// such as ActionForward.
type Action string
// The action a request line names: what waf did with the request. // The action a request line names: what waf did with the request.
const ( const (
// ActionForward is a request waf allowed, which rproxy passes to // ActionForward is a request waf allowed, which rproxy passes to
// the app. // the app.
ActionForward = "forward" ActionForward Action = "forward"
// ActionRequestLineTooLong is a request whose request line is too long for // ActionRequestLineTooLong is a request whose request line is too long for
// the header limits. // the header limits.
ActionRequestLineTooLong = "request_line_too_long" ActionRequestLineTooLong Action = "request_line_too_long"
// ActionRequestHeaderTooLarge is a request whose headers are over the // ActionRequestHeaderTooLarge is a request whose headers are over the
// header limits. // header limits.
ActionRequestHeaderTooLarge = "request_header_too_large" ActionRequestHeaderTooLarge Action = "request_header_too_large"
// ActionClientIsRateLimited is a request refused because it took its // ActionClientIsRateLimited is a request refused because it took its
// client over a rate limit, which bans the client. // client over a rate limit, which bans the client.
ActionClientIsRateLimited = "client_is_rate_limited" ActionClientIsRateLimited Action = "client_is_rate_limited"
// ActionTemporarilyBanned is a request refused because a ban that runs // ActionTemporarilyBanned is a request refused because a ban that runs
// out covers its client, or because it matched a ban rule or asked for a // out covers its client, or because it matched a ban rule or asked for a
// trap path, and so made such a ban. // trap path, and so made such a ban.
ActionTemporarilyBanned = "temporarily_banned" ActionTemporarilyBanned Action = "temporarily_banned"
// ActionPermanentlyBanned is a request refused because a ban that never // ActionPermanentlyBanned is a request refused because a ban that never
// runs out covers its client, or because the request made such a ban or // runs out covers its client, or because the request made such a ban or
// made its client's ban permanent. // made its client's ban permanent.
ActionPermanentlyBanned = "permanently_banned" ActionPermanentlyBanned Action = "permanently_banned"
// ActionBlockedByBlockRule is a request refused because it matched a // ActionBlockedByBlockRule is a request refused because it matched a
// block rule. // block rule.
ActionBlockedByBlockRule = "blocked_by_block_rule" ActionBlockedByBlockRule Action = "blocked_by_block_rule"
// ActionBlockedByCoreRuleSet is a request refused because the Core Rule // ActionBlockedByCoreRuleSet is a request refused because the Core Rule
// Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD. // Set scored it at or over SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD.
ActionBlockedByCoreRuleSet = "blocked_by_core_rule_set" ActionBlockedByCoreRuleSet Action = "blocked_by_core_rule_set"
// ActionClientDenied is a request refused because its client is in // ActionClientDenied is a request refused because its client is in
// SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny, // SWW_WAF_DENY_NETS, in a blocklist while SWW_WAF_BLOCKLIST_ACTION is deny,
// or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while // or listed by a DNSBL zone, or scored a hit by AbuseIPDB, while
// SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in // SWW_WAF_DNSBL_ABUSEIPDB_ACTION is deny, or that carries a tag in
// SWW_WAF_TAG_DENY. // SWW_WAF_TAG_DENY.
ActionClientDenied = "client_denied" ActionClientDenied Action = "client_denied"
// ActionCountryDenied is a request refused because its client's country // ActionCountryDenied is a request refused because its client's country
// is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES. // is not in SWW_WAF_EXCLUSIVELY_ALLOWED_COUNTRIES.
ActionCountryDenied = "country_denied" ActionCountryDenied Action = "country_denied"
// ActionChecksTimedOut is a request refused because the trap paths, the // ActionChecksTimedOut is a request refused because the trap paths, the
// rule files and the Core Rule Set had not finished with it within 500 // rule files and the Core Rule Set had not finished with it within 500
// milliseconds. // milliseconds.
ActionChecksTimedOut = "checks_timed_out" ActionChecksTimedOut Action = "checks_timed_out"
// ActionOwnEndpoint is a request waf answered at one of its own // ActionOwnEndpoint is a request waf answered at one of its own
// endpoints, or a question about a path under /_sww/. // endpoints, or a question about a path under /_sww/.
ActionOwnEndpoint = "own_endpoint" ActionOwnEndpoint Action = "own_endpoint"
) )
// Offence is what a request was held as, as its request line names it.
type Offence string
// OffenceLimit is the offence a request line names for a request that // OffenceLimit is the offence a request line names for a request that
// broke a rate limit or the refusal limit, or whose bytes broke a byte // broke a rate limit or the refusal limit, or whose bytes broke a byte
// limit. // limit.
const OffenceLimit = "limit" const OffenceLimit Offence = "limit"
// LimitHitMinuteRefusals is the limit_hit a request line names for a request // LimitHitMinuteRefusals is the limit_hit a request line names for a request
// that broke the refusal limit. // that broke the refusal limit.
const LimitHitMinuteRefusals = "minute_refusals" const LimitHitMinuteRefusals = "minute_refusals"
// WAFLineKind is which of waf's two lines about a question a request line
// is: WAFLineDecision or WAFLineReport.
type WAFLineKind string
// The waf_line a request line names, which says which of // The waf_line a request line names, which says which of
// waf's two lines about a question it is. // waf's two lines about a question it is.
const ( const (
// WAFLineDecision is waf's line at its decision. // WAFLineDecision is waf's line at its decision.
WAFLineDecision = "decision" WAFLineDecision WAFLineKind = "decision"
// WAFLineReport is waf's line when rproxy's report // WAFLineReport is waf's line when rproxy's report
// comes, with the decision's fields and the report's together. // comes, with the decision's fields and the report's together.
WAFLineReport = "report" WAFLineReport WAFLineKind = "report"
) )
// timeLayout is RFC 3339 with milliseconds. // timeLayout is RFC 3339 with milliseconds.
@@ -100,9 +112,8 @@ const timeLayout = "2006-01-02T15:04:05.000Z07:00"
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case //nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Line struct { type Line struct {
Type string `json:"type"` Type string `json:"type"`
// WAFLine is WAFLineDecision or WAFLineReport // WAFLine is set on waf's two lines about a question.
// on waf's two lines about a question. WAFLine WAFLineKind `json:"waf_line,omitempty"`
WAFLine string `json:"waf_line,omitempty"`
// The standard web log fields. Scheme is how the client reached rproxy, or // The standard web log fields. Scheme is how the client reached rproxy, or
// the trusted proxy in front of it. // the trusted proxy in front of it.
@@ -148,7 +159,7 @@ type Line struct {
Aborted bool `json:"aborted,omitempty"` Aborted bool `json:"aborted,omitempty"`
// The decision. // The decision.
Action string `json:"action"` Action Action `json:"action"`
// WouldAction is, in observe mode, the action enforce mode would have // WouldAction is, in observe mode, the action enforce mode would have
// taken with a request it would have refused: ActionClientDenied, // taken with a request it would have refused: ActionClientDenied,
// ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied, // ActionTemporarilyBanned, ActionPermanentlyBanned, ActionCountryDenied,
@@ -156,7 +167,7 @@ type Line struct {
// ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or // ActionBlockedByCoreRuleSet, or, in waf, ActionRequestLineTooLong or
// ActionRequestHeaderTooLarge for the header limits, or // ActionRequestHeaderTooLarge for the header limits, or
// ActionChecksTimedOut. // ActionChecksTimedOut.
WouldAction string `json:"would_action,omitempty"` WouldAction Action `json:"would_action,omitempty"`
// Heuristic is the name of the heuristic that refused the request, in // Heuristic is the name of the heuristic that refused the request, in
// observe mode too, and HeuristicFigures are the figures it acted on, by // observe mode too, and HeuristicFigures are the figures it acted on, by
// name. // name.
@@ -203,8 +214,7 @@ type Line struct {
// request got no tags from routewatch: no_answer_yet, request_failed or // request got no tags from routewatch: no_answer_yet, request_failed or
// no_route. // no_route.
RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"` RouteWatchTagsMissing string `json:"routewatch_tags_missing,omitempty"`
// Offence is the offence the request was held as, OffenceLimit. Offence Offence `json:"offence,omitempty"`
Offence string `json:"offence,omitempty"`
// BanExpires is when the ban the request made, or was refused under, // BanExpires is when the ban the request made, or was refused under,
// ends: a time, or "permanent". // ends: a time, or "permanent".
BanExpires string `json:"ban_expires,omitempty"` BanExpires string `json:"ban_expires,omitempty"`
@@ -246,7 +256,7 @@ type RProxyLine struct {
// WAFBackendNoAnswer is true, the value of // WAFBackendNoAnswer is true, the value of
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one. // gave one.
Disposition string `json:"disposition,omitempty"` Disposition wafprotocol.Disposition `json:"disposition,omitempty"`
WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"` WAFBackendNoAnswer bool `json:"waf_backend_no_answer,omitempty"`
} }
+9 -1
View File
@@ -1,6 +1,14 @@
package rproxy package rproxy
import "time" import (
"net/http"
"time"
)
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
}
// StopBreakerClock has the circuit breaker of s read at as the time, so that how // StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves. // long requests take cannot matter to it. It is called before s serves.
+5 -5
View File
@@ -56,7 +56,7 @@ func TestServerHasTheFixedLimits(t *testing.T) {
InstanceName: "test", InstanceName: "test",
Level: slog.LevelInfo, Level: slog.LevelInfo,
}), }),
}) }).HTTPServer()
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute { server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
@@ -709,7 +709,7 @@ func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) {
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != http.StatusNoContent { if w.Code != http.StatusNoContent {
t.Errorf("status %d, want the app's 204", w.Code) t.Errorf("status %d, want the app's 204", w.Code)
@@ -762,7 +762,7 @@ func TestABodyLeftUnreadIsReadNoMoreOnceTheTimeToSendTheAnswerRunsOut(
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != http.StatusOK { if w.Code != http.StatusOK {
t.Errorf("status %d, want the app's 200", w.Code) t.Errorf("status %d, want the app's 200", w.Code)
@@ -815,7 +815,7 @@ func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) {
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.Code != tc.status { if w.Code != tc.status {
t.Errorf("status %d, want %d", w.Code, tc.status) t.Errorf("status %d, want %d", w.Code, tc.status)
@@ -856,7 +856,7 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
before := time.Now() before := time.Now()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
if w.writeDeadline.Before(before.Add(least)) { if w.writeDeadline.Before(before.Add(least)) {
t.Errorf( t.Errorf(
+1 -1
View File
@@ -583,7 +583,7 @@ func TestAQuestionNamesNoZone(t *testing.T) {
req.RemoteAddr = tc.remoteAddr req.RemoteAddr = tc.remoteAddr
req.Header.Set(forwardedForHeader, tc.forwardedFor) req.Header.Set(forwardedForHeader, tc.forwardedFor)
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
+31 -163
View File
@@ -12,7 +12,6 @@ import (
"net/url" "net/url"
"os" "os"
"strconv" "strconv"
"sync"
"sync/atomic" "sync/atomic"
"time" "time"
@@ -63,7 +62,7 @@ type request struct {
// disposition is what is done with the request: waf's valid disposition, // disposition is what is done with the request: waf's valid disposition,
// or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither // or SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION. It is "" when neither
// gave one. // gave one.
disposition string disposition wafprotocol.Disposition
// answered is true when waf gave disposition, which makes a report due, // answered is true when waf gave disposition, which makes a report due,
// and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did. // and wafNoAnswer when SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION did.
answered bool answered bool
@@ -82,17 +81,9 @@ type request struct {
// upgraded is the connection to the app once the app has switched protocols, // upgraded is the connection to the app once the app has switched protocols,
// as for a WebSocket, and nil otherwise. // as for a WebSocket, and nil otherwise.
upgraded *upgradedConn upgraded *upgradedConn
// limits are the request's SWW_RPROXY_MAX_REQUEST_TIME_MS and
// mu guards what follows. The time limits run on goroutines of their own, // SWW_RPROXY_MAX_RESPONSE_TIME_MS.
// and the transport starts and stops them, and notes requestEnd, from its limits *timeLimits
// own; once timersStopped is set, neither time limit acts any more.
mu sync.Mutex
timersStopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
} }
// newRequest starts handling r: it notes the time, works out the visitor from // newRequest starts handling r: it notes the time, works out the visitor from
@@ -127,12 +118,25 @@ func (h *handler) newRequest(
} }
} }
rq.limits = &timeLimits{
maxRequestTime: h.config.MaxRequestTime,
maxResponseTime: h.config.MaxResponseTime,
start: rq.start,
refuse: rq.refuse,
waitingForVisitor: func() bool {
return rq.body != nil && rq.body.waiting.Load()
},
endVisitorRead: func() {
_ = rq.rc.SetReadDeadline(time.Now())
},
}
return rq return rq
} }
// answerHealth answers the health check. // answerHealth answers the health check.
func (rq *request) answerHealth() { func (rq *request) answerHealth() {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.limits.requestDeadline())
rq.out.Header().Set("Content-Type", plainText) rq.out.Header().Set("Content-Type", plainText)
rq.send(http.StatusOK, []byte("ok\n")) rq.send(http.StatusOK, []byte("ok\n"))
} }
@@ -148,7 +152,7 @@ func (rq *request) readFirstPartOfBody() bool {
return true return true
} }
_ = rq.rc.SetReadDeadline(rq.requestDeadline()) _ = rq.rc.SetReadDeadline(rq.limits.requestDeadline())
held, err := io.ReadAll( held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes), io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
) )
@@ -188,7 +192,7 @@ func (rq *request) forward(ctx context.Context) {
rq.cancel = cancel rq.cancel = cancel
ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{ ctx = httptrace.WithClientTrace(ctx, &httptrace.ClientTrace{
WroteRequest: rq.wroteRequest, WroteRequest: rq.limits.wroteRequest,
}) })
out := rq.in.WithContext(ctx) out := rq.in.WithContext(ctx)
@@ -205,7 +209,7 @@ func (rq *request) forward(ctx context.Context) {
ErrorHandler: rq.answerError, ErrorHandler: rq.answerError,
} }
rq.startRequestTimer() rq.limits.startRequestTimer()
rq.upstreamStart = time.Now() rq.upstreamStart = time.Now()
reverseProxy.ServeHTTP(rq.out, out) reverseProxy.ServeHTTP(rq.out, out)
} }
@@ -248,7 +252,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
// switch, as when the app's Upgrade is not the visitor's, so the status // switch, as when the app's Upgrade is not the visitor's, so the status
// is noted only once it has taken over the connection, by rq.out's // is noted only once it has taken over the connection, by rq.out's
// Hijack. // Hijack.
rq.stopTimers() rq.limits.stop()
conn, ok := res.Body.(io.ReadWriteCloser) conn, ok := res.Body.(io.ReadWriteCloser)
if ok { if ok {
@@ -259,7 +263,7 @@ func (rq *request) modifyResponse(res *http.Response) error {
return nil return nil
} }
requestEnd := rq.endRequest() requestEnd := rq.limits.endRequest()
// An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not // An answer that arrives once SWW_RPROXY_MAX_RESPONSE_TIME_MS has run out is not
// passed on: the visitor is answered 504. // passed on: the visitor is answered 504.
@@ -289,12 +293,12 @@ func (rq *request) modifyResponse(res *http.Response) error {
// An answer that arrives before the app has been sent the whole request ends // An answer that arrives before the app has been sent the whole request ends
// the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts. // the request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
rq.startResponseTimer(requestEnd) rq.limits.startResponseTimer(requestEnd)
res.Body = &responseBody{ res.Body = &responseBody{
body: limitBody(res.Body, maxBytes), body: limitBody(res.Body, maxBytes),
refuse: rq.refuse, refuse: rq.refuse,
responseReceived: rq.responseReceived, responseReceived: rq.limits.stop,
ctx: rq.in.Context(), ctx: rq.in.Context(),
} }
rq.out.afterStatus = rq.setResponseDeadline rq.out.afterStatus = rq.setResponseDeadline
@@ -347,7 +351,7 @@ func (rq *request) answer(status int) {
// A visitor found too slow is read no more; any other may go on sending until // A visitor found too slow is read no more; any other may go on sending until
// its time is up, so that the rest of the body is read and the request ends // its time is up, so that the rest of the body is read and the request ends
// cleanly: by Go's server, or by readRestOfBody. // cleanly: by Go's server, or by readRestOfBody.
deadline := rq.requestDeadline() deadline := rq.limits.requestDeadline()
if status == http.StatusRequestTimeout { if status == http.StatusRequestTimeout {
deadline = time.Now() deadline = time.Now()
} }
@@ -393,7 +397,7 @@ func (rq *request) closeAfterAnswer() {
// used that time up first, and at least minSendTime. // used that time up first, and at least minSendTime.
func (rq *request) prepareOwnAnswer(deadline time.Time) { func (rq *request) prepareOwnAnswer(deadline time.Time) {
rq.stopReadingBody(deadline) rq.stopReadingBody(deadline)
rq.endRequest() rq.limits.endRequest()
limit := rq.h.config.MaxResponseTime limit := rq.h.config.MaxResponseTime
if limit > 0 { if limit > 0 {
@@ -414,7 +418,7 @@ func (rq *request) refuse(status int) {
// finish ends the request's time limits, writes its log line and, if waf // finish ends the request's time limits, writes its log line and, if waf
// answered the question, sends the report. ctx is the request's own context. // answered the question, sends the report. ctx is the request's own context.
func (rq *request) finish(ctx context.Context) { func (rq *request) finish(ctx context.Context) {
rq.stopTimers() rq.limits.stop()
// After a cut-off answer, the reading of a body that has not arrived whole // After a cut-off answer, the reading of a body that has not arrived whole
// ends now. Go's server would otherwise keep the connection open while it // ends now. Go's server would otherwise keep the connection open while it
@@ -432,10 +436,10 @@ func (rq *request) finish(ctx context.Context) {
errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded): errors.Is(rq.rc.Flush(), os.ErrDeadlineExceeded):
rq.stopPassingOnBody(time.Now()) rq.stopPassingOnBody(time.Now())
case rq.refused.Load() == 0: case rq.refused.Load() == 0:
rq.stopPassingOnBody(rq.requestDeadline()) rq.stopPassingOnBody(rq.limits.requestDeadline())
} }
requestEnd := rq.endRequest() requestEnd := rq.limits.endRequest()
end := time.Now() end := time.Now()
requestBytes, responseBytes := rq.bytesEachWay() requestBytes, responseBytes := rq.bytesEachWay()
@@ -509,17 +513,6 @@ func (rq *request) bytesEachWay() (int64, int64) {
return requestBytes, responseBytes return requestBytes, responseBytes
} }
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (rq *request) requestDeadline() time.Time {
limit := rq.h.config.MaxRequestTime
if limit == 0 {
return time.Time{}
}
return rq.start.Add(limit)
}
// stopReadingBody ends, at deadline, the reading of a visitor's body that has not // stopReadingBody ends, at deadline, the reading of a visitor's body that has not
// arrived whole: Go's server then reads no more of it, and closes the connection // arrived whole: Go's server then reads no more of it, and closes the connection
// after the answer. // after the answer.
@@ -576,100 +569,6 @@ func (rq *request) stopPassingOnBody(deadline time.Time) {
rq.stopReadingBody(deadline) rq.stopReadingBody(deadline)
} }
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS, which runs from the
// request's headers being read until the app has been sent the whole request.
func (rq *request) startRequestTimer() {
deadline := rq.requestDeadline()
if deadline.IsZero() {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
rq.requestTimer = time.AfterFunc(
time.Until(deadline),
rq.requestTimedOut,
)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while the
// request is still on its way to the app. The answer names the side rproxy was
// waiting on at that moment: 408 when it was waiting for the visitor to send more
// of its body, 504 when it was waiting for the app to be reached or to take what
// it had.
func (rq *request) requestTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped {
return
}
if rq.body == nil || !rq.body.waiting.Load() {
rq.refuse(http.StatusGatewayTimeout)
return
}
rq.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
_ = rq.rc.SetReadDeadline(time.Now())
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS starts.
func (rq *request) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := rq.endRequest()
rq.mu.Lock()
stopTimer(rq.requestTimer)
rq.mu.Unlock()
rq.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (rq *request) startResponseTimer(requestEnd time.Time) {
limit := rq.h.config.MaxResponseTime
if limit == 0 {
return
}
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.timersStopped || rq.responseTimer != nil {
return
}
rq.responseTimer = time.AfterFunc(time.Until(requestEnd.Add(limit)),
rq.responseTimedOut)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out before the
// app has sent its whole answer.
func (rq *request) responseTimedOut() {
rq.mu.Lock()
defer rq.mu.Unlock()
if !rq.timersStopped {
rq.refuse(http.StatusGatewayTimeout)
}
}
// responseReceived is called once the app has sent its whole answer.
func (rq *request) responseReceived() {
rq.stopTimers()
}
// setResponseDeadline sends the status and headers of the app's answer, with // setResponseDeadline sends the status and headers of the app's answer, with
// at least minSendTime, before ReverseProxy reads the answer's body, and then // at least minSendTime, before ReverseProxy reads the answer's body, and then
// the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request. // the rest has until SWW_RPROXY_MAX_RESPONSE_TIME_MS after the end of the request.
@@ -679,7 +578,7 @@ func (rq *request) setResponseDeadline() {
return return
} }
deadline := rq.endRequest().Add(limit) deadline := rq.limits.endRequest().Add(limit)
_ = rq.rc.SetWriteDeadline( _ = rq.rc.SetWriteDeadline(
time.Now().Add(max(time.Until(deadline), minSendTime)), time.Now().Add(max(time.Until(deadline), minSendTime)),
@@ -687,34 +586,3 @@ func (rq *request) setResponseDeadline() {
_ = rq.rc.Flush() _ = rq.rc.Flush()
_ = rq.rc.SetWriteDeadline(deadline) _ = rq.rc.SetWriteDeadline(deadline)
} }
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (rq *request) endRequest() time.Time {
rq.mu.Lock()
defer rq.mu.Unlock()
if rq.requestEnd.IsZero() {
rq.requestEnd = time.Now()
}
return rq.requestEnd
}
// stopTimers stops the request's time limits and keeps any from starting later:
// the app's answer is complete, the connection upgraded, or the request handled.
func (rq *request) stopTimers() {
rq.mu.Lock()
defer rq.mu.Unlock()
rq.timersStopped = true
stopTimer(rq.requestTimer)
stopTimer(rq.responseTimer)
}
// stopTimer stops t, which is nil when its time limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+32 -8
View File
@@ -16,6 +16,7 @@ import (
"io" "io"
"log" "log"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"net/url" "net/url"
"slices" "slices"
@@ -63,8 +64,7 @@ type Params struct {
// Server is the server `sww rproxy` runs. // Server is the server `sww rproxy` runs.
type Server struct { type Server struct {
*http.Server httpServer *http.Server
h *handler h *handler
} }
@@ -94,17 +94,40 @@ func New(params Params) *Server {
breakerClock: time.Now, breakerClock: time.Now,
} }
return &Server{ s := &Server{h: h}
Server: &http.Server{ s.httpServer = &http.Server{
Addr: params.Config.ListenAddr, Addr: params.Config.ListenAddr,
Handler: h, Handler: s,
ErrorLog: errorLog, ErrorLog: errorLog,
ReadHeaderTimeout: params.Config.MaxRequestTime, ReadHeaderTimeout: params.Config.MaxRequestTime,
IdleTimeout: idleTimeout, IdleTimeout: idleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes, MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
},
h: h,
} }
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request from a visitor, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
} }
// WaitForReports returns once the reports being sent have gone or failed, // WaitForReports returns once the reports being sent have gone or failed,
@@ -208,7 +231,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
// This panic has Go's server close the connection without an answer, // This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs. // and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler) panic(http.ErrAbortHandler)
default: // a deny, an error, or an answer without a valid disposition case wafprotocol.DispositionDeny, "":
// "" is an error, or no valid disposition
rq.answerWith(answer) rq.answerWith(answer)
} }
} }
+4 -4
View File
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers. // What the stand-in for waf answers.
const ( const (
dispositionAllow = wafprotocol.DispositionAllow dispositionAllow = string(wafprotocol.DispositionAllow)
dispositionDeny = wafprotocol.DispositionDeny dispositionDeny = string(wafprotocol.DispositionDeny)
dispositionClose = wafprotocol.DispositionClose dispositionClose = string(wafprotocol.DispositionClose)
// deniedBody is the body of a deny. // deniedBody is the body of a deny.
deniedBody = "refused by waf" deniedBody = "refused by waf"
) )
@@ -710,7 +710,7 @@ func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
) )
w := httptest.NewRecorder() w := httptest.NewRecorder()
server.Handler.ServeHTTP(w, req) server.ServeHTTP(w, req)
// Go's server guesses a type when the first part of the body goes before the // Go's server guesses a type when the first part of the body goes before the
// headers are flushed, which is a matter of timing here. So this looks for what // headers are flushed, which is a matter of timing here. So this looks for what
+165
View File
@@ -0,0 +1,165 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync"
"time"
)
// timeLimits are one request's two time limits: SWW_RPROXY_MAX_REQUEST_TIME_MS,
// which runs from the request's headers being read until the app has been sent
// the whole request, and SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from then
// until the app has sent its whole answer. A limit of zero is off. Each limit
// runs on a goroutine of its own, and the transport starts and stops them from
// its own.
type timeLimits struct {
maxRequestTime time.Duration
maxResponseTime time.Duration
// start is when the request's headers had been read.
start time.Time
// refuse refuses the request with 408 or 504 when a limit runs out.
refuse func(status int)
// waitingForVisitor reports whether a read of the visitor's body is waiting
// for the visitor to send more.
waitingForVisitor func() bool
// endVisitorRead ends at once a read of the visitor's body that is waiting.
endVisitorRead func()
// mu guards what follows. Once stopped is set, neither limit acts any more.
mu sync.Mutex
stopped bool
requestTimer *time.Timer
responseTimer *time.Timer
// requestEnd is when the app had been sent the whole request, or the answer
// started, if that was sooner.
requestEnd time.Time
}
// requestDeadline is when the visitor must have sent its whole request, or zero
// when SWW_RPROXY_MAX_REQUEST_TIME_MS is off.
func (l *timeLimits) requestDeadline() time.Time {
if l.maxRequestTime == 0 {
return time.Time{}
}
return l.start.Add(l.maxRequestTime)
}
// startRequestTimer starts SWW_RPROXY_MAX_REQUEST_TIME_MS.
func (l *timeLimits) startRequestTimer() {
deadline := l.requestDeadline()
if deadline.IsZero() {
return
}
l.mu.Lock()
defer l.mu.Unlock()
l.requestTimer = time.AfterFunc(time.Until(deadline), l.requestTimedOut)
}
// requestTimedOut is called when SWW_RPROXY_MAX_REQUEST_TIME_MS runs out while
// the request is still on its way to the app. The answer names the side rproxy
// was waiting on at that moment: 408 when it was waiting for the visitor to send
// more of its body, 504 when it was waiting for the app to be reached or to take
// what it had.
func (l *timeLimits) requestTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped {
return
}
if !l.waitingForVisitor() {
l.refuse(http.StatusGatewayTimeout)
return
}
l.refuse(http.StatusRequestTimeout)
// The transport gives up on the app only once its Read of the visitor's body
// returns, so that Read is ended now. The lock keeps this from reaching the
// connection after the request is handled.
l.endVisitorRead()
}
// wroteRequest is called once the app has been sent the whole request:
// SWW_RPROXY_MAX_REQUEST_TIME_MS ends and SWW_RPROXY_MAX_RESPONSE_TIME_MS
// starts.
func (l *timeLimits) wroteRequest(info httptrace.WroteRequestInfo) {
if info.Err != nil {
return // the transport gives up, or tries again
}
requestEnd := l.endRequest()
l.mu.Lock()
stopTimer(l.requestTimer)
l.mu.Unlock()
l.startResponseTimer(requestEnd)
}
// startResponseTimer starts SWW_RPROXY_MAX_RESPONSE_TIME_MS, which runs from
// requestEnd, unless it has started already.
func (l *timeLimits) startResponseTimer(requestEnd time.Time) {
if l.maxResponseTime == 0 {
return
}
l.mu.Lock()
defer l.mu.Unlock()
if l.stopped || l.responseTimer != nil {
return
}
l.responseTimer = time.AfterFunc(
time.Until(requestEnd.Add(l.maxResponseTime)),
l.responseTimedOut,
)
}
// responseTimedOut is called when SWW_RPROXY_MAX_RESPONSE_TIME_MS runs out
// before the app has sent its whole answer.
func (l *timeLimits) responseTimedOut() {
l.mu.Lock()
defer l.mu.Unlock()
if !l.stopped {
l.refuse(http.StatusGatewayTimeout)
}
}
// endRequest notes now as the end of the request, unless an end is noted, and
// returns the end noted.
func (l *timeLimits) endRequest() time.Time {
l.mu.Lock()
defer l.mu.Unlock()
if l.requestEnd.IsZero() {
l.requestEnd = time.Now()
}
return l.requestEnd
}
// stop stops both limits and keeps either from starting later: the app's
// answer is complete, the connection upgraded, or the request handled.
func (l *timeLimits) stop() {
l.mu.Lock()
defer l.mu.Unlock()
l.stopped = true
stopTimer(l.requestTimer)
stopTimer(l.responseTimer)
}
// stopTimer stops t, which is nil when its limit is off.
func stopTimer(t *time.Timer) {
if t != nil {
t.Stop()
}
}
+171
View File
@@ -0,0 +1,171 @@
package rproxy
import (
"net/http"
"net/http/httptrace"
"sync/atomic"
"testing"
"testing/synctest"
"time"
)
// The tests below run timeLimits in a synctest bubble, where time is a clock of
// the test's own: time.Sleep moves it on at once, and synctest.Wait returns once
// a limit that ran out has refused the request.
const (
testRequestTime = 3 * time.Second
testResponseTime = 5 * time.Second
)
// limitsSeen is what a timeLimits under test did to its request.
type limitsSeen struct {
// refused is the status of the last refusal, or zero.
refused atomic.Int64
// readEnded is true once the read of the visitor's body was ended.
readEnded atomic.Bool
// waiting is what waitingForVisitor reports.
waiting atomic.Bool
}
// newTestLimits returns time limits that started now, with what they do noted
// in seen.
func newTestLimits(seen *limitsSeen) *timeLimits {
return &timeLimits{
maxRequestTime: testRequestTime,
maxResponseTime: testResponseTime,
start: time.Now(),
refuse: func(status int) {
seen.refused.Store(int64(status))
},
waitingForVisitor: seen.waiting.Load,
endVisitorRead: func() {
seen.readEnded.Store(true)
},
}
}
// sleepThenWait moves the bubble's clock on by d and waits for what is then
// due.
func sleepThenWait(d time.Duration) {
time.Sleep(d)
synctest.Wait()
}
func TestRequestTimeRunningOutWhileTheVisitorIsWaitedOnRefuses408(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusRequestTimeout {
t.Errorf("refused %d, want 408", got)
}
if !seen.readEnded.Load() {
t.Error("the read of the visitor's body goes on, want it ended")
}
})
}
func TestRequestTimeRunningOutWhileTheAppIsWaitedOnRefuses504(
t *testing.T,
) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(testRequestTime)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
func TestResponseTimeRunsFromTheEndOfTheRequest(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
limits := newTestLimits(seen)
limits.startRequestTimer()
sleepThenWait(time.Second)
end := time.Now()
limits.wroteRequest(httptrace.WroteRequestInfo{})
if got := limits.endRequest(); !got.Equal(end) {
t.Errorf("the request ended at %v, want %v", got, end)
}
// Past SWW_RPROXY_MAX_REQUEST_TIME_MS, which the end of the request
// stopped, and just short of SWW_RPROXY_MAX_RESPONSE_TIME_MS.
sleepThenWait(testResponseTime - time.Millisecond)
if got := seen.refused.Load(); got != 0 {
t.Fatalf(
"refused %d before the limit ran out, want nothing",
got,
)
}
sleepThenWait(time.Millisecond)
if got := seen.refused.Load(); got != http.StatusGatewayTimeout {
t.Errorf("refused %d, want 504", got)
}
})
}
func TestStoppedLimitsRefuseNothing(t *testing.T) {
t.Parallel()
synctest.Test(t, func(t *testing.T) {
seen := &limitsSeen{}
seen.waiting.Store(true)
limits := newTestLimits(seen)
limits.startRequestTimer()
limits.stop()
limits.startResponseTimer(limits.endRequest())
sleepThenWait(testRequestTime + testResponseTime)
if got := seen.refused.Load(); got != 0 {
t.Errorf("refused %d, want nothing", got)
}
if seen.readEnded.Load() {
t.Error("the read of the visitor's body was ended, want it not")
}
})
}
+1 -1
View File
@@ -189,7 +189,7 @@ func canBeAsked(r *http.Request) bool {
// answerWith gives the visitor waf's answer, less its disposition. // answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) { func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline()) rq.prepareOwnAnswer(rq.limits.requestDeadline())
header := rq.out.Header() header := rq.out.Header()
maps.Copy(header, answer.header) maps.Copy(header, answer.header)
+10 -7
View File
@@ -25,15 +25,19 @@ import (
"sneak.berlin/go/smallwebwaf/internal/dirwatch" "sneak.berlin/go/smallwebwaf/internal/dirwatch"
) )
// Action is what a rule does when it matches: ActionLog, ActionBlock or
// ActionBan.
type Action string
// The actions a rule takes when it matches. // The actions a rule takes when it matches.
const ( const (
// ActionLog notes the match in the request log, and does nothing else. // ActionLog notes the match in the request log, and does nothing else.
ActionLog = "log" ActionLog Action = "log"
// ActionBlock refuses the request with 403. // ActionBlock refuses the request with 403.
ActionBlock = "block" ActionBlock Action = "block"
// ActionBan refuses the request and bans the client's netblock: the // ActionBan refuses the request and bans the client's netblock: the
// request is a clear sign of attack. // request is a clear sign of attack.
ActionBan = "ban" ActionBan Action = "ban"
) )
// extension ends the name of every rule file. // extension ends the name of every rule file.
@@ -88,8 +92,7 @@ type Rule struct {
// Target is what the regex is matched against, such as path or // Target is what the regex is matched against, such as path or
// header:Accept. // header:Accept.
Target string Target string
// Action is ActionLog, ActionBlock or ActionBan. Action Action
Action string
regex *regexp.Regexp regex *regexp.Regexp
} }
@@ -337,7 +340,7 @@ func parse(line string) (Rule, bool, error) {
return Rule{}, false, errNotRule return Rule{}, false, errNotRule
} }
rule := Rule{ID: fields[1], Target: fields[2], Action: fields[3]} rule := Rule{ID: fields[1], Target: fields[2], Action: Action(fields[3])}
headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget) headerName, isHeader := strings.CutPrefix(rule.Target, headerTarget)
switch { switch {
@@ -365,7 +368,7 @@ func parse(line string) (Rule, bool, error) {
rule.Target, rule.Target,
errHeaderTakenOut, errHeaderTakenOut,
) )
case !slices.Contains([]string{ActionLog, ActionBlock, ActionBan}, rule.Action): case !slices.Contains([]Action{ActionLog, ActionBlock, ActionBan}, rule.Action):
return Rule{}, false, fmt.Errorf( return Rule{}, false, fmt.Errorf(
"the action %q %w", "the action %q %w",
rule.Action, rule.Action,
+2 -2
View File
@@ -181,7 +181,7 @@ type BanEntry struct {
Netblock netip.Prefix `json:"netblock"` Netblock netip.Prefix `json:"netblock"`
Start time.Time `json:"start"` Start time.Time `json:"start"`
Expires *time.Time `json:"expires"` Expires *time.Time `json:"expires"`
Cause string `json:"cause"` Cause bans.Cause `json:"cause"`
Reason string `json:"reason,omitempty"` Reason string `json:"reason,omitempty"`
Lifted *time.Time `json:"lifted,omitempty"` Lifted *time.Time `json:"lifted,omitempty"`
Notes bans.Notes `json:"notes"` Notes bans.Notes `json:"notes"`
@@ -240,7 +240,7 @@ type alertsFile struct {
Version int `json:"version"` Version int `json:"version"`
Cooldowns []alerts.Cooldown `json:"cooldowns"` Cooldowns []alerts.Cooldown `json:"cooldowns"`
Hour alerts.Hour `json:"hour"` Hour alerts.Hour `json:"hour"`
Waiting map[string][]alerts.Alert `json:"waiting"` Waiting map[alerts.DestinationName][]alerts.Alert `json:"waiting"`
AnomalyCounters []anomaly.Counter `json:"anomaly_counters"` AnomalyCounters []anomaly.Counter `json:"anomaly_counters"`
} }
+4 -4
View File
@@ -1758,8 +1758,8 @@ func TestEditOfRouteWatchAndAlertsFilesTakenIn(t *testing.T) {
Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"), Event: alerts.EventBan, Netblock: netip.MustParsePrefix("198.51.100.0/24"),
Sent: midnight(), Sent: midnight(),
}}, }},
Hour: alerts.Hour{HeldBack: map[string]int{}}, Hour: alerts.Hour{HeldBack: map[alerts.Event]int{}},
Waiting: map[string][]alerts.Alert{ Waiting: map[alerts.DestinationName][]alerts.Alert{
alerts.DestinationWebhook: { alerts.DestinationWebhook: {
{Event: alerts.EventFileError, Time: midnight()}, {Event: alerts.EventFileError, Time: midnight()},
}, },
@@ -2046,7 +2046,7 @@ func TestBanWithoutACauseTakenInAsAnAdmins(t *testing.T) {
var written struct { var written struct {
Bans []struct { Bans []struct {
Cause string `json:"cause"` Cause bans.Cause `json:"cause"`
Reason string `json:"reason"` Reason string `json:"reason"`
} `json:"bans"` } `json:"bans"`
} }
@@ -2430,7 +2430,7 @@ func midnight() time.Time {
// counts, with thresholds fill does not reach. // counts, with thresholds fill does not reach.
func newParams(dir string) state.Params { func newParams(dir string) state.Params {
discard := slog.New(slog.DiscardHandler) discard := slog.New(slog.DiscardHandler)
m := metrics.New(1, "app") m := metrics.New(metrics.Params{BusiestCount: 1, InstanceName: "app"})
queue := alerts.New(alerts.Params{ queue := alerts.New(alerts.Params{
WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"}, WebhookURL: &url.URL{Scheme: "https", Host: "alerts.example"},
Events: alerts.Events(), Events: alerts.Events(),
+9 -1
View File
@@ -95,12 +95,20 @@ func background(loop func(context.Context)) fx.Hook {
) )
} }
// daemonServer is the server a daemon runs, waf.Server or rproxy.Server: the
// methods of Go's http.Server that serve starts and stops it with.
type daemonServer interface {
Serve(listener net.Listener) error
Shutdown(ctx context.Context) error
Close() error
}
// serve serves requests on listener from the start. At the stop it gives the // serve serves requests on listener from the start. At the stop it gives the
// requests in progress shutdownTimeout to finish, then closes their // requests in progress shutdownTimeout to finish, then closes their
// connections. When serving fails, it logs why and sends the error to // connections. When serving fails, it logs why and sends the error to
// failed. // failed.
func serve( func serve(
lc fx.Lifecycle, server *http.Server, listener net.Listener, lc fx.Lifecycle, server daemonServer, listener net.Listener,
processLog *slog.Logger, failed servingFailed, processLog *slog.Logger, failed servingFailed,
) { ) {
served := make(chan error, 1) served := make(chan error, 1)
+6 -2
View File
@@ -4,7 +4,6 @@ import (
"context" "context"
"log/slog" "log/slog"
"net" "net"
"net/http"
"os" "os"
"go.uber.org/fx" "go.uber.org/fx"
@@ -44,8 +43,13 @@ func RunRProxy(ctx context.Context, params Params) int {
app := fx.New( app := fx.New(
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed), fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
fx.Provide( fx.Provide(
// run waits for rproxy.Server's reports, and serve takes it as a
// daemonServer.
fx.Annotate(
newRProxyServer, newRProxyServer,
func(server *rproxy.Server) *http.Server { return server.Server }, fx.As(fx.Self()),
fx.As(new(daemonServer)),
),
newRProxyListener, newRProxyListener,
), ),
fx.Invoke(serve), fx.Invoke(serve),
+8 -8
View File
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer. // One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe) line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != wafprotocol.DispositionDeny || if line["disposition"] != string(wafprotocol.DispositionDeny) ||
line["client_ip"] != placed { line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK) wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop") line := out.line(t, "path", "/after-stop")
if line["disposition"] != wafprotocol.DispositionAllow || if line["disposition"] != string(wafprotocol.DispositionAllow) ||
line["waf_backend_no_answer"] != true { line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line) t.Errorf("rproxy's line is %v", line)
} }
@@ -608,8 +608,8 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
protocols <- r.Proto protocols <- r.Proto
w.Header(). w.Header().Set(wafprotocol.DispositionHeader,
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny) string(wafprotocol.DispositionDeny))
w.WriteHeader(http.StatusForbidden) w.WriteHeader(http.StatusForbidden)
} }
})) }))
@@ -692,8 +692,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) { func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" { if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header(). w.Header().Set(wafprotocol.DispositionHeader,
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow) string(wafprotocol.DispositionAllow))
return return
} }
@@ -795,8 +795,8 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc( backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) { func(w http.ResponseWriter, _ *http.Request) {
w.Header(). w.Header().Set(wafprotocol.DispositionHeader,
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow) string(wafprotocol.DispositionAllow))
})) }))
t.Cleanup(backend.Close) t.Cleanup(backend.Close)
+20 -14
View File
@@ -6,7 +6,6 @@ import (
"io" "io"
"log/slog" "log/slog"
"net" "net"
"net/http"
"time" "time"
"go.uber.org/fx" "go.uber.org/fx"
@@ -97,8 +96,8 @@ func RunWAF(ctx context.Context, params Params) int {
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile, newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
newLimiter, newLedger, newAnomalyCounters, newLimiter, newLedger, newAnomalyCounters,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch, newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newMetrics, newGeoJS, newWAFServer, newMetrics, newGeoJS,
func(server *waf.Server) *http.Server { return server.Server }, fx.Annotate(newWAFServer, fx.As(new(daemonServer))),
newWAFListener, newStateFiles, newWAFListener, newStateFiles,
), ),
// fx stops hooks in the reverse of the order they were added in. The // fx stops hooks in the reverse of the order they were added in. The
@@ -392,26 +391,33 @@ func newMetrics(
alertQueue *alerts.Queue, lookupFile *lookup.File, alertQueue *alerts.Queue, lookupFile *lookup.File,
remote *remotelog.Sender, remote *remotelog.Sender,
) *metrics.Metrics { ) *metrics.Metrics {
m := metrics.New(cfg.MetricsBusiestCountriesAndASNs, cfg.InstanceName) params := metrics.Params{
m.AddBansAndClients(ledger, limiter, utcNow) BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
m.AddRules(ruleFiles) InstanceName: cfg.InstanceName,
m.AddReputation(lists, dnsbl) Now: utcNow,
Bans: ledger,
if cfg.AbuseIPDBKey != "" { Clients: limiter,
m.AddAbuseIPDB(abuseIPDB) Rules: ruleFiles,
Lists: lists,
DNSBL: dnsbl,
Alerts: alertQueue,
} }
m.AddAlerts(alertQueue) // A nil pointer in an interface field is not a nil interface, so a part
// that is off is left out rather than set to nil.
if cfg.AbuseIPDBKey != "" {
params.AbuseIPDB = abuseIPDB
}
if lookupFile != nil { if lookupFile != nil {
m.AddLookupFile(lookupFile.LastRead, lookupFile.ReadFailures) params.LookupFile = lookupFile
} }
if remote != nil { if remote != nil {
m.AddRemoteLog(remote) params.RemoteLog = remote
} }
return m return metrics.New(params)
} }
// newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each // newGeoJS returns what looks clients up at GeoJS, none looked up yet. Each
+9 -4
View File
@@ -267,7 +267,12 @@ func runMain(t *testing.T, command string) (int, string, string) {
// ask sends waf at addr rproxy's question about a request for path from // ask sends waf at addr rproxy's question about a request for path from
// placed, checks the answer's status and disposition, and closes it. // placed, checks the answer's status and disposition, and closes it.
func ask(t *testing.T, addr, path string, status int, disposition string) { func ask(
t *testing.T,
addr, path string,
status int,
disposition wafprotocol.Disposition,
) {
t.Helper() t.Helper()
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet, req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
@@ -290,12 +295,12 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_, _ = io.Copy(io.Discard, res.Body) _, _ = io.Copy(io.Discard, res.Body)
_ = res.Body.Close() _ = res.Body.Close()
if res.StatusCode != status || answered := res.Header.Get(wafprotocol.DispositionHeader)
res.Header.Get(wafprotocol.DispositionHeader) != disposition { if res.StatusCode != status || answered != string(disposition) {
t.Errorf( t.Errorf(
"answered %d with disposition %q, want %d with %q", "answered %d with disposition %q, want %d with %q",
res.StatusCode, res.StatusCode,
res.Header.Get(wafprotocol.DispositionHeader), answered,
status, status,
disposition, disposition,
) )
+2 -2
View File
@@ -54,7 +54,7 @@ func TestClientsOnTheAbuseIPDBBlacklistAreBannedForTheBanDays(t *testing.T) {
}) })
// Two clients in one netblock make one ban, with the higher score. // Two clients in one netblock make one ban, with the higher score.
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{ server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 95}, {Addr: netip.MustParseAddr(client), Score: 95},
{Addr: netip.MustParseAddr(otherClient), Score: 100}, {Addr: netip.MustParseAddr(otherClient), Score: 100},
{Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90}, {Addr: netip.MustParseAddr("2001:db8:5::1"), Score: 90},
@@ -88,7 +88,7 @@ func TestAbuseIPDBBlacklistMakesNoBanInObserveMode(t *testing.T) {
map[string]string{mode: "observe"}, map[string]string{mode: "observe"},
) )
server.BanAbuseIPDBBlacklist([]reputation.BlacklistEntry{ server.banAbuseIPDBBlacklist([]reputation.BlacklistEntry{
{Addr: netip.MustParseAddr(client), Score: 100}, {Addr: netip.MustParseAddr(client), Score: 100},
}) })
+52 -107
View File
@@ -2,7 +2,6 @@ package waf
import ( import (
"bytes" "bytes"
"crypto/subtle"
"encoding/json" "encoding/json"
"errors" "errors"
"fmt" "fmt"
@@ -15,7 +14,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config" "sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
) )
@@ -41,80 +39,23 @@ var (
errNotAddress = errors.New("is not an address, such as 203.0.113.9") errNotAddress = errors.New("is not an address, such as 203.0.113.9")
) )
// answerHealth answers a request for the health endpoint. // handleHealth answers GET wafprotocol.HealthPath.
func (rq *request) answerHealth() { func handleHealth() http.HandlerFunc {
rq.line.Action = requestlog.ActionOwnEndpoint return func(w http.ResponseWriter, _ *http.Request) {
// Set here rather than left to Go's server, which would set it only // Set here rather than left to Go's server, which would set it only
// after the log line has taken the response's headers. // after the log line has taken the response's headers.
rq.out.Header().Set("Content-Type", "text/plain; charset=utf-8") w.Header().Set("Content-Type", "text/plain; charset=utf-8")
_, _ = io.WriteString(rq.out, "ok\n") _, _ = io.WriteString(w, "ok\n")
}
// answerAdmin answers a request for one of waf's own endpoints. Each
// needs a token, sent as Authorization: Bearer <token>: the metrics
// SWW_WAF_METRICS_TOKEN, the others SWW_WAF_ADMIN_TOKEN. A request without it is
// refused with 401. An endpoint whose token is unset answers 404, as any other
// path does.
func (rq *request) answerAdmin() {
rq.line.Action = requestlog.ActionOwnEndpoint
token, answer := rq.endpoint()
switch {
case token == "":
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
case !hasToken(rq.in, token):
rq.out.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
rq.out,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
answer()
} }
} }
// endpoint returns the token the request's endpoint needs, and what // handleListBans answers GET BansPath with every ban held.
// answers the request there; "" when there is no such endpoint. func (h *handler) handleListBans() http.HandlerFunc {
func (rq *request) endpoint() (string, func()) { return func(w http.ResponseWriter, _ *http.Request) {
cfg := rq.h.config h.answerBans(w, h.ledger.Snapshot())
method, path := rq.in.Method, rq.in.URL.Path
switch {
case method == http.MethodGet && path == MetricsPath:
return cfg.MetricsToken, func() { rq.h.metrics.ServeHTTP(rq.out, rq.in) }
case method == http.MethodGet && path == BansPath:
return cfg.AdminToken, rq.listBans
case method == http.MethodPost && path == BansPath:
return cfg.AdminToken, rq.addBan
case method == http.MethodDelete && strings.HasPrefix(path, BansPath+"/"):
return cfg.AdminToken, rq.liftBans
case method == http.MethodGet && strings.HasPrefix(path, ClientsPath):
return cfg.AdminToken, rq.showClient
default:
return "", nil
} }
} }
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// listBans answers GET BansPath with every ban held.
func (rq *request) listBans() {
rq.answerBans(rq.h.ledger.Snapshot())
}
// banToAdd is the body of POST BansPath. // banToAdd is the body of POST BansPath.
type banToAdd struct { type banToAdd struct {
// Netblock is a netblock, or a client's address, which stands for the // Netblock is a netblock, or a client's address, which stands for the
@@ -126,20 +67,21 @@ type banToAdd struct {
Reason string `json:"reason"` Reason string `json:"reason"`
} }
// addBan answers POST BansPath: it bans the netblock the body names, as // handleAddBan answers POST BansPath: it bans the netblock the body names,
// an admin, from now for the duration the body gives, with its reason, // as an admin, from now for the duration the body gives, with its reason,
// and answers with that ban. // and answers with that ban.
func (rq *request) addBan() { func (h *handler) handleAddBan() http.HandlerFunc {
toAdd, err := rq.readBanToAdd() return func(w http.ResponseWriter, r *http.Request) {
toAdd, err := readBanToAdd(r)
var ( var (
netblock netip.Prefix netblock netip.Prefix
expires time.Time expires time.Time
now = rq.h.now() now = h.now()
) )
if err == nil { if err == nil {
netblock, err = rq.h.parseBanNetblock(toAdd.Netblock) netblock, err = h.parseBanNetblock(toAdd.Netblock)
} }
if err == nil { if err == nil {
@@ -147,23 +89,22 @@ func (rq *request) addBan() {
} }
if err != nil { if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
} }
ban := rq.h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason) ban := h.ledger.BanForAdmin(netblock, now, expires, toAdd.Reason)
rq.answerBans([]bans.Ban{ban}) h.answerBans(w, []bans.Ban{ban})
}
} }
// readBanToAdd reads the body of POST BansPath: a JSON object with // readBanToAdd reads the body of POST BansPath: a JSON object with
// nothing but whitespace after it, in at most banBodyMaxBytes. // nothing but whitespace after it, in at most banBodyMaxBytes.
func (rq *request) readBanToAdd() (banToAdd, error) { func readBanToAdd(r *http.Request) (banToAdd, error) {
data, err := io.ReadAll( data, err := io.ReadAll(
http.MaxBytesReader(nil, rq.in.Body, banBodyMaxBytes), http.MaxBytesReader(nil, r.Body, banBodyMaxBytes),
) )
rq.requestBytes = int64(len(data))
if err != nil { if err != nil {
return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err) return banToAdd{}, fmt.Errorf("%w: %w", errNotBanToAdd, err)
} }
@@ -250,21 +191,22 @@ func expiry(duration string, now time.Time) (time.Time, error) {
return now.Add(length), nil return now.Add(length), nil
} }
// liftBans answers DELETE BansPath/<client>: it lifts every ban active on // handleLiftBans answers DELETE BansPath/<client>: it lifts every ban
// a netblock the client's address is in, and answers with those bans, or // active on a netblock the client's address is in, and answers with those
// with 404 when none is active. // bans, or with 404 when none is active.
func (rq *request) liftBans() { func (h *handler) handleLiftBans() http.HandlerFunc {
client, err := pathAddress(rq.in.URL.Path, BansPath+"/") return func(w http.ResponseWriter, r *http.Request) {
client, err := pathAddress(r.URL.Path, BansPath+"/")
if err != nil { if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
} }
lifted := rq.h.ledger.Lift(client, rq.h.now()) lifted := h.ledger.Lift(client, h.now())
if len(lifted) == 0 { if len(lifted) == 0 {
http.Error( http.Error(
rq.out, w,
"no ban is active on "+client.String(), "no ban is active on "+client.String(),
http.StatusNotFound, http.StatusNotFound,
) )
@@ -272,7 +214,8 @@ func (rq *request) liftBans() {
return return
} }
rq.answerBans(lifted) h.answerBans(w, lifted)
}
} }
// clientAnswer is the answer to GET ClientsPath<ip>: the client the // clientAnswer is the answer to GET ClientsPath<ip>: the client the
@@ -284,29 +227,31 @@ type clientAnswer struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
} }
// showClient answers GET ClientsPath<ip> with what waf knows of the client: // handleShowClient answers GET ClientsPath<ip> with what waf knows of the
// its counters, its history, which holds its country as last looked up and its // client: its counters, its history, which holds its country as last looked
// offences, and its bans with their notes. // up and its offences, and its bans with their notes.
func (rq *request) showClient() { func (h *handler) handleShowClient() http.HandlerFunc {
addr, err := pathAddress(rq.in.URL.Path, ClientsPath) return func(w http.ResponseWriter, r *http.Request) {
addr, err := pathAddress(r.URL.Path, ClientsPath)
if err != nil { if err != nil {
http.Error(rq.out, err.Error(), http.StatusBadRequest) http.Error(w, err.Error(), http.StatusBadRequest)
return return
} }
answer := clientAnswer{ answer := clientAnswer{
Bans: state.BanEntries(rq.h.ledger.Covering(addr)), Bans: state.BanEntries(h.ledger.Covering(addr)),
} }
client, seen := rq.h.limiter.Client( client, seen := h.limiter.Client(
clientGroup(rq.h.config.IPv6GroupPrefix, addr), clientGroup(h.config.IPv6GroupPrefix, addr),
) )
if seen { if seen {
answer.Client = &client answer.Client = &client
} }
rq.answerJSON(answer) h.answerJSON(w, answer)
}
} }
// pathAddress reads the client's address that follows prefix in path. // pathAddress reads the client's address that follows prefix in path.
@@ -322,27 +267,27 @@ func pathAddress(path, prefix string) (netip.Addr, error) {
} }
// answerBans answers with held under bans, as bans.json lists them. // answerBans answers with held under bans, as bans.json lists them.
func (rq *request) answerBans(held []bans.Ban) { func (h *handler) answerBans(w http.ResponseWriter, held []bans.Ban) {
rq.answerJSON(struct { h.answerJSON(w, struct {
Bans []state.BanEntry `json:"bans"` Bans []state.BanEntry `json:"bans"`
}{state.BanEntries(held)}) }{state.BanEntries(held)})
} }
// answerJSON answers with value as indented JSON. // answerJSON answers with value as indented JSON.
func (rq *request) answerJSON(value any) { func (h *handler) answerJSON(w http.ResponseWriter, value any) {
body, err := json.MarshalIndent(value, "", " ") body, err := json.MarshalIndent(value, "", " ")
if err != nil { if err != nil {
rq.h.processLog.Error( h.processLog.Error(
"encoding an answer failed", "encoding an answer failed",
"error", "error",
err.Error(), err.Error(),
) )
http.Error(rq.out, http.StatusText(http.StatusInternalServerError), http.Error(w, http.StatusText(http.StatusInternalServerError),
http.StatusInternalServerError) http.StatusInternalServerError)
return return
} }
rq.out.Header().Set("Content-Type", "application/json") w.Header().Set("Content-Type", "application/json")
_, _ = rq.out.Write(append(body, '\n')) _, _ = w.Write(append(body, '\n'))
} }
+34
View File
@@ -14,6 +14,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/state" "sneak.berlin/go/smallwebwaf/internal/state"
"sneak.berlin/go/smallwebwaf/internal/waf" "sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
const ( const (
@@ -333,6 +334,39 @@ func TestBanToAddThatCannotBeReadIsRefused(t *testing.T) {
} }
} }
func TestBanToAddIsLoggedWithTheBytesOfItsBody(t *testing.T) {
t.Parallel()
s, _, _ := startWithClock(
t,
"",
map[string]string{adminToken: adminSecret},
)
s.admin(http.MethodPost, waf.BansPath, banOtherClient, http.StatusOK)
line := s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != int64(len(banOtherClient)) {
t.Errorf(
"the ban to add was logged with request_bytes %d, want %d",
line.RequestBytes,
len(banOtherClient),
)
}
// The health endpoint reads none of a body sent to it.
s.ownEndpointRequest("", http.MethodGet, wafprotocol.HealthPath, "ok",
http.StatusOK)
line = s.out.endedLines(t, s.sent)[s.sent-1]
if line.RequestBytes != 0 {
t.Errorf(
"the health request was logged with request_bytes %d, want 0",
line.RequestBytes,
)
}
}
func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) { func TestClientEndpointShowsTheClientAndItsBans(t *testing.T) {
t.Parallel() t.Parallel()
+7 -6
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts" "sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -250,7 +249,7 @@ func TestObserveModeWorksOutABanOnlyWhenItsAlertWouldBeSent(t *testing.T) {
// never sent them, and returns the queue they wait in as well. // never sent them, and returns the queue they wait in as well.
func startWithAlerts( func startWithAlerts(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
return startAppWithAlerts( return startAppWithAlerts(
@@ -263,7 +262,7 @@ func startWithAlerts(
// startAppWithAlerts is startWithAlerts in front of the app handler. // startAppWithAlerts is startWithAlerts in front of the app handler.
func startAppWithAlerts( func startAppWithAlerts(
t *testing.T, handler http.HandlerFunc, env map[string]string, t *testing.T, handler http.HandlerFunc, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
app := startApp(t, handler) app := startApp(t, handler)
@@ -292,7 +291,7 @@ func startAppWithAlerts(
// the time raised, for ban, with its netblock, cause, reason and notes, // the time raised, for ban, with its netblock, cause, reason and notes,
// which ends at expires, as the log line gives it. // which ends at expires, as the log line gives it.
func banAlert( func banAlert(
event string, event alerts.Event,
raised time.Time, raised time.Time,
client string, client string,
ban bans.Ban, ban bans.Ban,
@@ -307,7 +306,9 @@ func banAlert(
Netblock: ban.Netblock, Netblock: ban.Netblock,
Reason: ban.Reason, Reason: ban.Reason,
Detail: map[string]any{ Detail: map[string]any{
"cause": ban.Cause, "ban_expires": expires, "notes": ban.Notes, "cause": string(
ban.Cause,
), "ban_expires": expires, "notes": ban.Notes,
}, },
} }
} }
@@ -315,7 +316,7 @@ func banAlert(
// attackAlert is banAlert for a ban for the probe rule of testRules, with // attackAlert is banAlert for a ban for the probe rule of testRules, with
// the netblock and the notes of ban. // the netblock and the notes of ban.
func attackAlert( func attackAlert(
event string, event alerts.Event,
raised time.Time, raised time.Time,
client string, client string,
ban bans.Ban, ban bans.Ban,
+6 -7
View File
@@ -15,7 +15,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/anomaly" "sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// The anomaly thresholds: the prefix of a scope followed by the end of a // The anomaly thresholds: the prefix of a scope followed by the end of a
@@ -86,10 +85,10 @@ func TestEachScopeAndWindowOverItsThresholdAlertsOncePerCooldown(
value int64 value int64
count float64 count float64
}{ }{
{requestsPerMinute, ratelimit.KindRequests, minute, 2, 3}, {requestsPerMinute, string(ratelimit.KindRequests), minute, 2, 3},
{requestsPerHour, ratelimit.KindRequests, hour, 2, 3}, {requestsPerHour, string(ratelimit.KindRequests), hour, 2, 3},
{bytesPerMinute, ratelimit.KindBytes, minute, 250, 300}, {bytesPerMinute, string(ratelimit.KindBytes), minute, 250, 300},
{bytesPerHour, ratelimit.KindBytes, hour, 250, 300}, {bytesPerHour, string(ratelimit.KindBytes), hour, 250, 300},
} { } {
setting := scope.prefix + threshold.end setting := scope.prefix + threshold.end
value := strconv.FormatInt(threshold.value, 10) value := strconv.FormatInt(threshold.value, 10)
@@ -186,7 +185,7 @@ func TestEveryRequestIsCountedWhateverIsDoneWithIt(t *testing.T) {
for _, sent := range []struct { for _, sent := range []struct {
from, path string from, path string
status int status int
action string action requestlog.Action
}{ }{
{allowed, "/", http.StatusOK, requestlog.ActionForward}, {allowed, "/", http.StatusOK, requestlog.ActionForward},
{exempt, "/", http.StatusOK, requestlog.ActionForward}, {exempt, "/", http.StatusOK, requestlog.ActionForward},
@@ -332,7 +331,7 @@ func TestRequestCountsForTheASNumberGeoJSGivesBeforeItEnds(t *testing.T) {
// kept. // kept.
geojsURL, _, release := startHeldGeoJS(t) geojsURL, _, release := startHeldGeoJS(t)
var server atomic.Pointer[waf.Server] var server atomic.Pointer[wafAndParts]
app := startApp(t, func(http.ResponseWriter, *http.Request) { app := startApp(t, func(http.ResponseWriter, *http.Request) {
release() release()
+17 -17
View File
@@ -135,7 +135,7 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
// cookies is how many cookies the request carries. // cookies is how many cookies the request carries.
cookies int cookies int
status int status int
action string action requestlog.Action
}{ }{
{ {
"at the limit", lineLimit, 0, http.StatusOK, "at the limit", lineLimit, 0, http.StatusOK,
@@ -439,7 +439,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
env map[string]string env map[string]string
path string path string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
}{ }{
{"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs}, {"SWW_WAF_DENY_NETS", map[string]string{denyNets: client}, "/", leaveAsItIs},
{"a ban", nil, "/", banClient}, {"a ban", nil, "/", banClient},
@@ -454,21 +454,21 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
"a DNS blocklist", map[string]string{ "a DNS blocklist", map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny, dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionDeny,
}, },
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}}) loadVerdicts(server, map[string][]string{client: {dnsblZone}})
}, },
}, },
{ {
"AbuseIPDB", "AbuseIPDB",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny}, map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
"/", func(_ *testing.T, server *waf.Server, _ time.Time) { "/", func(_ *testing.T, server *wafAndParts, _ time.Time) {
// Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default. // Over SWW_WAF_ABUSEIPDB_MIN_SCORE, 75 by default.
loadScores(server, map[string]int64{client: 80}) loadScores(server, map[string]int64{client: 80})
}, },
}, },
{ {
"SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"}, "SWW_WAF_TAG_DENY", map[string]string{tagLists: "vpn=" + vpnList, tagDeny: "vpn"},
"/", func(t *testing.T, server *waf.Server, _ time.Time) { "/", func(t *testing.T, server *wafAndParts, _ time.Time) {
t.Helper() t.Helper()
loadLists(t, server, map[string][]string{vpnList: {client}}) loadLists(t, server, map[string][]string{vpnList: {client}})
}, },
@@ -685,7 +685,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
// for env. No request to it may use the network. // for env. No request to it may use the network.
func newWAFInBubble( func newWAFInBubble(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*waf.Server, *output) { ) (*wafAndParts, *output) {
t.Helper() t.Helper()
settings := map[string]string{tagDeny: "country=KP"} settings := map[string]string{tagDeny: "country=KP"}
@@ -699,7 +699,7 @@ func newWAFInBubble(
// askDirectly hands a question about a request from client straight to server, // askDirectly hands a question about a request from client straight to server,
// as change leaves it, and returns the answer's status and how long it took. // as change leaves it, and returns the answer's status and how long it took.
func askDirectly( func askDirectly(
t *testing.T, server *waf.Server, change func(*http.Request), t *testing.T, server *wafAndParts, change func(*http.Request),
) (int, time.Duration) { ) (int, time.Duration) {
t.Helper() t.Helper()
@@ -717,13 +717,13 @@ func askDirectly(
recorder := httptest.NewRecorder() recorder := httptest.NewRecorder()
began := time.Now() began := time.Now()
server.Handler.ServeHTTP(recorder, req) server.ServeHTTP(recorder, req)
return recorder.Code, time.Since(began) return recorder.Code, time.Since(began)
} }
// banClient bans client at now. // banClient bans client at now.
func banClient(_ *testing.T, server *waf.Server, now time.Time) { func banClient(_ *testing.T, server *wafAndParts, now time.Time) {
server.Ledger.Ban( server.Ledger.Ban(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
@@ -732,7 +732,7 @@ func banClient(_ *testing.T, server *waf.Server, now time.Time) {
} }
// countClient counts one request from client at now for the rate limits. // countClient counts one request from client at now for the rate limits.
func countClient(_ *testing.T, server *waf.Server, now time.Time) { func countClient(_ *testing.T, server *wafAndParts, now time.Time) {
server.Limiter.Count( server.Limiter.Count(
netip.MustParsePrefix(client+"/32"), netip.MustParsePrefix(client+"/32"),
now, now,
@@ -741,7 +741,7 @@ func countClient(_ *testing.T, server *waf.Server, now time.Time) {
} }
// listClient puts client on the blocklist at dropURL. // listClient puts client on the blocklist at dropURL.
func listClient(t *testing.T, server *waf.Server, now time.Time) { func listClient(t *testing.T, server *wafAndParts, now time.Time) {
t.Helper() t.Helper()
err := server.Lists.Load([]reputation.List{{ err := server.Lists.Load([]reputation.List{{
@@ -753,7 +753,7 @@ func listClient(t *testing.T, server *waf.Server, now time.Time) {
} }
// leaveAsItIs prepares nothing. // leaveAsItIs prepares nothing.
func leaveAsItIs(*testing.T, *waf.Server, time.Time) {} func leaveAsItIs(*testing.T, *wafAndParts, time.Time) {}
func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) { func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
t.Parallel() t.Parallel()
@@ -764,11 +764,11 @@ func TestARefusedRequestDoesNotWaitForASlowGeoJS(t *testing.T) {
name string name string
env map[string]string env map[string]string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
// cookie is the request's Cookie header. // cookie is the request's Cookie header.
cookie string cookie string
status int status int
action string action requestlog.Action
}{ }{
{ {
name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client}, name: "SWW_WAF_DENY_NETS", env: map[string]string{denyNets: client},
@@ -849,9 +849,9 @@ func TestAQuestionWaitsForASlowGeoJSOnlyAsLongAsTheDefaultSays(
name string name string
env map[string]string env map[string]string
// prepare gets the server ready, at now, before the question. // prepare gets the server ready, at now, before the question.
prepare func(t *testing.T, server *waf.Server, now time.Time) prepare func(t *testing.T, server *wafAndParts, now time.Time)
status int status int
action string action requestlog.Action
}{ }{
{"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward}, {"nothing refuses it", nil, leaveAsItIs, http.StatusOK, requestlog.ActionForward},
{ {
@@ -915,7 +915,7 @@ func TestWAFAnswersWithinTheTimeLimitWhileEveryOutsideSourceHangs(
held, _, _ := startHeldGeoJS(t) held, _, _ := startHeldGeoJS(t)
resolver, releaseResolver := startHeldResolver(t) resolver, releaseResolver := startHeldResolver(t)
server, out, _ := newWAFWith(t, waf.New, server, out, _ := newWAFWith(t, func(*waf.Params) {},
silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{ silentGeoJSURL, silentAbuseIPDBURL, time.Now, map[string]string{
tagDeny: "country=KP", coreRuleSetMode: block, tagDeny: "country=KP", coreRuleSetMode: block,
dnsblZones: dnsblZone, dnsblResolver: resolver, dnsblZones: dnsblZone, dnsblResolver: resolver,
+22 -15
View File
@@ -18,7 +18,7 @@ import (
// banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged // banResponse is a refusal answered with SWW_WAF_BAN_RESPONSE, and logged
// with action. // with action.
func (rq *request) banResponse(action string) *refusal { func (rq *request) banResponse(action requestlog.Action) *refusal {
return &refusal{status: rq.h.config.BanResponse, action: action} return &refusal{status: rq.h.config.BanResponse, action: action}
} }
@@ -45,8 +45,11 @@ func (a activeBan) check(
} }
ban, banned, madePermanent := check(rq.client, now) ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned { if banned {
rq.line.BanExpires = banExpires(ban) expires = banExpires(ban)
rq.line.BanExpires = expires
} }
if madePermanent { if madePermanent {
@@ -61,14 +64,14 @@ func (a activeBan) check(
return &verdict{ return &verdict{
action: banAction(ban.Permanent()), action: banAction(ban.Permanent()),
figures: map[string]any{ figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": rq.line.BanExpires, figureNetblock: ban.Netblock.String(), "ban_expires": expires,
}, },
} }
} }
// banAction is the action of a request refused under a ban or for making one: // banAction is the action of a request refused under a ban or for making one:
// permanent says the ban never runs out. // permanent says the ban never runs out.
func banAction(permanent bool) string { func banAction(permanent bool) requestlog.Action {
if permanent { if permanent {
return requestlog.ActionPermanentlyBanned return requestlog.ActionPermanentlyBanned
} }
@@ -144,10 +147,11 @@ func (b banMaker) banForLimit(
rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals rq.line.LimitHit = hit.Window + "_bytes" // as counts names the byte totals
case ratelimit.KindRefusals: case ratelimit.KindRefusals:
rq.line.LimitHit = requestlog.LimitHitMinuteRefusals rq.line.LimitHit = requestlog.LimitHitMinuteRefusals
default: case ratelimit.KindRequests:
rq.line.LimitHit = hit.Window rq.line.LimitHit = hit.Window
} }
rq.brokeLimit = true
rq.line.Offence = requestlog.OffenceLimit rq.line.Offence = requestlog.OffenceLimit
b.metrics.LimitHit(hit) b.metrics.LimitHit(hit)
@@ -164,6 +168,7 @@ func (b banMaker) banForLimit(
notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.rateLimitPercent.logged()
case ratelimit.KindBytes: case ratelimit.KindBytes:
notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged() notes.LimitPercent, notes.LimitPercentSetting = rq.bytesLimitPercent.logged()
case ratelimit.KindRefusals: // the refusal limit is not lowered
} }
b.banClient( b.banClient(
@@ -180,7 +185,7 @@ func (b banMaker) banForLimit(
// says what happens instead. // says what happens instead.
func (b banMaker) banForAttack( func (b banMaker) banForAttack(
rq *request, now time.Time, notes bans.Notes, rq *request, now time.Time, notes bans.Notes,
) string { ) requestlog.Action {
return b.banClient( return b.banClient(
rq, rq,
now, now,
@@ -201,7 +206,7 @@ func (b banMaker) banClient(
now time.Time, now time.Time,
status int, status int,
grounds bans.Grounds, grounds bans.Grounds,
) string { ) requestlog.Action {
netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client) netblock := banNetblock(b.banScopeV4Prefix, b.ipv6GroupPrefix, rq.client)
if b.observe { if b.observe {
@@ -259,9 +264,9 @@ func (b banMaker) withClientNotes(
status int, status int,
netblock netip.Prefix, netblock netip.Prefix,
) bans.Notes { ) bans.Notes {
notes.ASN = rq.line.ASN notes.ASN = rq.lookupAnswer.ASN
notes.ASName = rq.line.ASName notes.ASName = rq.lookupAnswer.ASName
notes.Country = rq.line.Country notes.Country = rq.lookupAnswer.Country
notes.Reputation = rq.reputation notes.Reputation = rq.reputation
notes.Request = rq.noted(now, status) notes.Request = rq.noted(now, status)
notes.Requests = b.netblockRequests(netblock) notes.Requests = b.netblockRequests(netblock)
@@ -275,7 +280,7 @@ func (b banMaker) withClientNotes(
// SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the // SWW_WAF_ALERT_COOLDOWN and never with no webhook set: its notes count the
// netblock's requests, which can mean going through every client. // netblock's requests, which can mean going through every client.
func (b banMaker) wouldAlertBan( func (b banMaker) wouldAlertBan(
netblock netip.Prefix, now time.Time, cause string, netblock netip.Prefix, now time.Time, cause bans.Cause,
) bool { ) bool {
event := alerts.EventBan event := alerts.EventBan
if b.ledger.WouldBePermanent(netblock, now, cause) { if b.ledger.WouldBePermanent(netblock, now, cause) {
@@ -296,7 +301,9 @@ func (b banMaker) alertBan(rq *request, ban bans.Ban) {
} }
detail := map[string]any{ detail := map[string]any{
"cause": ban.Cause, "ban_expires": banExpires(ban), "notes": ban.Notes, "cause": string(ban.Cause),
"ban_expires": banExpires(ban),
"notes": ban.Notes,
} }
if b.observe { if b.observe {
detail["mode"] = "observe" detail["mode"] = "observe"
@@ -328,11 +335,11 @@ func (b banMaker) netblockRequests(netblock netip.Prefix) int64 {
func (rq *request) noted(now time.Time, status int) bans.Request { func (rq *request) noted(now time.Time, status int) bans.Request {
return bans.Request{ return bans.Request{
Time: now, Time: now,
Method: rq.line.Method, Method: rq.method,
Host: rq.line.Host, Host: rq.site,
Path: rq.requestURI, Path: rq.requestURI,
Status: status, Status: status,
UserAgent: rq.line.UserAgent, UserAgent: rq.userAgent,
} }
} }
+9 -6
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
const ( const (
@@ -466,7 +465,7 @@ func (c *clock) advance(d time.Duration) {
// window. // window.
func startWithClock( func startWithClock(
t *testing.T, geojsURL string, env map[string]string, t *testing.T, geojsURL string, env map[string]string,
) (*sender, *clock, *waf.Server) { ) (*sender, *clock, *wafAndParts) {
t.Helper() t.Helper()
app := startApp(t, func(http.ResponseWriter, *http.Request) {}) app := startApp(t, func(http.ResponseWriter, *http.Request) {})
@@ -504,7 +503,9 @@ type sender struct {
} }
// get sends a GET request for / from the client at from. // get sends a GET request for / from the client at from.
func (s *sender) get(from string, status int, action string) logLine { func (s *sender) get(
from string, status int, action requestlog.Action,
) logLine {
s.t.Helper() s.t.Helper()
return s.request(from, "/", status, action) return s.request(from, "/", status, action)
@@ -517,7 +518,7 @@ func (s *sender) get(from string, status int, action string) logLine {
func (s *sender) request( func (s *sender) request(
from, path string, from, path string,
status int, status int,
action string, action requestlog.Action,
) logLine { ) logLine {
s.t.Helper() s.t.Helper()
@@ -527,7 +528,7 @@ func (s *sender) request(
// requestWithHeader is request with header, such as "Authorization: // requestWithHeader is request with header, such as "Authorization:
// Bearer x", added to the request unless it is "". // Bearer x", added to the request unless it is "".
func (s *sender) requestWithHeader( func (s *sender) requestWithHeader(
from, path, header string, status int, action string, from, path, header string, status int, action requestlog.Action,
) logLine { ) logLine {
s.t.Helper() s.t.Helper()
@@ -553,7 +554,9 @@ func (s *sender) requestWithHeader(
// the Core Rule Set past any shorter wait. go test's -timeout ends a test // the Core Rule Set past any shorter wait. go test's -timeout ends a test
// whose answer never comes. // whose answer never comes.
func (s *sender) requestWithBody( func (s *sender) requestWithBody(
method, from, path, header, body string, status int, action string, method, from, path, header, body string,
status int,
action requestlog.Action,
) (logLine, answer) { ) (logLine, answer) {
s.t.Helper() s.t.Helper()
+4 -6
View File
@@ -62,7 +62,7 @@ func coreRuleSetMatch() coreruleset.Result {
// at its default but for env. No request to it may use the network. // at its default but for env. No request to it may use the network.
func newWAFHoldingTheCoreRuleSet( func newWAFHoldingTheCoreRuleSet(
t *testing.T, crs *heldCoreRuleSet, env map[string]string, t *testing.T, crs *heldCoreRuleSet, env map[string]string,
) (*waf.Server, *output, *alerts.Queue) { ) (*wafAndParts, *output, *alerts.Queue) {
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
@@ -71,11 +71,9 @@ func newWAFHoldingTheCoreRuleSet(
} }
maps.Copy(settings, env) maps.Copy(settings, env)
server, out, queue := newWAFWith(t, func(params waf.Params) *waf.Server { server, out, queue := newWAFWith(t, func(params *waf.Params) {
params.CoreRuleSet = crs params.CoreRuleSet = crs
params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit params.BesideLookupTimeLimit = waf.BesideLookupTimeLimit
return waf.New(params)
}, silentGeoJSURL, abuseIPDBURL, time.Now, settings) }, silentGeoJSURL, abuseIPDBURL, time.Now, settings)
return server, out, queue return server, out, queue
@@ -105,7 +103,7 @@ func wantAnswer(
// left no trace: line has none of its fields, and server has counted no match // left no trace: line has none of its fields, and server has counted no match
// of it and raised no alert for one. // of it and raised no alert for one.
func wantNoCoreRuleSetTrace( func wantNoCoreRuleSetTrace(
t *testing.T, server *waf.Server, queue *alerts.Queue, line logLine, t *testing.T, server *wafAndParts, queue *alerts.Queue, line logLine,
) { ) {
t.Helper() t.Helper()
@@ -198,7 +196,7 @@ type firstToRefuse struct {
env map[string]string env map[string]string
inspection time.Duration inspection time.Duration
took time.Duration took time.Duration
action string action requestlog.Action
} }
// firstToRefuseCases are the cases of // firstToRefuseCases are the cases of
+2 -2
View File
@@ -68,7 +68,7 @@ type biasedThresholds struct {
func (b biasedThresholds) limitPercentages( func (b biasedThresholds) limitPercentages(
rq *request, rq *request,
) (percentage, percentage) { ) (percentage, percentage) {
asn, country := rq.line.ASN, rq.line.Country asn, country := rq.lookupAnswer.ASN, rq.lookupAnswer.Country
unknown := percentage{percent: whole} unknown := percentage{percent: whole}
if country == "" && !rq.skippedLookupWait { if country == "" && !rq.skippedLookupWait {
@@ -115,7 +115,7 @@ func (b biasedThresholds) limitPercentages(
otherTags := percentage{percent: whole} otherTags := percentage{percent: whole}
for _, tag := range rq.line.Tags { for _, tag := range rq.tags {
if tag != asnTag(asn) && tag != countryTag(country) { if tag != asnTag(asn) && tag != countryTag(country) {
otherTags = lowest( otherTags = lowest(
otherTags, otherTags,
+6 -7
View File
@@ -18,7 +18,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest" "sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
@@ -74,7 +73,7 @@ func TestEachBiasedThresholdLowersTheRateLimits(t *testing.T) {
// Half of 4 requests a minute: the third breaks the limit. // Half of 4 requests a minute: the third breaks the limit.
for _, sent := range []struct { for _, sent := range []struct {
status int status int
action string action requestlog.Action
}{ }{
{http.StatusOK, requestlog.ActionForward}, {http.StatusOK, requestlog.ActionForward},
{http.StatusOK, requestlog.ActionForward}, {http.StatusOK, requestlog.ActionForward},
@@ -453,7 +452,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// limit, which is answered with status and logged with action. // limit, which is answered with status and logged with action.
before int before int
status int status int
action string action requestlog.Action
// reason and want are the ban's reason, and its notes' limit // reason and want are the ban's reason, and its notes' limit
// percentage, as percentText gives it. // percentage, as percentText gives it.
reason, want string reason, want string
@@ -530,7 +529,7 @@ func TestBanForALoweredLimitGivesThePercentageInItsNotesAndItsAlert(
// clock. // clock.
func startWithLookups( func startWithLookups(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *waf.Server, *alerts.Queue) { ) (*sender, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
s, _, server, queue := startWithLookupsAndClock(t, env) s, _, server, queue := startWithLookupsAndClock(t, env)
@@ -545,7 +544,7 @@ func startWithLookups(
// country, and no other address. // country, and no other address.
func startWithLookupsAndClock( func startWithLookupsAndClock(
t *testing.T, env map[string]string, t *testing.T, env map[string]string,
) (*sender, *clock, *waf.Server, *alerts.Queue) { ) (*sender, *clock, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb") path := filepath.Join(t.TempDir(), "ipinfo_lite.mmdb")
@@ -585,7 +584,7 @@ func (s *sender) uploadFrom(from string) logLine {
// straight to server's handler, without the network, and returns once it is // straight to server's handler, without the network, and returns once it is
// answered. // answered.
func serveFromDE( func serveFromDE(
t *testing.T, server *waf.Server, method string, body io.Reader, t *testing.T, server *wafAndParts, method string, body io.Reader,
) { ) {
t.Helper() t.Helper()
@@ -593,7 +592,7 @@ func serveFromDE(
req.Header.Set(wafprotocol.ClientHeader, fromDE) req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req) server.ServeHTTP(httptest.NewRecorder(), req)
} }
// wantPercent checks a limit percentage that a log line or a ban's notes // wantPercent checks a limit percentage that a log line or a ban's notes
+9
View File
@@ -43,6 +43,15 @@ func (b blocklists) check(
} }
} }
// refusesWhateverTheLookupAnswers reports whether blocklists refuses rq, as
// refusedByBlocklists says. The lookup's answer plays no part in it.
func (b blocklists) refusesWhateverTheLookupAnswers(
rq *request,
_ time.Time,
) bool {
return b.refusedByBlocklists(b.lists.ListedBy(rq.client))
}
// refusedByBlocklists reports whether the heuristic blocklists refuses a // refusedByBlocklists reports whether the heuristic blocklists refuses a
// client, given listedBy, the blocklists that list it: it does when one lists // client, given listedBy, the blocklists that list it: it does when one lists
// it and SWW_WAF_BLOCKLIST_ACTION is deny. // it and SWW_WAF_BLOCKLIST_ACTION is deny.
+1 -1
View File
@@ -33,7 +33,7 @@ func (b byteLimits) check(
rq *request, rq *request,
now time.Time, now time.Time,
) *verdict { ) *verdict {
if !rq.counted || rq.line.WouldAction != "" { if !rq.counted || rq.wouldAction != "" {
return nil return nil
} }
+3 -3
View File
@@ -121,9 +121,9 @@ func (c coreRuleSet) alertCoreRuleSetMatch(
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: clientGroup(c.ipv6GroupPrefix, rq.client), Netblock: clientGroup(c.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN, ASN: rq.lookupAnswer.ASN,
ASName: rq.line.ASName, ASName: rq.lookupAnswer.ASName,
Country: rq.line.Country, Country: rq.lookupAnswer.Country,
Reason: "scored by the Core Rule Set at or over " + Reason: "scored by the Core Rule Set at or over " +
"SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD", "SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD",
Detail: detail, Detail: detail,
+5 -3
View File
@@ -84,7 +84,7 @@ func TestCoreRuleSetMatchAddsTheExploitSignatureTag(t *testing.T) {
name string name string
mode string mode string
status int status int
action string action requestlog.Action
}{ }{
{block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet}, {block, block, http.StatusForbidden, requestlog.ActionBlockedByCoreRuleSet},
{detect, detect, http.StatusOK, requestlog.ActionForward}, {detect, detect, http.StatusOK, requestlog.ActionForward},
@@ -158,7 +158,8 @@ func TestCoreRuleSetRefusesAttacksInBlockModeAndOnlyLogsThemInDetectMode(
t.Parallel() t.Parallel()
for _, tc := range []struct { for _, tc := range []struct {
mode, action string mode string
action requestlog.Action
status int status int
}{ }{
{block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden}, {block, requestlog.ActionBlockedByCoreRuleSet, http.StatusForbidden},
@@ -653,7 +654,8 @@ func TestCoreRuleSetMatchRaisesTheCoreRuleSetMatchAlert(t *testing.T) {
// with, and alertMode what the alert's detail gives as mode, if // with, and alertMode what the alert's detail gives as mode, if
// anything. // anything.
status int status int
action, alertMode string action requestlog.Action
alertMode string
}{ }{
{ {
"block", map[string]string{coreRuleSetMode: block}, "block", map[string]string{coreRuleSetMode: block},
+1 -1
View File
@@ -27,7 +27,7 @@ func (c countries) check(
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
allowed := c.allowed allowed := c.allowed
country := rq.line.Country country := rq.lookupAnswer.Country
if len(allowed) == 0 || slices.Contains(allowed, country) { if len(allowed) == 0 || slices.Contains(allowed, country) {
return nil return nil
+1 -1
View File
@@ -316,7 +316,7 @@ func TestExclusiveListRefusesAPrivateAddressUnlessAllowed(t *testing.T) {
name string name string
allowNets string allowNets string
status int status int
action string action requestlog.Action
}{ }{
{ {
"not in SWW_WAF_ALLOW_NETS", "", "not in SWW_WAF_ALLOW_NETS", "",
+4 -6
View File
@@ -1,13 +1,11 @@
package waf package waf
import "sneak.berlin/go/smallwebwaf/internal/reputation" import "net/http"
// BesideLookupTimeLimit is besideLookupTimeLimit. // BesideLookupTimeLimit is besideLookupTimeLimit.
const BesideLookupTimeLimit = besideLookupTimeLimit const BesideLookupTimeLimit = besideLookupTimeLimit
// BanAbuseIPDBBlacklist does on s what a fetch of AbuseIPDB's blacklist with // HTTPServer is Go's server s holds, for tests that read its settings.
// the clients on listed does. func (s *Server) HTTPServer() *http.Server {
func (s *Server) BanAbuseIPDBBlacklist(listed []reputation.BlacklistEntry) { return s.httpServer
h := s.Handler.(*handler) //nolint:forcetypeassert // New makes it one
BanAbuseIPDBBlacklist(h.config, h.ledger, h.processLog, h.now(), listed)
} }
+11
View File
@@ -58,3 +58,14 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
} }
} }
} }
func TestHealthEndpointIsFoundAtItsPathPercentEncoded(t *testing.T) {
t.Parallel()
// %68 is h. An endpoint is found by the decoded path.
s, _, _ := startWithClock(t, "", nil)
got := s.ownEndpointRequest("", http.MethodGet, "/_sww/%68ealthz", "",
http.StatusOK)
wantBody(t, got, "ok\n")
}
+24 -3
View File
@@ -3,12 +3,24 @@ package waf
import ( import (
"context" "context"
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// To add a heuristic, write one file with a struct that implements heuristic // To add a heuristic, write one file with a struct that implements heuristic
// and holds the settings and parts it reads. Add one line for it to one of the // and holds the settings and parts it reads. Add one line for it to one of the
// lists that newHandler builds, which hands it those settings and parts. Its // lists that newHandler builds, which hands it those settings and parts. Its
// settings go in config.WAF, and the comment at the top of its file names them. // settings go in config.WAF, and the comment at the top of its file names them.
// A heuristic of afterLookup that can refuse a request whatever the lookup
// answers also implements refusesWithoutLookup.
//
// Three places outside its file may change with it:
// - waitForGeoJS in newHandler lists the settings that need the lookup's
// answer;
// - requestCanGetTag in internal/config/waf.go holds which setting can give a
// request which tag;
// - findBesideLookup does the work of each heuristic of besideLookup, by
// name.
// heuristic is one of the checks that decide what waf does with a // heuristic is one of the checks that decide what waf does with a
// request. // request.
@@ -20,11 +32,20 @@ type heuristic interface {
check(ctx context.Context, rq *request, now time.Time) *verdict check(ctx context.Context, rq *request, now time.Time) *verdict
} }
// refusesWithoutLookup is a heuristic of afterLookup that can tell, before the
// lookup answers, that it will refuse a request whatever the answer.
type refusesWithoutLookup interface {
// refusesWhateverTheLookupAnswers reports whether the heuristic will refuse
// rq at now, whatever the lookup answers about its client. It changes
// nothing.
refusesWhateverTheLookupAnswers(rq *request, now time.Time) bool
}
// verdict is a heuristic refusing a request, or banning its client: the action // verdict is a heuristic refusing a request, or banning its client: the action
// the log line names, one of requestlog's Action constants, and the figures the // the log line names, and the figures the heuristic acted on, by name, such as
// heuristic acted on, by name, such as the count and the limit. // the count and the limit.
type verdict struct { type verdict struct {
action string action requestlog.Action
figures map[string]any figures map[string]any
} }
+1 -1
View File
@@ -151,7 +151,7 @@ func TestPathUnderTheOwnPrefixIsNeitherForwardedNorRefused(t *testing.T) {
// historyOf returns the history of the client at addr. // historyOf returns the history of the client at addr.
func historyOf( func historyOf(
t *testing.T, server *waf.Server, addr string, t *testing.T, server *wafAndParts, addr string,
) ratelimit.History { ) ratelimit.History {
t.Helper() t.Helper()
+2 -2
View File
@@ -94,11 +94,11 @@ func (rq *request) noteLookup(result lookupResult) {
rq.line.ASName = rq.lookupAnswer.ASName rq.line.ASName = rq.lookupAnswer.ASName
rq.line.Country = rq.lookupAnswer.Country rq.line.Country = rq.lookupAnswer.Country
if tag := asnTag(rq.line.ASN); tag != "" { if tag := asnTag(rq.lookupAnswer.ASN); tag != "" {
rq.addTag(tag) rq.addTag(tag)
} }
if tag := countryTag(rq.line.Country); tag != "" { if tag := countryTag(rq.lookupAnswer.Country); tag != "" {
rq.addTag(tag) rq.addTag(tag)
} }
} }
+8 -5
View File
@@ -22,12 +22,12 @@ func TestBanActionSaysWhetherTheBanRunsOut(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
cause string cause bans.Cause
// endsIn is how long after the request the ban ends: negative for a // endsIn is how long after the request the ban ends: negative for a
// ban that has ended, and zero for a permanent ban. // ban that has ended, and zero for a permanent ban.
endsIn time.Duration endsIn time.Duration
path string path string
want string want requestlog.Action
}{ }{
{ {
"temporary ban", bans.CauseAdmin, time.Hour, "/", "temporary ban", bans.CauseAdmin, time.Hour, "/",
@@ -170,7 +170,10 @@ func TestObserveModeForwardsWhatEnforceModeRefuses(t *testing.T) {
// and its second breaks it. // and its second breaks it.
s.get(fromDE, http.StatusOK, requestlog.ActionForward) s.get(fromDE, http.StatusOK, requestlog.ActionForward)
for _, sent := range []struct{ from, refusal string }{ for _, sent := range []struct {
from string
refusal requestlog.Action
}{
{denied, requestlog.ActionClientDenied}, {denied, requestlog.ActionClientDenied},
{banned, requestlog.ActionTemporarilyBanned}, {banned, requestlog.ActionTemporarilyBanned},
{fromKP, requestlog.ActionClientDenied}, {fromKP, requestlog.ActionClientDenied},
@@ -253,7 +256,7 @@ func TestObserveModeMakesNoBanAndKeepsTheBansItHas(t *testing.T) {
// wantWouldAction checks the request log line's would_action, and that a // wantWouldAction checks the request log line's would_action, and that a
// line that should have none has no such field. // line that should have none has no such field.
func wantWouldAction(t *testing.T, line logLine, want string) { func wantWouldAction(t *testing.T, line logLine, want requestlog.Action) {
t.Helper() t.Helper()
got, present := line.fields["would_action"] got, present := line.fields["would_action"]
@@ -261,7 +264,7 @@ func wantWouldAction(t *testing.T, line logLine, want string) {
switch { switch {
case want == "" && present: case want == "" && present:
t.Errorf("log line has would_action %v, want none", got) t.Errorf("log line has would_action %v, want none", got)
case want != "" && got != want: case want != "" && got != string(want):
t.Errorf("log line has would_action %v, want %s", got, want) t.Errorf("log line has would_action %v, want %s", got, want)
} }
} }
+33 -45
View File
@@ -107,21 +107,10 @@ type CoreRuleSetInspector interface {
) (coreruleset.Result, []byte, error) ) (coreruleset.Result, []byte, error)
} }
// Server is the server waf runs, with the parts of its Params. The tests // Server is the server waf runs.
// read them.
type Server struct { type Server struct {
*http.Server httpServer *http.Server
h *handler
Ledger *bans.Ledger
Limiter *ratelimit.Limiter
GeoJS *geojs.GeoJS
Anomalies *anomaly.Counters
LookupFile *lookup.File
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
RouteWatch *routewatch.Client
Metrics *metrics.Metrics
} }
// newHandler returns the handler with what every request shares, the parts // newHandler returns the handler with what every request shares, the parts
@@ -177,30 +166,6 @@ func newHandler(params Params) *handler {
ipv6GroupPrefix: cfg.IPv6GroupPrefix, ipv6GroupPrefix: cfg.IPv6GroupPrefix,
} }
h.blocklists = blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
}
h.rateLimits = rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker} h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{ h.ruleFiles = ruleFiles{
rules: params.Rules, rules: params.Rules,
@@ -230,7 +195,11 @@ func newHandler(params Params) *handler {
} }
h.afterLookup = []heuristic{ h.afterLookup = []heuristic{
countries{allowed: cfg.ExclusivelyAllowedCountries}, countries{allowed: cfg.ExclusivelyAllowedCountries},
h.blocklists, blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
hitNoter: hitNoter,
},
dnsbl{ dnsbl{
dnsbl: params.DNSBL, dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction, action: cfg.DNSBLAbuseIPDBAction,
@@ -246,7 +215,25 @@ func newHandler(params Params) *handler {
hitNoter: hitNoter, hitNoter: hitNoter,
}, },
tagDeny{tags: cfg.TagDeny}, tagDeny{tags: cfg.TagDeny},
h.rateLimits, rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
} }
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet} h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}} h.afterBesideLookup = []heuristic{tagDeny{tags: cfg.TagDeny}}
@@ -268,6 +255,8 @@ func newHandler(params Params) *handler {
}, },
} }
h.ownEndpoints = h.routes()
return h return h
} }
@@ -293,6 +282,9 @@ type handler struct {
// awaitingReport holds the decision on each answered question by its id // awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled. // until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision] awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client // The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all. // in SWW_WAF_ALLOW_NETS skips them all.
@@ -323,11 +315,7 @@ type handler struct {
// afterAnswer, they only ban. // afterAnswer, they only ban.
onReport []heuristic onReport []heuristic
// The heuristics of the lists that other steps of a request use too: // The heuristics of besideLookup, whose work findBesideLookup does.
// refusedWhateverTheAnswer asks blocklists and rateLimits, and
// findBesideLookup does the work of trapPaths, ruleFiles and coreRuleSet.
blocklists blocklists
rateLimits rateLimits
trapPaths trapPaths trapPaths trapPaths
ruleFiles ruleFiles ruleFiles ruleFiles
coreRuleSet coreRuleSet coreRuleSet coreRuleSet
+71 -35
View File
@@ -316,7 +316,7 @@ func startProxyWithGeoJS(
func startProxyWithClock( func startProxyWithClock(
t *testing.T, appURL, geojsURL string, now func() time.Time, t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string, env map[string]string,
) (string, *output, *waf.Server) { ) (string, *output, *wafAndParts) {
t.Helper() t.Helper()
addr, out, server, _ := startProxyWithAlerts( addr, out, server, _ := startProxyWithAlerts(
@@ -335,7 +335,7 @@ func startProxyWithClock(
func startProxyWithAlerts( func startProxyWithAlerts(
t *testing.T, appURL, geojsURL string, now func() time.Time, t *testing.T, appURL, geojsURL string, now func() time.Time,
env map[string]string, env map[string]string,
) (string, *output, *waf.Server, *alerts.Queue) { ) (string, *output, *wafAndParts, *alerts.Queue) {
t.Helper() t.Helper()
server, out, alertQueue := newWAF(t, geojsURL, now, env) server, out, alertQueue := newWAF(t, geojsURL, now, env)
@@ -344,9 +344,9 @@ func startProxyWithAlerts(
return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue
} }
// serve serves server on a port of its own until the test ends, and sets its // serve serves server on a port of its own until the test ends, and sets
// Addr to where it listens. // server.Addr to where it listens.
func serve(t *testing.T, server *waf.Server) { func serve(t *testing.T, server *wafAndParts) {
t.Helper() t.Helper()
listener, err := (&net.ListenConfig{}).Listen( listener, err := (&net.ListenConfig{}).Listen(
@@ -422,8 +422,32 @@ func startRProxy(
return listener.Addr().String() return listener.Addr().String()
} }
// newWAF makes waf's server, without starting it, and // wafAndParts is waf's server with the Params it was built from, so that a
// returns it, what it writes, and the queue of the alerts it raises, as the // test can read the parts the server uses. Addr is where it listens once serve
// has it serving.
type wafAndParts struct {
*waf.Server
waf.Params
Addr string
}
// banAbuseIPDBBlacklist does on w what a fetch of AbuseIPDB's blacklist with
// the clients on listed does.
func (w *wafAndParts) banAbuseIPDBBlacklist(
listed []reputation.BlacklistEntry,
) {
waf.BanAbuseIPDBBlacklist(
w.Config,
w.Ledger,
w.ProcessLog,
w.Now(),
listed,
)
}
// newWAF makes waf's server, without starting it, and returns it with its
// parts, what it writes, and the queue of the alerts it raises, as the
// settings in env make it. No alert is sent from the queue: they wait in it, // settings in env make it. No alert is sent from the queue: they wait in it,
// for the test to look at. // for the test to look at.
// With no geojsURL, there is no stand-in for GeoJS to look clients up at, and // With no geojsURL, there is no stand-in for GeoJS to look clients up at, and
@@ -438,19 +462,21 @@ func newWAF(
geojsURL string, geojsURL string,
now func() time.Time, now func() time.Time,
env map[string]string, env map[string]string,
) (*waf.Server, *output, *alerts.Queue) { ) (*wafAndParts, *output, *alerts.Queue) {
t.Helper() t.Helper()
return newWAFWith(t, waf.New, geojsURL, abuseIPDBURL, now, env) return newWAFWith(
t, func(*waf.Params) {}, geojsURL, abuseIPDBURL, now, env,
)
} }
// newWAFWith is newWAF for the server build makes, with clients checked with // newWAFWith is newWAF with change applied to the Params before the server is
// AbuseIPDB at abuseIPDBCheckURL. // built from them, and clients checked with AbuseIPDB at abuseIPDBCheckURL.
func newWAFWith( func newWAFWith(
t *testing.T, build func(waf.Params) *waf.Server, t *testing.T, change func(*waf.Params),
geojsURL, abuseIPDBCheckURL string, now func() time.Time, geojsURL, abuseIPDBCheckURL string, now func() time.Time,
env map[string]string, env map[string]string,
) (*waf.Server, *output, *alerts.Queue) { ) (*wafAndParts, *output, *alerts.Queue) {
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
@@ -518,8 +544,9 @@ func newWAFWith(
addCoreRuleSet(t, &params) addCoreRuleSet(t, &params)
addCounters(&params) addCounters(&params)
addOutsideSources(&params, geojsURL, abuseIPDBCheckURL) addOutsideSources(&params, geojsURL, abuseIPDBCheckURL)
change(&params)
server := build(params) server := &wafAndParts{Server: waf.New(params), Params: params}
return server, out, alertQueue return server, out, alertQueue
} }
@@ -548,16 +575,11 @@ func addCoreRuleSet(t *testing.T, params *waf.Params) {
params.CoreRuleSet = coreRuleSet params.CoreRuleSet = coreRuleSet
} }
// addCounters adds to params the metrics, the table of clients, the ledger // addCounters adds to params the table of clients, the ledger and the
// and the anomaly counters, as sww builds them from params' settings. The // anomaly counters, as sww builds them from params' settings.
// metrics read the ledger, the table of clients and the rules.
func addCounters(params *waf.Params) { func addCounters(params *waf.Params) {
cfg := params.Config cfg := params.Config
params.Metrics = metrics.New(
cfg.MetricsBusiestCountriesAndASNs,
cfg.InstanceName,
)
params.Limiter = ratelimit.New(ratelimit.Params{ params.Limiter = ratelimit.New(ratelimit.Params{
Limits: ratelimit.Limits{ Limits: ratelimit.Limits{
PerMinute: cfg.RateLimitPerMinute, PerMinute: cfg.RateLimitPerMinute,
@@ -587,26 +609,18 @@ func addCounters(params *waf.Params) {
NamedNetblocks: cfg.AnomalyWatchedNets, NamedNetblocks: cfg.AnomalyWatchedNets,
Alerts: params.Alerts, Alerts: params.Alerts,
}) })
params.Metrics.AddBansAndClients(
params.Ledger,
params.Limiter,
params.Now,
)
params.Metrics.AddRules(params.Rules)
} }
// addOutsideSources adds to params the lists fetched from URLs, the DNSBL // addOutsideSources adds to params the lists fetched from URLs, the DNSBL
// zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its // zones' verdicts, AbuseIPDB's scores from abuseIPDBCheckURL and its
// blacklist from abuseIPDBBlacklistURL, routewatch's answers while its // blacklist from abuseIPDBBlacklistURL, routewatch's answers while its
// setting is set, and GeoJS at geojsURL, as sww builds // setting is set, the metrics newMetrics returns, and GeoJS at geojsURL, as
// them from params' settings and the parts addCounters added. The metrics // sww builds them from params' settings and the parts addCounters added.
// read the lists, the zones' verdicts, and AbuseIPDB's while its key is set.
func addOutsideSources( func addOutsideSources(
params *waf.Params, geojsURL, abuseIPDBCheckURL string, params *waf.Params, geojsURL, abuseIPDBCheckURL string,
) { ) {
cfg := params.Config cfg := params.Config
m, limiter, ledger := params.Metrics, params.Limiter, params.Ledger limiter, ledger := params.Limiter, params.Ledger
processLog, alertQueue := params.ProcessLog, params.Alerts processLog, alertQueue := params.ProcessLog, params.Alerts
params.Lists = reputation.New(reputation.Params{ params.Lists = reputation.New(reputation.Params{
@@ -655,6 +669,9 @@ func addOutsideSources(
}) })
} }
m := newMetrics(params)
params.Metrics = m
params.GeoJS = geojs.New(geojs.Params{ params.GeoJS = geojs.New(geojs.Params{
URL: geojsURL, URL: geojsURL,
Answered: func(answer geojs.Answer) { Answered: func(answer geojs.Answer) {
@@ -667,12 +684,29 @@ func addOutsideSources(
}, },
WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc, WentWithoutAnswer: m.RequestsWithoutGeoJSAnswer.Inc,
}) })
}
m.AddReputation(params.Lists, params.DNSBL) // newMetrics returns the metrics, as sww builds them from params' settings,
// read from the ledger, the table of clients, the rules, the lists, the
// zones' verdicts, and AbuseIPDB's while its key is set.
func newMetrics(params *waf.Params) *metrics.Metrics {
cfg := params.Config
metricsParams := metrics.Params{
BusiestCount: cfg.MetricsBusiestCountriesAndASNs,
InstanceName: cfg.InstanceName,
Now: params.Now,
Bans: params.Ledger,
Clients: params.Limiter,
Rules: params.Rules,
Lists: params.Lists,
DNSBL: params.DNSBL,
}
if cfg.AbuseIPDBKey != "" { if cfg.AbuseIPDBKey != "" {
m.AddAbuseIPDB(params.AbuseIPDB) metricsParams.AbuseIPDB = params.AbuseIPDB
} }
return metrics.New(metricsParams)
} }
// newClient returns an HTTP client that sends requests as they are made, // newClient returns an HTTP client that sends requests as they are made,
@@ -794,7 +828,9 @@ func readResponse(t *testing.T, conn net.Conn) answer {
} }
// wantLine checks the request log line's status and action. // wantLine checks the request log line's status and action.
func wantLine(t *testing.T, line logLine, status int, action string) { func wantLine(
t *testing.T, line logLine, status int, action requestlog.Action,
) {
t.Helper() t.Helper()
if line.Status != status || line.Action != action { if line.Status != status || line.Action != action {
+8 -5
View File
@@ -67,11 +67,14 @@ func (r rateLimits) check(
} }
} }
// overAWholeRateLimit reports whether the heuristic rate_limits will refuse rq // refusesWhateverTheLookupAnswers reports whether the heuristic rate_limits
// at now, whatever its client's limit percentages: the rate limits count it, // will refuse rq at now, whatever its client's limit percentages, which the
// and one more request from its client takes it over a whole rate limit. It // lookup's answer can lower: the rate limits count it, and one more request
// counts nothing. // from its client takes it over a whole rate limit. It counts nothing.
func (r rateLimits) overAWholeRateLimit(rq *request, now time.Time) bool { func (r rateLimits) refusesWhateverTheLookupAnswers(
rq *request,
now time.Time,
) bool {
return r.countedByRateLimits(rq) && return r.countedByRateLimits(rq) &&
r.limiter.OverARateLimit( r.limiter.OverARateLimit(
clientGroup(r.ipv6GroupPrefix, rq.client), clientGroup(r.ipv6GroupPrefix, rq.client),
+1 -1
View File
@@ -121,7 +121,7 @@ func TestIPv6GroupPrefixSetsTheClientTheLimitsCount(t *testing.T) {
// status and action are those of the request that breaks the // status and action are those of the request that breaks the
// limit: a rate limit refuses it, a byte limit passes it on. // limit: a rate limit refuses it, a byte limit passes it on.
status int status int
action string action requestlog.Action
}{ }{
{rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited}, {rateLimitPerMinute, "1", http.StatusForbidden, requestlog.ActionClientIsRateLimited},
{bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward}, {bytesLimitPerMinute, byteLimit, http.StatusOK, requestlog.ActionForward},
+9 -1
View File
@@ -54,13 +54,21 @@ func (r refusalLimit) check(
// What the client was sent, or in observe mode would have been. // What the client was sent, or in observe mode would have been.
status := rq.out.status status := rq.out.status
switch rq.line.WouldAction { switch rq.wouldAction {
case requestlog.ActionBlockedByBlockRule, case requestlog.ActionBlockedByBlockRule,
requestlog.ActionBlockedByCoreRuleSet: requestlog.ActionBlockedByCoreRuleSet:
status = http.StatusForbidden status = http.StatusForbidden
case requestlog.ActionTemporarilyBanned, case requestlog.ActionTemporarilyBanned,
requestlog.ActionPermanentlyBanned: requestlog.ActionPermanentlyBanned:
status = r.banResponse status = r.banResponse
case requestlog.ActionForward,
requestlog.ActionRequestLineTooLong,
requestlog.ActionRequestHeaderTooLarge,
requestlog.ActionClientIsRateLimited,
requestlog.ActionClientDenied,
requestlog.ActionCountryDenied,
requestlog.ActionChecksTimedOut,
requestlog.ActionOwnEndpoint:
} }
r.banMaker.banForLimit(rq, now, hit, status) r.banMaker.banForLimit(rq, now, hit, status)
+2 -3
View File
@@ -12,7 +12,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/bans" "sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
const refusalsLimitPerMinute = "SWW_WAF_REFUSALS_LIMIT_PER_MINUTE" const refusalsLimitPerMinute = "SWW_WAF_REFUSALS_LIMIT_PER_MINUTE"
@@ -52,7 +51,7 @@ func (r refused) send(s *sender, from string) logLine {
// SWW_WAF_REFUSALS_LIMIT_PER_MINUTE at limit, and the settings in env. // SWW_WAF_REFUSALS_LIMIT_PER_MINUTE at limit, and the settings in env.
func startForRefusalLimit( func startForRefusalLimit(
t *testing.T, limit string, env map[string]string, t *testing.T, limit string, env map[string]string,
) (*sender, *clock, *waf.Server) { ) (*sender, *clock, *wafAndParts) {
t.Helper() t.Helper()
settings := map[string]string{ settings := map[string]string{
@@ -74,7 +73,7 @@ func TestRefusalLimitBreaksAtOneOverTheLimit(t *testing.T) {
refusals []refused refusals []refused
// nextAction is the action of the client's next request: the next // nextAction is the action of the client's next request: the next
// request after a ban for a ban rule makes it permanent. // request after a ban for a ban rule makes it permanent.
nextAction string nextAction requestlog.Action
}{ }{
{ {
"block rule", "block rule",
+3 -3
View File
@@ -47,9 +47,9 @@ func (n reputationHitNoter) noteHit(
Host: rq.site, Host: rq.site,
Client: rq.client, Client: rq.client,
Netblock: clientGroup(n.ipv6GroupPrefix, rq.client), Netblock: clientGroup(n.ipv6GroupPrefix, rq.client),
ASN: rq.line.ASN, ASN: rq.lookupAnswer.ASN,
ASName: rq.line.ASName, ASName: rq.lookupAnswer.ASName,
Country: rq.line.Country, Country: rq.lookupAnswer.Country,
Reason: reason, Reason: reason,
Detail: detail, Detail: detail,
}) })
+24 -25
View File
@@ -18,7 +18,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit" "sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation" "sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// The reputation settings. // The reputation settings.
@@ -58,22 +57,22 @@ func TestEachBlocklistActionForAListedAddressAndAListedNetblock(
// statuses and actions are those of a listed client's three // statuses and actions are those of a listed client's three
// requests, and percent their rate_limit_percent, as percentText gives it. // requests, and percent their rate_limit_percent, as percentText gives it.
statuses []int statuses []int
actions []string actions []requestlog.Action
percent string percent string
}{ }{
{ {
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden}, actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none, []requestlog.Action{denied, denied, denied}, none,
}, },
{ {
// Half of 4 requests a minute: the third breaks the limit. // Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden}, limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionClientIsRateLimited}, []requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + blocklistAction, "50 from " + blocklistAction,
}, },
{ {
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK}, actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none, []requestlog.Action{forward, forward, forward}, none,
}, },
} { } {
t.Run(tc.action, func(t *testing.T) { t.Run(tc.action, func(t *testing.T) {
@@ -139,13 +138,13 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
for _, tc := range []struct { for _, tc := range []struct {
heuristic string heuristic string
env map[string]string env map[string]string
load func(*testing.T, *waf.Server) load func(*testing.T, *wafAndParts)
figures string figures string
}{ }{
{ {
"blocklists", "blocklists",
map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny}, map[string]string{blocklistURLs: dropURL, blocklistAction: actionDeny},
func(t *testing.T, server *waf.Server) { func(t *testing.T, server *wafAndParts) {
t.Helper() t.Helper()
loadLists(t, server, map[string][]string{dropURL: {client}}) loadLists(t, server, map[string][]string{dropURL: {client}})
}, },
@@ -157,7 +156,7 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblZones: dnsblZone, dnsblResolver: noResolver,
dnsblAbuseIPDBAction: actionDeny, dnsblAbuseIPDBAction: actionDeny,
}, },
func(_ *testing.T, server *waf.Server) { func(_ *testing.T, server *wafAndParts) {
loadVerdicts(server, map[string][]string{client: {dnsblZone}}) loadVerdicts(server, map[string][]string{client: {dnsblZone}})
}, },
`{"zones": ["` + dnsblZone + `"]}`, `{"zones": ["` + dnsblZone + `"]}`,
@@ -165,7 +164,7 @@ func TestReputationRefusalNamesItsHeuristicAndWhatListedTheClient(
{ {
"abuseipdb", "abuseipdb",
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny}, map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionDeny},
func(_ *testing.T, server *waf.Server) { func(_ *testing.T, server *wafAndParts) {
loadScores(server, map[string]int64{client: 80}) loadScores(server, map[string]int64{client: 80})
}, },
`{"score": 80, "min_score": 75}`, `{"score": 80, "min_score": 75}`,
@@ -442,22 +441,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientADNSBLZoneLists(t *testing.T) {
// statuses and actions are those of a listed client's three // statuses and actions are those of a listed client's three
// requests, and percent their rate_limit_percent, as percentText gives it. // requests, and percent their rate_limit_percent, as percentText gives it.
statuses []int statuses []int
actions []string actions []requestlog.Action
percent string percent string
}{ }{
{ {
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden}, actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none, []requestlog.Action{denied, denied, denied}, none,
}, },
{ {
// Half of 4 requests a minute: the third breaks the limit. // Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden}, limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionClientIsRateLimited}, []requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + dnsblAbuseIPDBAction, "50 from " + dnsblAbuseIPDBAction,
}, },
{ {
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK}, actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none, []requestlog.Action{forward, forward, forward}, none,
}, },
} { } {
t.Run(tc.action, func(t *testing.T) { t.Run(tc.action, func(t *testing.T) {
@@ -560,7 +559,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
for _, tc := range []struct { for _, tc := range []struct {
name string name string
env map[string]string env map[string]string
load func(*waf.Server) load func(*wafAndParts)
want []string want []string
}{ }{
{ {
@@ -568,7 +567,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
map[string]string{ map[string]string{
dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionLog, dnsblZones: dnsblZone, dnsblResolver: noResolver, dnsblAbuseIPDBAction: actionLog,
}, },
func(server *waf.Server) { func(server *wafAndParts) {
loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}}) loadVerdicts(server, map[string][]string{fromDE: {dnsblZone}})
}, },
[]string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone}, []string{"asn=64496", "country=DE", "dnsbl=" + dnsblZone},
@@ -576,7 +575,7 @@ func TestReputationSourcesTagTheRequestWhateverTheirAction(t *testing.T) {
{ {
abuseipdb, abuseipdb,
map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionLog}, map[string]string{abuseIPDBKey: accountKey, dnsblAbuseIPDBAction: actionLog},
func(server *waf.Server) { func(server *wafAndParts) {
loadScores(server, map[string]int64{fromDE: 80}) loadScores(server, map[string]int64{fromDE: 80})
}, },
[]string{abuseipdb, "asn=64496", "country=DE"}, []string{abuseipdb, "asn=64496", "country=DE"},
@@ -954,7 +953,7 @@ func TestClientRefusedForAnOffenceIsCheckedWithAbuseIPDBAtItsNextRequest(
// history then counts. // history then counts.
path string path string
status int status int
action string action requestlog.Action
want ratelimit.Offences want ratelimit.Offences
}{ }{
{ {
@@ -1018,22 +1017,22 @@ func TestEachDNSBLAbuseIPDBActionForAClientAbuseIPDBScoresAtOrOverTheMinimum(
// statuses and actions are those of fromDE's three requests, and // statuses and actions are those of fromDE's three requests, and
// percent their rate_limit_percent, as percentText gives it. // percent their rate_limit_percent, as percentText gives it.
statuses []int statuses []int
actions []string actions []requestlog.Action
percent string percent string
}{ }{
{ {
actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden}, actionDeny, []int{http.StatusForbidden, http.StatusForbidden, http.StatusForbidden},
[]string{denied, denied, denied}, none, []requestlog.Action{denied, denied, denied}, none,
}, },
{ {
// Half of 4 requests a minute: the third breaks the limit. // Half of 4 requests a minute: the third breaks the limit.
limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden}, limitHalf, []int{http.StatusOK, http.StatusOK, http.StatusForbidden},
[]string{forward, forward, requestlog.ActionClientIsRateLimited}, []requestlog.Action{forward, forward, requestlog.ActionClientIsRateLimited},
"50 from " + dnsblAbuseIPDBAction, "50 from " + dnsblAbuseIPDBAction,
}, },
{ {
actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK}, actionLog, []int{http.StatusOK, http.StatusOK, http.StatusOK},
[]string{forward, forward, forward}, none, []requestlog.Action{forward, forward, forward}, none,
}, },
} { } {
t.Run(tc.action, func(t *testing.T) { t.Run(tc.action, func(t *testing.T) {
@@ -1256,7 +1255,7 @@ func verdictsFetched() time.Time {
// verdictsFetched, as dnsbl.json would at start: one that lists the // verdictsFetched, as dnsbl.json would at start: one that lists the
// client from each zone listedBy gives for it, and one that does not from // client from each zone listedBy gives for it, and one that does not from
// each other zone. // each other zone.
func loadVerdicts(server *waf.Server, listedBy map[string][]string) { func loadVerdicts(server *wafAndParts, listedBy map[string][]string) {
verdicts := make( verdicts := make(
[]reputation.Verdict, []reputation.Verdict,
0, 0,
@@ -1281,7 +1280,7 @@ func loadVerdicts(server *waf.Server, listedBy map[string][]string) {
// loadScores puts into server's AbuseIPDB the score scores gives each // loadScores puts into server's AbuseIPDB the score scores gives each
// client, an IPv4 address, fetched at verdictsFetched, as // client, an IPv4 address, fetched at verdictsFetched, as
// abuseipdb-scores.json would at start. // abuseipdb-scores.json would at start.
func loadScores(server *waf.Server, scores map[string]int64) { func loadScores(server *wafAndParts, scores map[string]int64) {
kept := make([]reputation.Score, 0, len(scores)) kept := make([]reputation.Score, 0, len(scores))
for client, score := range scores { for client, score := range scores {
kept = append(kept, reputation.Score{ kept = append(kept, reputation.Score{
@@ -1295,7 +1294,7 @@ func loadScores(server *waf.Server, scores map[string]int64) {
// wantAbuseIPDBChecks checks how many clients server has checked with // wantAbuseIPDBChecks checks how many clients server has checked with
// AbuseIPDB. // AbuseIPDB.
func wantAbuseIPDBChecks(t *testing.T, server *waf.Server, want int) { func wantAbuseIPDBChecks(t *testing.T, server *wafAndParts, want int) {
t.Helper() t.Helper()
if got := server.AbuseIPDB.Checked(); got != want { if got := server.AbuseIPDB.Checked(); got != want {
@@ -1307,7 +1306,7 @@ func wantAbuseIPDBChecks(t *testing.T, server *waf.Server, want int) {
// its lines, fetched at listsFetched, as lists.json would at start. // its lines, fetched at listsFetched, as lists.json would at start.
func loadLists( func loadLists(
t *testing.T, t *testing.T,
server *waf.Server, server *wafAndParts,
copies map[string][]string, copies map[string][]string,
) { ) {
t.Helper() t.Helper()
+42 -20
View File
@@ -21,45 +21,67 @@ import (
// to close its connection without an answer, and the action the log line names. // to close its connection without an answer, and the action the log line names.
type refusal struct { type refusal struct {
status int status int
action string action requestlog.Action
} }
// decision is what the checks note of a request: its log line, and what // decision is what the checks note of a request: its log line, what the
// counting the request needs once it has ended. waf keeps a copy of it // heuristics tell each other, and what counting the request needs once it has
// until the request's report comes. It holds no credentials, since the log line // ended. Nothing reads the log line back to decide. waf keeps a copy of it
// never does, and neither the request's objects nor their bodies. // until the request's report comes. It holds no credentials, since the log
// line never does, and neither the request's objects nor their bodies.
type decision struct { type decision struct {
line requestlog.Line line requestlog.Line
client netip.Addr client netip.Addr
// requestURI is the path with the query, as a ban's notes keep it. // method and userAgent are the request's method and User-Agent, and
requestURI string // requestURI its path with the query, as a ban's notes keep them.
// startRequest sets them.
method, userAgent, requestURI string
// tags are the request's tags, sorted, each once. addTag adds them, for
// tagLists, routeWatchTags, noteLookup, dnsbl, abuseIPDB, ruleFiles and
// coreRuleSet.
tags []string
// site is, for a question, the host the visitor asked for, which the // site is, for a question, the host the visitor asked for, which the
// alerts name, since one waf serves many sites. // alerts and a ban's notes name, since one waf serves many sites.
// answerQuestion sets it.
site string site string
// allowed is true for a question answered with DispositionAllow. // allowed is true for a question answered with DispositionAllow.
allowed bool allowed bool
// lookedUp is true once the client's AS number and country have been // lookedUp is true once the client's AS number and country have been
// looked up, whether or not an answer was there, and lookupAnswer is // looked up, whether or not an answer was there, and lookupAnswer is
// what the lookup gave then, the zero Answer while GeoJS had given none. // what the lookup gave then, the zero Answer while GeoJS had given none.
// noteLookup sets them.
lookedUp bool lookedUp bool
lookupAnswer geojs.Answer lookupAnswer geojs.Answer
// skippedLookupWait is true for a request that did not wait for GeoJS and // skippedLookupWait is true for a request that did not wait for GeoJS and
// found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it, // found no answer kept. SWW_WAF_UNKNOWN_COUNTRY_LIMIT_PERCENT does not apply to it,
// so that not waiting never lowers its client's limits. // so that not waiting never lowers its client's limits. noteLookup sets
// it.
skippedLookupWait bool skippedLookupWait bool
// counted is true for a request the rate limits counted, whose bytes // counted is true for a request the rate limits counted, whose bytes
// the byte limits count once it has ended. rateLimitPercent and // the byte limits count once it has ended. rateLimitPercent and
// bytesLimitPercent are then its client's limit percentages for the rate // bytesLimitPercent are then its client's limit percentages for the rate
// limits and for the byte limits. // limits and for the byte limits. rateLimits sets them.
counted bool counted bool
rateLimitPercent, bytesLimitPercent percentage rateLimitPercent, bytesLimitPercent percentage
// attack is true for a request that matched a ban rule or asked for a // attack is true for a request that matched a ban rule or asked for a
// trap path, blockedByBlockRule for one a block rule refused, and // trap path, blockedByBlockRule for one a block rule refused, and
// blockedByCoreRuleSet for one the Core Rule Set refused, each an offence // blockedByCoreRuleSet for one the Core Rule Set refused, each an offence
// its client's history counts. // its client's history counts. trapPaths and ruleFiles set attack,
// ruleFiles blockedByBlockRule, and coreRuleSet blockedByCoreRuleSet.
attack, blockedByBlockRule, blockedByCoreRuleSet bool attack, blockedByBlockRule, blockedByCoreRuleSet bool
// wouldAction is, in observe mode, the action enforce mode would have
// refused the request with, one of requestlog's Action constants, and ""
// otherwise. check sets it.
wouldAction requestlog.Action
// brokeLimit is true for a request that broke a limit, an offence its
// client's history counts. banForLimit sets it.
brokeLimit bool
// blocklisted is true once blocklists finds a blocklist that lists the
// client, dnsblListed once dnsbl finds a DNSBL zone's verdict that does,
// and abuseIPDBHit once abuseIPDB finds AbuseIPDB's score of it a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
// reputation is the reputation sources that list the client, for the // reputation is the reputation sources that list the client, for the
// notes of a ban the request makes. // notes of a ban the request makes. noteHit adds them.
reputation []bans.ReputationHit reputation []bans.ReputationHit
} }
@@ -72,10 +94,6 @@ type request struct {
in *http.Request in *http.Request
out *responseWriter out *responseWriter
// blocklisted is true once a blocklist is found to list the client,
// dnsblListed once a DNSBL zone's verdict is, and abuseIPDBHit once
// AbuseIPDB's score of it is a hit.
blocklisted, dnsblListed, abuseIPDBHit bool
start time.Time start time.Time
// checked is when the checks were done. // checked is when the checks were done.
checked time.Time checked time.Time
@@ -106,6 +124,8 @@ func (h *handler) startRequest(
rq := &request{ rq := &request{
decision: decision{ decision: decision{
client: client, client: client,
method: r.Method,
userAgent: r.UserAgent(),
requestURI: r.URL.RequestURI(), requestURI: r.URL.RequestURI(),
line: requestlog.Line{ line: requestlog.Line{
Time: requestlog.FormatTime(start), Time: requestlog.FormatTime(start),
@@ -151,11 +171,12 @@ func (h *handler) startRequest(
} }
// addTag adds tag to the request's tags, which stay sorted and hold each tag // addTag adds tag to the request's tags, which stay sorted and hold each tag
// once. // once, and to its log line.
func (rq *request) addTag(tag string) { func (rq *request) addTag(tag string) {
at, found := slices.BinarySearch(rq.line.Tags, tag) at, found := slices.BinarySearch(rq.tags, tag)
if !found { if !found {
rq.line.Tags = slices.Insert(rq.line.Tags, at, tag) rq.tags = slices.Insert(rq.tags, at, tag)
rq.line.Tags = rq.tags
} }
} }
@@ -196,6 +217,7 @@ func (rq *request) check(ctx context.Context) *refusal {
switch { switch {
case rq.h.config.Observe: case rq.h.config.Observe:
rq.wouldAction = action
rq.line.WouldAction = action rq.line.WouldAction = action
return nil return nil
@@ -371,7 +393,7 @@ func (rq *request) addToHistory() {
Status: rq.out.status, Status: rq.out.status,
RequestBytes: rq.requestBytes, RequestBytes: rq.requestBytes,
ResponseBytes: rq.out.bytes, ResponseBytes: rq.out.bytes,
BrokeLimit: rq.line.Offence == requestlog.OffenceLimit, BrokeLimit: rq.brokeLimit,
Attack: rq.attack, Attack: rq.attack,
BlockedByBlockRule: rq.blockedByBlockRule, BlockedByBlockRule: rq.blockedByBlockRule,
BlockedByCoreRuleSet: rq.blockedByCoreRuleSet, BlockedByCoreRuleSet: rq.blockedByCoreRuleSet,
+142
View File
@@ -0,0 +1,142 @@
package waf
import (
"crypto/subtle"
"io"
"net/http"
"strings"
"github.com/go-chi/chi/v5"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// routes returns the router of waf's own endpoints. A question never reaches
// it, since ServeHTTP answers a question first. No check runs on a request
// here: waf's address is on the private network, where the peers are rproxy
// and the admin's tools, not visitors.
func (h *handler) routes() http.Handler {
r := chi.NewRouter()
r.Use(matchDecodedPath)
// The report writes no log line of its own: it finishes the line of the
// request it reports.
r.Post(wafprotocol.ReportPath, h.handleReport())
r.Group(func(r chi.Router) {
r.Use(h.logOwnEndpoint)
// The answer to a path with no endpoint, and to a method the path has
// no endpoint for, is set in this group, so that it is logged too.
r.NotFound(answerNotFound)
r.MethodNotAllowed(answerNotFound)
r.Get(wafprotocol.HealthPath, handleHealth())
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.MetricsToken))
r.Get(MetricsPath, h.metrics.ServeHTTP)
})
// The handlers under BansPath/ and ClientsPath read the address from
// the path themselves, so that one that is not an address gets 400
// after the token check.
r.Group(func(r chi.Router) {
r.Use(requireToken(h.config.AdminToken))
r.Get(BansPath, h.handleListBans())
r.Post(BansPath, h.handleAddBan())
r.Delete(BansPath+"/*", h.handleLiftBans())
r.Get(ClientsPath+"*", h.handleShowClient())
})
})
return r
}
// matchDecodedPath has the router match the decoded path, which the handlers
// read. chi matches the path as sent when it is encoded in an unusual way,
// which would make /_sww/%68ealthz a path with no endpoint.
func matchDecodedPath(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
chi.RouteContext(r.Context()).RoutePath = r.URL.Path
next.ServeHTTP(w, r)
})
}
// logOwnEndpoint writes one log line for each request, with the action
// own_endpoint. The line's request_bytes is how much of the body the handler
// read.
func (h *handler) logOwnEndpoint(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
rq.line.Action = requestlog.ActionOwnEndpoint
counted := r.Clone(r.Context())
counted.Body = &countedBody{
ReadCloser: r.Body,
read: &rq.requestBytes,
}
next.ServeHTTP(rq.out, counted)
})
}
// countedBody is a request's body that adds the bytes read of it to read.
type countedBody struct {
io.ReadCloser
read *int64
}
// Read reads part of the body.
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.ReadCloser.Read(p)
*b.read += int64(n)
return n, err
}
// requireToken returns middleware that lets a request through only when it
// carries token, as Authorization: Bearer <token>. It answers 401 to a request
// without it. While token is unset, it answers 404, as a path with no endpoint
// does.
func requireToken(token string) func(http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
switch {
case token == "":
answerNotFound(w, r)
case !hasToken(r, token):
w.Header().Set("WWW-Authenticate", "Bearer")
http.Error(
w,
http.StatusText(http.StatusUnauthorized),
http.StatusUnauthorized,
)
default:
next.ServeHTTP(w, r)
}
},
)
}
}
// hasToken reports whether r carries token, as Authorization: Bearer
// <token>.
func hasToken(r *http.Request, token string) bool {
scheme, sent, _ := strings.Cut(r.Header.Get("Authorization"), " ")
return strings.EqualFold(scheme, "Bearer") &&
subtle.ConstantTimeCompare([]byte(sent), []byte(token)) == 1
}
// answerNotFound answers 404, for a path with no endpoint, and for a method
// the path has no endpoint for.
func answerNotFound(w http.ResponseWriter, _ *http.Request) {
http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
}
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"time" "time"
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
) )
// routeWatchURL is SWW_WAF_ROUTEWATCH_URL, and tagsMissing the log line's field // routeWatchURL is SWW_WAF_ROUTEWATCH_URL, and tagsMissing the log line's field
@@ -58,7 +57,7 @@ func startRouteWatch(t *testing.T, routes map[string]standInRoute) string {
// waitForRouteWatch waits until routewatch's answer about from, in the client // waitForRouteWatch waits until routewatch's answer about from, in the client
// group of the given length, is kept, so that the next request gets its tags. // group of the given length, is kept, so that the next request gets its tags.
func waitForRouteWatch( func waitForRouteWatch(
t *testing.T, server *waf.Server, from string, bits int, t *testing.T, server *wafAndParts, from string, bits int,
) { ) {
t.Helper() t.Helper()
+3 -2
View File
@@ -72,9 +72,10 @@ func (r ruleFiles) check(
), ),
figures: figures, figures: figures,
} }
default: case rules.ActionLog:
return nil
} }
return nil
} }
// refusedByRule reports whether ruleFiles refuses a request that matched // refusedByRule reports whether ruleFiles refuses a request that matched
+1 -1
View File
@@ -184,7 +184,7 @@ func TestRateLimitExemptNetsAreNeitherCountedNorRefused(t *testing.T) {
type sentRequest struct { type sentRequest struct {
client string client string
status int status int
action string action requestlog.Action
} }
// wantAnswers sends requests to sww at addr one after another and // wantAnswers sends requests to sww at addr one after another and
+1 -1
View File
@@ -23,7 +23,7 @@ func (t tagDeny) check(
rq *request, rq *request,
_ time.Time, _ time.Time,
) *verdict { ) *verdict {
for _, tag := range rq.line.Tags { for _, tag := range rq.tags {
if slices.Contains(t.tags, tag) { if slices.Contains(t.tags, tag) {
return &verdict{ return &verdict{
action: requestlog.ActionClientDenied, action: requestlog.ActionClientDenied,
+54 -58
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"errors" "errors"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"strings" "strings"
"time" "time"
@@ -30,12 +31,10 @@ const (
// is far above rproxy's, and a question refused for its size would have rproxy // is far above rproxy's, and a question refused for its size would have rproxy
// let the request through. // let the request through.
func New(params Params) *Server { func New(params Params) *Server {
h := newHandler(params) s := &Server{h: newHandler(params)}
s.httpServer = &http.Server{
return &Server{
Server: &http.Server{
Addr: params.Config.ListenAddr, Addr: params.Config.ListenAddr,
Handler: h, Handler: s,
ErrorLog: slog.NewLogLogger( ErrorLog: slog.NewLogLogger(
params.ProcessLog.Handler(), params.ProcessLog.Handler(),
slog.LevelWarn, slog.LevelWarn,
@@ -43,23 +42,37 @@ func New(params Params) *Server {
ReadTimeout: wafTimeout, ReadTimeout: wafTimeout,
WriteTimeout: wafTimeout, WriteTimeout: wafTimeout,
IdleTimeout: wafIdleTimeout, IdleTimeout: wafIdleTimeout,
},
Ledger: h.ledger,
Limiter: h.limiter,
GeoJS: h.geojs,
Anomalies: h.anomalies,
LookupFile: h.lookupFile,
Lists: params.Lists,
DNSBL: params.DNSBL,
AbuseIPDB: params.AbuseIPDB,
RouteWatch: params.RouteWatch,
Metrics: h.metrics,
} }
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request to waf, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
} }
// ServeHTTP handles one request to waf: a request carrying // ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for waf // wafprotocol.ClientHeader is rproxy's question, and any other is for one of
// itself. // waf's own endpoints.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) { func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 { if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 {
h.answerQuestion(w, r) h.answerQuestion(w, r)
@@ -67,23 +80,27 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
return return
} }
h.answerOwnEndpoint(w, r) h.ownEndpoints.ServeHTTP(w, r)
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait // refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about // for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: the heuristic blocklists or rate_limits will refuse it, as a // its client: a heuristic of afterLookup that implements refusesWithoutLookup
// function in that heuristic's file says. In observe mode, which refuses // says it will refuse it. In observe mode, which refuses nothing, none is.
// nothing, none is. rproxy lets a request through when waf answers too late. // rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool { func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe { if rq.h.config.Observe {
return false return false
} }
listedBy := rq.h.blocklists.lists.ListedBy(rq.client) for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return rq.h.blocklists.refusedByBlocklists(listedBy) || return false
rq.h.rateLimits.overAWholeRateLimit(rq, now)
} }
// answerQuestion runs the checks on the visitor's request a question // answerQuestion runs the checks on the visitor's request a question
@@ -202,8 +219,8 @@ func (rq *request) allow() {
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow} answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders { if rq.h.config.AddLookupHeaders {
answer.ASN = rq.line.ASN answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.line.Country answer.Country = rq.lookupAnswer.Country
} }
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer) wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
@@ -240,34 +257,11 @@ func (rq *request) deny(r refusal) {
http.Error(rq.out, http.StatusText(r.status), r.status) http.Error(rq.out, http.StatusText(r.status), r.status)
} }
// answerOwnEndpoint answers a request to waf's own address. No check // handleReport answers POST wafprotocol.ReportPath: it finishes the request
// runs on it: the address is on the private network, where the peers are // the report names, if it holds that request's decision, and ignores the
// rproxy and the admin's tools, not visitors. // report otherwise.
func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) { func (h *handler) handleReport() http.HandlerFunc {
if r.Method == http.MethodPost && r.URL.Path == wafprotocol.ReportPath { return func(w http.ResponseWriter, r *http.Request) {
h.takeReport(w, r)
return
}
peer := peerAddress(r)
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
rq.answerHealth()
return
}
rq.answerAdmin()
}
// takeReport answers POST wafprotocol.ReportPath: it finishes the request the
// report names, if it holds that request's decision, and ignores the report
// otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r) report, err := wafprotocol.ReadReport(w, r)
if err != nil { if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error()) h.processLog.Warn("a report is refused", "error", err.Error())
@@ -277,13 +271,15 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
} }
held, found := h.awaitingReport.Peek(report.RequestID) held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the id. A // Remove reports true to one report only, of several that name the
// question still being handled is held as nil. // id. A question still being handled is held as nil.
if found && held != nil && h.awaitingReport.Remove(report.RequestID) { if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report) h.finishWith(r.Context(), held, report)
} }
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
}
} }
// finishWith finishes a request with what its report says, and writes the // finishWith finishes a request with what its report says, and writes the
+16 -14
View File
@@ -58,7 +58,7 @@ type WAF struct {
t *testing.T t *testing.T
addr string addr string
out *output out *output
server *waf.Server server *wafAndParts
clock *clock clock *clock
queue *alerts.Queue queue *alerts.Queue
} }
@@ -70,14 +70,14 @@ func startWAF(
) *WAF { ) *WAF {
t.Helper() t.Helper()
return startWAFWith(t, geojsURL, env, func(*waf.Server) {}) return startWAFWith(t, geojsURL, env, func(*wafAndParts) {})
} }
// startWAFWith is startWAF with change applied to the server before // startWAFWith is startWAF with change applied to the server before
// it starts. // it starts.
func startWAFWith( func startWAFWith(
t *testing.T, geojsURL string, env map[string]string, t *testing.T, geojsURL string, env map[string]string,
change func(*waf.Server), change func(*wafAndParts),
) *WAF { ) *WAF {
t.Helper() t.Helper()
@@ -214,13 +214,13 @@ func wantDisposition(
t *testing.T, t *testing.T,
got answer, got answer,
status int, status int,
disposition string, disposition wafprotocol.Disposition,
) { ) {
t.Helper() t.Helper()
wantStatus(t, got, status) wantStatus(t, got, status)
if answered := got.header.Get(wafprotocol.DispositionHeader); answered != if answered := got.header.Get(wafprotocol.DispositionHeader); answered !=
disposition { string(disposition) {
t.Errorf( t.Errorf(
"answered with disposition %q, want %q", "answered with disposition %q, want %q",
answered, answered,
@@ -703,7 +703,9 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t, t,
"", "",
map[string]string{metricsToken: token}, map[string]string{metricsToken: token},
func(server *waf.Server) { server.ReadTimeout = readTimeout }, func(server *wafAndParts) {
server.HTTPServer().ReadTimeout = readTimeout
},
) )
conn := dial(t, s.addr) conn := dial(t, s.addr)
@@ -1562,7 +1564,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
env map[string]string env map[string]string
path string path string
status int status int
action string action requestlog.Action
}{ }{
{ {
"deny", map[string]string{denyNets: client}, "/", http.StatusForbidden, "deny", map[string]string{denyNets: client}, "/", http.StatusForbidden,
@@ -1834,16 +1836,16 @@ func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
func TestServerHasTheTimeLimitsSPECGives(t *testing.T) { func TestServerHasTheTimeLimitsSPECGives(t *testing.T) {
t.Parallel() t.Parallel()
s := startWAF(t, "", nil) server := startWAF(t, "", nil).server.HTTPServer()
if s.server.ReadTimeout != time.Minute || if server.ReadTimeout != time.Minute ||
s.server.WriteTimeout != time.Minute || server.WriteTimeout != time.Minute ||
s.server.IdleTimeout != 2*time.Minute { server.IdleTimeout != 2*time.Minute {
t.Errorf( t.Errorf(
"the read, write and idle limits are %v, %v and %v, want 1m, 1m and 2m", "the read, write and idle limits are %v, %v and %v, want 1m, 1m and 2m",
s.server.ReadTimeout, server.ReadTimeout,
s.server.WriteTimeout, server.WriteTimeout,
s.server.IdleTimeout, server.IdleTimeout,
) )
} }
} }
+3 -3
View File
@@ -8,7 +8,7 @@ import "net/http"
type Answer struct { type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow, // Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose. // DispositionDeny or DispositionClose.
Disposition string Disposition Disposition
// ASN and Country are the client's AS number and country, each empty // ASN and Country are the client's AS number and country, each empty
// when the answer does not give it. // when the answer does not give it.
ASN, Country string ASN, Country string
@@ -17,7 +17,7 @@ type Answer struct {
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer, // SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty. // leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) { func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition) header.Set(DispositionHeader, string(answer.Disposition))
if answer.ASN != "" { if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN) header.Set(ASNHeader, answer.ASN)
@@ -31,7 +31,7 @@ func SetAnswerHeaders(header http.Header, answer Answer) {
// ReadAnswer returns what the headers of waf's answer tell rproxy, and // ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition. // whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) { func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader) disposition := Disposition(header.Get(DispositionHeader))
switch disposition { switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose: case DispositionAllow, DispositionDeny, DispositionClose:
+7 -3
View File
@@ -27,18 +27,22 @@ const (
CountryHeader = "X-Client-Country" CountryHeader = "X-Client-Country"
) )
// Disposition is what waf tells rproxy to do with a request, in
// DispositionHeader.
type Disposition string
// The values of DispositionHeader. // The values of DispositionHeader.
const ( const (
// DispositionAllow tells rproxy to pass the request to the app. // DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow" DispositionAllow Disposition = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status, // DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body. // headers and body.
DispositionDeny = "deny" DispositionDeny Disposition = "deny"
// DispositionClose tells rproxy to close the visitor's connection // DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close. // without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as // waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through. // waf being down, and rproxy would then let the request through.
DispositionClose = "close" DispositionClose Disposition = "close"
) )
// OwnEndpointPrefix starts the path of every request for sww itself: waf's // OwnEndpointPrefix starts the path of every request for sww itself: waf's