Compare commits

..
Author SHA1 Message Date
sneak 55216c34c6 waf: the verdict comment no longer reads as banning nothing
check / check (push) Waiting to run
The doc comment of verdict said a verdict of afterAnswer or onReport
"bans and refuses nothing", which reads as banning nothing, while
refusalLimit and byteLimits give one only when they ban. It now says
that such a verdict refuses nothing.

Model: opus-5-5
2026-11-20 00:54:42 +00:00
sneak c61f054e50 waf: a verdict carries the status the visitor gets (closes #443)
check / check (push) Waiting to run
Each heuristic that refuses a request sets the status on its verdict.
check copies it into the refusal, or in observe mode notes it, and no
longer works the status out from the action. refusalLimit takes the
status from that verdict instead of from a switch over the action.

Model: opus-5-5
2026-11-20 00:36:19 +00:00
32 changed files with 2230 additions and 2475 deletions

No files matched your search

+30 -31
View File
@@ -828,8 +828,8 @@ The supported apps are:
- `vaultwarden`; - `vaultwarden`;
- `wordpress`. - `wordpress`.
Each app's fixes are in one file named for the app, such as Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are: one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system 1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path: files (930120), shell paths (932160) and command names (932260), on any path:
@@ -862,8 +862,8 @@ Each app's fixes are in one file named for the app, such as
3. On gitea's paths, some fields skip the rules that refused ordinary gitea 3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name. or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it `internal/coreruleset/gitea.conf` lists each rule and the request it refused.
refused. The fields are: The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of - `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`, commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull `SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1048,7 +1048,7 @@ Each app's fixes are in one file named for the app, such as
- The `Content-Range` header of an artifact upload from - The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450). `actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The - The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/apps/gitea.conf`. numbers are in `internal/coreruleset/gitea.conf`.
- `password` on these paths: - `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`; - `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`; - `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1080,8 +1080,8 @@ Each app's fixes are in one file named for the app, such as
such as `owner/python3-app` would have that push refused. such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf` ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
lists each rule and the request it refused. The REST API has four addresses: each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...` - `/wp-json/...`
- `/index.php/wp-json/...` - `/index.php/wp-json/...`
@@ -1173,9 +1173,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root. the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
each rule and the request it refused. These query fields skip the lists of rule and the request it refused. These query fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260): files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`, - On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`. `info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1265,10 +1265,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too. hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf` ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
lists each rule and the request it refused. These fields skip the lists of each rule and the request it refused. These fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260), in the query or files and command names (930120, 932120, 932250, 932260), in the query or the
the body: body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/` - On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`: and `/wiki/`:
@@ -1374,9 +1374,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`. site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf` ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
lists each rule and the request it refused. These fields skip the lists of each rule and the request it refused. These fields skip the lists of system
system files and command names (930120, 932120, 932250, 932260), in the query: files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and - On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`, `/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1537,7 +1537,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there. refused ordinary Vaultwarden requests there.
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it `internal/coreruleset/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text. sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1588,8 +1588,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips `token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has `internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden; `<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused. `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1611,7 +1611,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed. ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it `internal/coreruleset/nextcloud.conf` lists each rule and the request it
refused. refused.
The sync client and the Files app use WebDAV on these paths: The sync client and the Files app use WebDAV on these paths:
@@ -1621,8 +1621,7 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`; - `/remote.php/webdav/`;
- `/public.php/webdav/`. - `/public.php/webdav/`.
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which These methods are allowed there, besides those of the first change:
every app gets:
- `PROPFIND` and `PROPPATCH`; - `PROPFIND` and `PROPPATCH`;
- `MKCOL`; - `MKCOL`;
@@ -1766,7 +1765,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not. gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
request it refused. request it refused.
`q` skips rules on some of Forgejo's searches. The searches are: `q` skips rules on some of Forgejo's searches. The searches are:
@@ -1812,10 +1811,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix). puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
rule and the request it refused. Ghost's admin and integrations call the Admin and the request it refused. Ghost's admin and integrations call the Admin API
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`. ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`, Eight rules refused words that name commands and files, such as `python3`,
@@ -1953,9 +1952,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved. skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and `internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
the passwords sent when creating the first staff user, accepting an invitation, passwords sent when creating the first staff user, accepting an invitation, and
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`, resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach `; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused. Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
+1450 -1627
View File
File diff suppressed because it is too large. Load diff
+65 -43
View File
@@ -12,10 +12,11 @@
package coreruleset package coreruleset
import ( import (
"embed" _ "embed"
"errors" "errors"
"fmt" "fmt"
"io" "io"
"maps"
"net" "net"
"net/http" "net/http"
"net/netip" "net/netip"
@@ -34,7 +35,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the // directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while // paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for // SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the fixes of every app in appFixes for the third. // the second, and the directives of every app in apps for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the // Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which // Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no // SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -53,12 +54,13 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\ SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d" setvar:tx.blocking_paranoia_level=%d"
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS. # The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\ SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'" setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# Expect and Content-Encoding are taken off the Core Rule Set's list of the # The second: Expect and Content-Encoding are taken off the Core Rule Set's
# headers it refuses. Content-Encoding goes back on it for a # list of the headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives). # body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\ SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \ setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -84,17 +86,18 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason. # The apps' fixes come before the rules for the same reason.
%s %s
# Only the rules for requests are loaded, and no response is inspected. # The fifth: only the rules for requests are loaded, and no response is
# inspected.
Include @owasp_crs/REQUEST-*.conf Include @owasp_crs/REQUEST-*.conf
# redirect_uri is not checked for a URL naming an IP address or # The third: redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case. # localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as # ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too. # well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri" SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri" SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# Referer is not checked for a Unix command without arguments # The fourth: Referer is not checked for a Unix command without arguments
# or for Java starting a process. # or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer" SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -119,9 +122,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \ "@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON" "id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# Content-Encoding goes back on the list of the headers the Core Rule Set # The rest of the second change: Content-Encoding is refused again on a
# refuses (900250 in directives) for a body of a kind it reads, since a # body of a kind the Core Rule Set reads, since a compressed body cannot be
# compressed body cannot be inspected. # inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \ SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\ "id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \ setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -142,19 +145,15 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'" severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
` `
// appFixes holds the fixes of each app whose ordinary requests the Core // app is an app whose ordinary requests the Core Rule Set would refuse,
// Rule Set would refuse, in a file named for the app by the name // with the fixes that let them pass.
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
type app struct { type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without. // cookies are the cookies the Core Rule Set reads a request without.
cookies []string cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname // forkOf names the app this one is a fork of, "" for none. A hostname
@@ -162,34 +161,60 @@ type app struct {
forkOf string forkOf string
} }
// apps are the apps that need more than their file in appFixes, by the //go:embed gitea.conf
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash. var giteaDirectives string
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
// //
//nolint:gochecknoglobals // a constant cannot be a map //nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{ var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": { "forgejo": {
cookies: []string{"flash"}, directives: forgejoDirectives,
forkOf: "gitea", cookies: []string{"flash"},
forkOf: "gitea",
}, },
"ghost": {directives: ghostDirectives},
"gitea": { "gitea": {
cookies: []string{"gitea_flash", "redirect_to"}, directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
}, },
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
} }
// AppNames returns the names of the apps whose fixes sww has, in // AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order. // alphabetical order.
func AppNames() []string { func AppNames() []string {
// An embedded directory always reads. return slices.Sorted(maps.Keys(apps))
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
} }
// Params are what New needs. // Params are what New needs.
@@ -348,11 +373,8 @@ func New(params Params) (*CoreRuleSet, error) {
} }
var fixes strings.Builder var fixes strings.Builder
for _, name := range AppNames() { for _, name := range AppNames() {
// An embedded file always reads. fixes.WriteString(apps[name].directives + "\n")
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
} }
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions, text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
+3 -3
View File
@@ -17,7 +17,7 @@ const (
) )
// giteaSearches are searches of the pages gitea and Forgejo share whose q // giteaSearches are searches of the pages gitea and Forgejo share whose q
// skips the lists of system files and command names, in gitea's fixes. // skips the lists of system files and command names, in the eighth change.
// gitea serves its admin pages under /-/admin/ and Forgejo under /admin/. // gitea serves its admin pages under /-/admin/ and Forgejo under /admin/.
// //
//nolint:gochecknoglobals // a constant cannot be a list //nolint:gochecknoglobals // a constant cannot be a list
@@ -222,8 +222,8 @@ func TestGiteaSearchExclusionsAreNarrow(t *testing.T) {
} }
// Under /admin/, where Forgejo serves its admin pages, only the hard // Under /admin/, where Forgejo serves its admin pages, only the hard
// values are left out: Drupal's fixes let them through in q on every path // values are left out: the tenth change lets them through in q on every
// there. // path there, for Drupal.
for _, target := range []string{ for _, target := range []string{
"/admin/packages/delete?q=", "/admin/packages/delete?q=",
"/admin/packages?other=", "/admin/packages?other=",
File renamed without changes.
File renamed without changes.
@@ -37,10 +37,9 @@
SecRule &TX:app_nextcloud "@eq 0" \ SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD" "id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides PUT, # The sync client and the Files app use these WebDAV methods besides those
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and # the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/. # older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
# 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite # PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload # MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload # COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
+1 -9
View File
@@ -1,14 +1,6 @@
package rproxy package rproxy
import ( import "time"
"net/http"
"time"
)
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
}
// StopBreakerClock has the circuit breaker of s read at as the time, so that how // StopBreakerClock has the circuit breaker of s read at as the time, so that how
// long requests take cannot matter to it. It is called before s serves. // long requests take cannot matter to it. It is called before s serves.
+5 -5
View File
@@ -56,7 +56,7 @@ func TestServerHasTheFixedLimits(t *testing.T) {
InstanceName: "test", InstanceName: "test",
Level: slog.LevelInfo, Level: slog.LevelInfo,
}), }),
}).HTTPServer() })
if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 || if server.Addr != ":8080" || server.MaxHeaderBytes != 28<<10 ||
server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute { server.IdleTimeout != 2*time.Minute || server.ReadHeaderTimeout != time.Minute {
@@ -709,7 +709,7 @@ func TestABodyLeftUnreadIsReadOnlyUntilTheRequestTimeLimit(t *testing.T) {
before := time.Now() before := time.Now()
server.ServeHTTP(w, req) server.Handler.ServeHTTP(w, req)
if w.Code != http.StatusNoContent { if w.Code != http.StatusNoContent {
t.Errorf("status %d, want the app's 204", w.Code) t.Errorf("status %d, want the app's 204", w.Code)
@@ -762,7 +762,7 @@ func TestABodyLeftUnreadIsReadNoMoreOnceTheTimeToSendTheAnswerRunsOut(
before := time.Now() before := time.Now()
server.ServeHTTP(w, req) server.Handler.ServeHTTP(w, req)
if w.Code != http.StatusOK { if w.Code != http.StatusOK {
t.Errorf("status %d, want the app's 200", w.Code) t.Errorf("status %d, want the app's 200", w.Code)
@@ -815,7 +815,7 @@ func TestAnAnswerRProxyGivesItselfHasTheResponseTimeToBeSent(t *testing.T) {
before := time.Now() before := time.Now()
server.ServeHTTP(w, req) server.Handler.ServeHTTP(w, req)
if w.Code != tc.status { if w.Code != tc.status {
t.Errorf("status %d, want %d", w.Code, tc.status) t.Errorf("status %d, want %d", w.Code, tc.status)
@@ -856,7 +856,7 @@ func TestAnAnswerRProxyGivesItselfHasTenSecondsToBeSentAtLeast(
before := time.Now() before := time.Now()
server.ServeHTTP(w, req) server.Handler.ServeHTTP(w, req)
if w.writeDeadline.Before(before.Add(least)) { if w.writeDeadline.Before(before.Add(least)) {
t.Errorf( t.Errorf(
+1 -1
View File
@@ -583,7 +583,7 @@ func TestAQuestionNamesNoZone(t *testing.T) {
req.RemoteAddr = tc.remoteAddr req.RemoteAddr = tc.remoteAddr
req.Header.Set(forwardedForHeader, tc.forwardedFor) req.Header.Set(forwardedForHeader, tc.forwardedFor)
server.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t) asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client || if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
+13 -36
View File
@@ -16,7 +16,6 @@ import (
"io" "io"
"log" "log"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"net/url" "net/url"
"slices" "slices"
@@ -64,8 +63,9 @@ type Params struct {
// Server is the server `sww rproxy` runs. // Server is the server `sww rproxy` runs.
type Server struct { type Server struct {
httpServer *http.Server *http.Server
h *handler
h *handler
} }
// New returns the server `sww rproxy` runs. Go's server itself refuses // New returns the server `sww rproxy` runs. Go's server itself refuses
@@ -94,40 +94,17 @@ func New(params Params) *Server {
breakerClock: time.Now, breakerClock: time.Now,
} }
s := &Server{h: h} return &Server{
s.httpServer = &http.Server{ Server: &http.Server{
Addr: params.Config.ListenAddr, Addr: params.Config.ListenAddr,
Handler: s, Handler: h,
ErrorLog: errorLog, ErrorLog: errorLog,
ReadHeaderTimeout: params.Config.MaxRequestTime, ReadHeaderTimeout: params.Config.MaxRequestTime,
IdleTimeout: idleTimeout, IdleTimeout: idleTimeout,
MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes, MaxHeaderBytes: requestHeaderMaxBytes - goServerExtraHeaderBytes,
},
h: h,
} }
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request from a visitor, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
} }
// WaitForReports returns once the reports being sent have gone or failed, // WaitForReports returns once the reports being sent have gone or failed,
+1 -1
View File
@@ -710,7 +710,7 @@ func TestAnAnswerWithoutAContentTypeGetsNone(t *testing.T) {
) )
w := httptest.NewRecorder() w := httptest.NewRecorder()
server.ServeHTTP(w, req) server.Handler.ServeHTTP(w, req)
// Go's server guesses a type when the first part of the body goes before the // Go's server guesses a type when the first part of the body goes before the
// headers are flushed, which is a matter of timing here. So this looks for what // headers are flushed, which is a matter of timing here. So this looks for what
+1 -9
View File
@@ -95,20 +95,12 @@ func background(loop func(context.Context)) fx.Hook {
) )
} }
// daemonServer is the server a daemon runs, waf.Server or rproxy.Server: the
// methods of Go's http.Server that serve starts and stops it with.
type daemonServer interface {
Serve(listener net.Listener) error
Shutdown(ctx context.Context) error
Close() error
}
// serve serves requests on listener from the start. At the stop it gives the // serve serves requests on listener from the start. At the stop it gives the
// requests in progress shutdownTimeout to finish, then closes their // requests in progress shutdownTimeout to finish, then closes their
// connections. When serving fails, it logs why and sends the error to // connections. When serving fails, it logs why and sends the error to
// failed. // failed.
func serve( func serve(
lc fx.Lifecycle, server daemonServer, listener net.Listener, lc fx.Lifecycle, server *http.Server, listener net.Listener,
processLog *slog.Logger, failed servingFailed, processLog *slog.Logger, failed servingFailed,
) { ) {
served := make(chan error, 1) served := make(chan error, 1)
+3 -7
View File
@@ -4,6 +4,7 @@ import (
"context" "context"
"log/slog" "log/slog"
"net" "net"
"net/http"
"os" "os"
"go.uber.org/fx" "go.uber.org/fx"
@@ -43,13 +44,8 @@ func RunRProxy(ctx context.Context, params Params) int {
app := fx.New( app := fx.New(
fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed), fx.Supply(cfg, processLog, requestLog{params.Stdout}, failed),
fx.Provide( fx.Provide(
// run waits for rproxy.Server's reports, and serve takes it as a newRProxyServer,
// daemonServer. func(server *rproxy.Server) *http.Server { return server.Server },
fx.Annotate(
newRProxyServer,
fx.As(fx.Self()),
fx.As(new(daemonServer)),
),
newRProxyListener, newRProxyListener,
), ),
fx.Invoke(serve), fx.Invoke(serve),
+3 -2
View File
@@ -6,6 +6,7 @@ import (
"io" "io"
"log/slog" "log/slog"
"net" "net"
"net/http"
"time" "time"
"go.uber.org/fx" "go.uber.org/fx"
@@ -96,8 +97,8 @@ func RunWAF(ctx context.Context, params Params) int {
newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile, newAlertQueue, newRuleFiles, newCoreRuleSet, newLookupFile,
newLimiter, newLedger, newAnomalyCounters, newLimiter, newLedger, newAnomalyCounters,
newLists, newDNSBL, newAbuseIPDB, newRouteWatch, newLists, newDNSBL, newAbuseIPDB, newRouteWatch,
newMetrics, newGeoJS, newMetrics, newGeoJS, newWAFServer,
fx.Annotate(newWAFServer, fx.As(new(daemonServer))), func(server *waf.Server) *http.Server { return server.Server },
newWAFListener, newStateFiles, newWAFListener, newStateFiles,
), ),
// fx stops hooks in the reverse of the order they were added in. The // fx stops hooks in the reverse of the order they were added in. The
-58
View File
@@ -1,58 +0,0 @@
package waf
// The heuristic active_ban. It reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the
// bans are the ban ledger's.
import (
"context"
"time"
)
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
+1 -1
View File
@@ -717,7 +717,7 @@ func askDirectly(
recorder := httptest.NewRecorder() recorder := httptest.NewRecorder()
began := time.Now() began := time.Now()
server.ServeHTTP(recorder, req) server.Handler.ServeHTTP(recorder, req)
return recorder.Code, time.Since(began) return recorder.Code, time.Since(began)
} }
+53 -2
View File
@@ -1,9 +1,11 @@
package waf package waf
// banMaker, which makes the bans of several heuristics. The bans are the ban // The heuristic active_ban, and banMaker, which makes the bans of several
// ledger's. // heuristics. active_ban reads SWW_WAF_MODE and SWW_WAF_BAN_RESPONSE; the bans
// are the ban ledger's.
import ( import (
"context"
"net/netip" "net/netip"
"time" "time"
@@ -15,6 +17,55 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog" "sneak.berlin/go/smallwebwaf/internal/requestlog"
) )
// activeBan refuses a client a ban covers, with SWW_WAF_BAN_RESPONSE: a ban on
// a netblock the client is in, in force at now. It notes for the log line when
// that ban ends. A request that makes the ban permanent, or in observe mode
// would have, raises the alert for it. Its figures are the ban's netblock and
// when it ends.
type activeBan struct {
banMaker banMaker
}
func (activeBan) name() string { return "active_ban" }
func (a activeBan) check(
_ context.Context,
rq *request,
now time.Time,
) *verdict {
ledger := a.banMaker.ledger
check := ledger.Check
if a.banMaker.observe {
check = ledger.Find // the ban refuses nothing, and stays as it is
}
ban, banned, madePermanent := check(rq.client, now)
var expires string
if banned {
expires = banExpires(ban)
rq.line.BanExpires = expires
}
if madePermanent {
ban.Expires = time.Time{} // the ban made permanent, which Find leaves as it is
a.banMaker.alertBan(rq, ban)
}
if !banned {
return nil
}
return &verdict{
action: banAction(ban.Permanent()),
status: a.banMaker.banResponse,
figures: map[string]any{
figureNetblock: ban.Netblock.String(), "ban_expires": expires,
},
}
}
// banAction is the action of a request refused under a ban or for making one: // banAction is the action of a request refused under a ban or for making one:
// permanent says the ban never runs out. // permanent says the ban never runs out.
func banAction(permanent bool) requestlog.Action { func banAction(permanent bool) requestlog.Action {
+1 -1
View File
@@ -592,7 +592,7 @@ func serveFromDE(
req.Header.Set(wafprotocol.ClientHeader, fromDE) req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id") req.Header.Set(requestIDHeader, "the-id")
server.ServeHTTP(httptest.NewRecorder(), req) server.Handler.ServeHTTP(httptest.NewRecorder(), req)
} }
// wantPercent checks a limit percentage that a log line or a ban's notes // wantPercent checks a limit percentage that a log line or a ban's notes
-7
View File
@@ -1,11 +1,4 @@
package waf package waf
import "net/http"
// BesideLookupTimeLimit is besideLookupTimeLimit. // BesideLookupTimeLimit is besideLookupTimeLimit.
const BesideLookupTimeLimit = besideLookupTimeLimit const BesideLookupTimeLimit = besideLookupTimeLimit
// HTTPServer is Go's server s holds, for tests that read its settings.
func (s *Server) HTTPServer() *http.Server {
return s.httpServer
}
-20
View File
@@ -80,26 +80,6 @@ func (rq *request) startLookUp(
return result return result
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// noteLookup notes result for the log line and as the request's asn= and // noteLookup notes result for the log line and as the request's asn= and
// country= tags. // country= tags.
func (rq *request) noteLookup(result lookupResult) { func (rq *request) noteLookup(result lookupResult) {
+328
View File
@@ -0,0 +1,328 @@
// Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf
import (
"cmp"
"io"
"log/slog"
"net/http"
"net/netip"
"time"
lru "github.com/hashicorp/golang-lru/v2"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
// BansPath is where an admin lists and adds bans, and, followed by / and
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
const BansPath = "/_sww/bans"
// ClientsPath is where an admin asks what waf knows of a client, by
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
const ClientsPath = "/_sww/clients/"
// Params are what New needs.
type Params struct {
Config *config.WAF
// RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own
// endpoints.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// Metrics count what waf does. Whoever builds them adds the metrics
// read from the other parts.
Metrics *metrics.Metrics
// Limiter is the table of clients the rate limits count, with each
// client's history.
Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
// request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
}
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
type CoreRuleSetInspector interface {
Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (coreruleset.Result, []byte, error)
}
// Server is the server waf runs.
type Server struct {
*http.Server
}
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
metrics: params.Metrics,
awaitingReport: awaitingReport,
limiter: params.Limiter,
ledger: params.Ledger,
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets, banResponse: cfg.BanResponse},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{
allowed: cfg.ExclusivelyAllowedCountries,
banResponse: cfg.BanResponse,
},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
},
}
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.ownEndpoints = h.routes()
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
// besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
}
+3 -6
View File
@@ -344,8 +344,8 @@ func startProxyWithAlerts(
return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue return startRProxy(t, server.Addr, appURL, env), out, server, alertQueue
} }
// serve serves server on a port of its own until the test ends, and sets // serve serves server on a port of its own until the test ends, and sets its
// server.Addr to where it listens. // Addr to where it listens.
func serve(t *testing.T, server *wafAndParts) { func serve(t *testing.T, server *wafAndParts) {
t.Helper() t.Helper()
@@ -423,13 +423,10 @@ func startRProxy(
} }
// wafAndParts is waf's server with the Params it was built from, so that a // wafAndParts is waf's server with the Params it was built from, so that a
// test can read the parts the server uses. Addr is where it listens once serve // test can read the parts the server uses.
// has it serving.
type wafAndParts struct { type wafAndParts struct {
*waf.Server *waf.Server
waf.Params waf.Params
Addr string
} }
// banAbuseIPDBBlacklist does on w what a fetch of AbuseIPDB's blacklist with // banAbuseIPDBBlacklist does on w what a fetch of AbuseIPDB's blacklist with
-235
View File
@@ -1,235 +0,0 @@
package waf
import (
"context"
"errors"
"net/http"
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
if !ok {
return
}
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
err.Error(),
)
// This panic has Go's server close the connection without an answer,
// and log nothing.
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return question, false
}
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
h.processLog.Warn("a question is refused", "reason", reason)
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.lookupAnswer.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
// holdForReport holds a copy of what the report's steps need, by the request's
// id. A report only ever reaches that copy, never the request being handled.
func (rq *request) holdForReport() {
held := rq.decision
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
}
// deny answers a question with what the visitor is to get for r.
func (rq *request) deny(r refusal) {
rq.refused = &r
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
// request to its client's history, the anomaly counters and the metrics, with
// the status and bytes the report gives. A denied request's counting was
// finished at its decision. ctx is the report's own context.
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
h: h,
out: &responseWriter{
status: report.Status,
bytes: report.ResponseBytes,
},
requestBytes: report.RequestBytes,
}
rq.line.WAFLine = requestlog.WAFLineReport
rq.line.Status = report.Status
rq.line.RequestBytes = report.RequestBytes
rq.line.ResponseBytes = report.ResponseBytes
rq.line.DurationTotalMS = report.DurationTotalMS
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
time.Duration(
report.DurationUpstreamTotalMS*float64(time.Millisecond),
),
)
}
rq.writeLine()
}
+257 -357
View File
@@ -1,48 +1,17 @@
// Package waf is the server the waf command runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints, and
// writes a log line for each. It also bans the clients on AbuseIPDB's
// blacklist each time the reputation package fetches it.
package waf package waf
import ( import (
"cmp"
"context" "context"
"io" "errors"
"log/slog" "log/slog"
"net"
"net/http" "net/http"
"net/netip" "strings"
"time" "time"
lru "github.com/hashicorp/golang-lru/v2" "sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/anomaly"
"sneak.berlin/go/smallwebwaf/internal/bans"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/coreruleset"
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/lookup"
"sneak.berlin/go/smallwebwaf/internal/metrics"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/routewatch"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol" "sneak.berlin/go/smallwebwaf/internal/wafprotocol"
) )
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
// BansPath is where an admin lists and adds bans, and, followed by / and
// a client's address, lifts them, with SWW_WAF_ADMIN_TOKEN.
const BansPath = "/_sww/bans"
// ClientsPath is where an admin asks what waf knows of a client, by
// the client's address after it, with SWW_WAF_ADMIN_TOKEN.
const ClientsPath = "/_sww/clients/"
const ( const (
// maxAwaitingReport is the most answered requests held until their // maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped. // reports come. Past it, the request answered longest ago is dropped.
@@ -54,79 +23,6 @@ const (
wafIdleTimeout = 120 * time.Second wafIdleTimeout = 120 * time.Second
) )
// Params are what New needs.
type Params struct {
Config *config.WAF
// RequestLog receives two JSON lines per question, one at its decision and
// one when its report comes, and one per request to waf's own
// endpoints.
RequestLog io.Writer
// ProcessLog receives the process's own messages.
ProcessLog *slog.Logger
// Metrics count what waf does. Whoever builds them adds the metrics
// read from the other parts.
Metrics *metrics.Metrics
// Limiter is the table of clients the rate limits count, with each
// client's history.
Limiter *ratelimit.Limiter
// Ledger holds the bans.
Ledger *bans.Ledger
// Anomalies are the counters of the anomaly thresholds.
Anomalies *anomaly.Counters
// Lists are the lists fetched from URLs, DNSBL the DNSBL zones'
// verdicts, and AbuseIPDB AbuseIPDB's scores and the checks spent.
Lists *reputation.Lists
DNSBL *reputation.DNSBL
AbuseIPDB *reputation.AbuseIPDB
// RouteWatch asks routewatch about clients, and is nil while
// SWW_WAF_ROUTEWATCH_URL is unset.
RouteWatch *routewatch.Client
// GeoJS looks clients' AS numbers and countries up while
// SWW_WAF_LOOKUP_SOURCE is geojs. It is given AddLookup as its Answered
// and GeoJSFailed as its Failed.
GeoJS *geojs.GeoJS
// LookupFile is the lookup database they are looked up in while
// SWW_WAF_LOOKUP_SOURCE is file, and nil otherwise.
LookupFile *lookup.File
// CoreRuleSet is what inspects requests while
// SWW_WAF_CORE_RULE_SET_MODE is not off, and nil while it is.
CoreRuleSet CoreRuleSetInspector
// Now tells the time by which requests are counted for the rate
// limits, bans are made and run out, and GeoJS's answers are kept,
// normally time.Now in UTC, the time the state files give.
Now func() time.Time
// Rules are the rule files' rules, which each request is checked
// against.
Rules *rules.Files
// Alerts receive the alert for each ban waf makes or makes
// permanent, for each count over an anomaly threshold, for each request
// whose client a blocklist, a DNSBL zone or AbuseIPDB lists, for each
// request the Core Rule Set scores at or over
// SWW_WAF_CORE_RULE_SET_SCORE_THRESHOLD, and for GeoJS failing, a fetch of a list
// failing, a query to a DNSBL zone or a check with AbuseIPDB failing,
// or the day's AbuseIPDB checks used up.
Alerts *alerts.Queue
// BesideLookupTimeLimit is how long the heuristics of besideLookup have
// for a request. Zero means besideLookupTimeLimit. Tests that run the
// Core Rule Set lengthen it, since the race detector and a busy host can
// slow an inspection past besideLookupTimeLimit.
BesideLookupTimeLimit time.Duration
}
// CoreRuleSetInspector inspects a request with the Core Rule Set, as
// coreruleset.CoreRuleSet's Inspect says. Tests give waf a stand-in for it.
type CoreRuleSetInspector interface {
Inspect(
r *http.Request, client netip.Addr, body io.Reader,
) (coreruleset.Result, []byte, error)
}
// Server is the server waf runs.
type Server struct {
httpServer *http.Server
h *handler
}
// New returns the server `sww waf` runs. It answers // New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It // rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the // passes nothing to an app. rproxy applies the size and time limits to the
@@ -134,258 +30,19 @@ type Server struct {
// is far above rproxy's, and a question refused for its size would have rproxy // is far above rproxy's, and a question refused for its size would have rproxy
// let the request through. // let the request through.
func New(params Params) *Server { func New(params Params) *Server {
s := &Server{h: newHandler(params)} return &Server{
s.httpServer = &http.Server{ Server: &http.Server{
Addr: params.Config.ListenAddr, Addr: params.Config.ListenAddr,
Handler: s, Handler: newHandler(params),
ErrorLog: slog.NewLogLogger( ErrorLog: slog.NewLogLogger(
params.ProcessLog.Handler(), params.ProcessLog.Handler(),
slog.LevelWarn, slog.LevelWarn,
), ),
ReadTimeout: wafTimeout, ReadTimeout: wafTimeout,
WriteTimeout: wafTimeout, WriteTimeout: wafTimeout,
IdleTimeout: wafIdleTimeout, IdleTimeout: wafIdleTimeout,
}
return s
}
// Serve serves requests on listener until Shutdown or Close, as Go's
// http.Server.Serve does.
func (s *Server) Serve(listener net.Listener) error {
return s.httpServer.Serve(listener)
}
// Shutdown stops serving once the requests in progress have finished, or
// when ctx is done, as Go's http.Server.Shutdown does.
func (s *Server) Shutdown(ctx context.Context) error {
return s.httpServer.Shutdown(ctx)
}
// Close stops serving at once and closes every connection, as Go's
// http.Server.Close does.
func (s *Server) Close() error {
return s.httpServer.Close()
}
// ServeHTTP handles one request to waf, as handler.ServeHTTP says.
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
s.h.ServeHTTP(w, r)
}
// newHandler returns the handler with what every request shares, the parts
// in params among it. It builds the heuristics, each with the settings and
// parts it reads, and the lists they run in.
//
//nolint:funlen // a few lines for each heuristic, a list that grows with them
func newHandler(params Params) *handler {
awaitingReport, err := lru.New[string, *decision](maxAwaitingReport)
if err != nil {
panic(err) // New fails only for a size below one
}
cfg := params.Config
h := &handler{
config: cfg,
requestLog: params.RequestLog,
processLog: params.ProcessLog,
now: params.Now,
metrics: params.Metrics,
awaitingReport: awaitingReport,
limiter: params.Limiter,
ledger: params.Ledger,
geojs: params.GeoJS,
anomalies: params.Anomalies,
lookupFile: params.LookupFile,
// The country list, the headers, the tag settings that name an asn=
// or country= tag and the biased thresholds act on the answer before
// the request goes on.
waitForGeoJS: len(cfg.ExclusivelyAllowedCountries) > 0 ||
cfg.AddLookupHeaders ||
cfg.TagsNeedLookup() ||
biasedThresholdsSet(cfg),
besideLookupLimit: cmp.Or(
params.BesideLookupTimeLimit,
besideLookupTimeLimit,
),
}
sharedBanMaker := banMaker{
ledger: params.Ledger,
limiter: params.Limiter,
metrics: params.Metrics,
alerts: params.Alerts,
observe: cfg.Observe,
banResponse: cfg.BanResponse,
banScopeV4Prefix: cfg.BanScopeV4Prefix,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
hitNoter := reputationHitNoter{
metrics: params.Metrics,
alerts: params.Alerts,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.trapPaths = trapPaths{paths: cfg.TrapPaths, banMaker: sharedBanMaker}
h.ruleFiles = ruleFiles{
rules: params.Rules,
metrics: params.Metrics,
banMaker: sharedBanMaker,
}
h.coreRuleSet = coreRuleSet{
inspector: params.CoreRuleSet,
metrics: params.Metrics,
alerts: params.Alerts,
mode: cfg.CoreRuleSetMode,
exemptPaths: cfg.CoreRuleSetExemptPaths,
scoreThreshold: cfg.CoreRuleSetScoreThreshold,
observe: cfg.Observe,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
}
h.beforeLookup = []heuristic{
headerLimits{},
tagLists{lists: params.Lists},
routeWatchTags{
routeWatch: params.RouteWatch,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
},
denyNets{netblocks: cfg.DenyNets, banResponse: cfg.BanResponse},
activeBan{banMaker: sharedBanMaker},
}
h.afterLookup = []heuristic{
countries{
allowed: cfg.ExclusivelyAllowedCountries,
banResponse: cfg.BanResponse,
},
blocklists{
lists: params.Lists,
action: cfg.BlocklistAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
dnsbl{
dnsbl: params.DNSBL,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
hitNoter: hitNoter,
},
abuseIPDB{
abuseIPDB: params.AbuseIPDB,
limiter: params.Limiter,
keySet: cfg.AbuseIPDBKey != "",
minScore: cfg.AbuseIPDBMinScore,
action: cfg.DNSBLAbuseIPDBAction,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
hitNoter: hitNoter,
},
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
rateLimits{
limiter: params.Limiter,
exemptNets: cfg.RateLimitExemptNets,
exemptPaths: cfg.RateLimitExemptPaths,
banResponse: cfg.BanResponse,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
biasedThresholds: biasedThresholds{
lists: params.Lists,
tagLimitPercent: cfg.TagLimitPercent,
asnBytesLimitPercent: cfg.ASNBytesLimitPercent,
countryBytesLimitPercent: cfg.CountryBytesLimitPercent,
unknownCountryLimitPercent: cfg.UnknownCountryLimitPercent,
blocklistAction: cfg.BlocklistAction,
blocklistLimitPercent: cfg.BlocklistLimitPercent,
dnsblAbuseIPDBAction: cfg.DNSBLAbuseIPDBAction,
dnsblAbuseIPDBLimitPercent: cfg.DNSBLAbuseIPDBLimitPercent,
},
banMaker: sharedBanMaker,
}, },
} }
h.besideLookup = []heuristic{h.trapPaths, h.ruleFiles, h.coreRuleSet}
h.afterBesideLookup = []heuristic{
tagDeny{tags: cfg.TagDeny, banResponse: cfg.BanResponse},
}
h.afterAnswer = []heuristic{
refusalLimit{
limiter: params.Limiter,
perMinute: cfg.RefusalsLimitPerMinute,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.onReport = []heuristic{
byteLimits{
limiter: params.Limiter,
bytesCounted: cfg.BytesCounted,
ipv6GroupPrefix: cfg.IPv6GroupPrefix,
banMaker: sharedBanMaker,
},
}
h.ownEndpoints = h.routes()
return h
}
// handler is waf's handler. It holds what every request shares; what
// belongs to one request is in a request.
type handler struct {
config *config.WAF
requestLog io.Writer
processLog *slog.Logger
now func() time.Time
metrics *metrics.Metrics
limiter *ratelimit.Limiter
ledger *bans.Ledger
geojs *geojs.GeoJS
anomalies *anomaly.Counters
lookupFile *lookup.File
// waitForGeoJS is true when a setting needs each request's answer from
// GeoJS before the request goes on.
waitForGeoJS bool
// besideLookupLimit is how long the heuristics of besideLookup have for a
// request, as Params.BesideLookupTimeLimit says.
besideLookupLimit time.Duration
// awaitingReport holds the decision on each answered question by its id
// until its report comes, and nil for a question still being handled.
awaitingReport *lru.Cache[string, *decision]
// ownEndpoints answers every request that is not a question, as routes
// lists the endpoints.
ownEndpoints http.Handler
// The lists of heuristics, each in the order its heuristics run. A client
// in SWW_WAF_ALLOW_NETS skips them all.
// beforeLookup are the heuristics that run before the client's AS number
// and country are looked up, so that a client they refuse is not looked
// up. header_limits comes first: it only measures and counts, and a
// request it refuses causes no other work.
beforeLookup []heuristic
// afterLookup are the heuristics that run once the lookup has been made.
afterLookup []heuristic
// besideLookup are the heuristics that only read the request. Their work
// runs beside the lookup and afterLookup, in findBesideLookup, which
// changes nothing of the request. Their check then applies what that work
// found, and runs only once waf takes it.
besideLookup []heuristic
// afterBesideLookup are the heuristics that run once afterLookup and
// besideLookup have both let the request through. tag_deny runs a second
// time, on the tag rule_files and core_rule_set can add.
afterBesideLookup []heuristic
// afterAnswer are the heuristics that run on a request once waf has
// answered it, before its decision line is written. They ban the client,
// and do not change what the request got: their verdict's action is not
// used.
afterAnswer []heuristic
// onReport are the heuristics that run on an allowed request when
// rproxy's report of it comes, before its report line is written. Like
// afterAnswer, they only ban.
onReport []heuristic
// The heuristics of besideLookup, whose work findBesideLookup does.
trapPaths trapPaths
ruleFiles ruleFiles
coreRuleSet coreRuleSet
} }
// ServeHTTP handles one request to waf: a request carrying // ServeHTTP handles one request to waf: a request carrying
@@ -400,3 +57,246 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
h.ownEndpoints.ServeHTTP(w, r) h.ownEndpoints.ServeHTTP(w, r)
} }
// refusedWhateverTheAnswer reports whether the request is one that does not wait
// for GeoJS, since the checks will refuse it at now whatever GeoJS answers about
// its client: a heuristic of afterLookup that implements refusesWithoutLookup
// says it will refuse it. In observe mode, which refuses nothing, none is.
// rproxy lets a request through when waf answers too late.
func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
if rq.h.config.Observe {
return false
}
for _, each := range rq.h.afterLookup {
knows, ok := each.(refusesWithoutLookup)
if ok && knows.refusesWhateverTheLookupAnswers(rq, now) {
return true
}
}
return false
}
// answerQuestion runs the checks on the visitor's request a question
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
if !ok {
return
}
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
true,
)
// These run even when a check panics. finish runs first, so that what is
// held has the finished log line. Go's server sends the answer only once
// this function has returned, so the report cannot come before it is held.
defer func() {
rq.finish()
rq.holdForReport()
}()
rq.site = r.Host
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
if refused == nil && !underOwnPrefix {
rq.allow()
// In observe mode, what enforce mode would have refused counts here.
rq.runBanningHeuristics(r.Context(), rq.h.afterAnswer)
return
}
if refused != nil {
rq.deny(*refused)
} else {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
http.StatusNotFound,
)
}
rq.banAndCount(r.Context(), rq.h.afterAnswer)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
err.Error(),
)
// This panic has Go's server close the connection without an answer,
// and log nothing.
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
)
return question, false
}
return question, true
}
// refuseQuestion answers a question waf does not take, for reason.
func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
h.processLog.Warn("a question is refused", "reason", reason)
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.lookupAnswer.ASN
answer.Country = rq.lookupAnswer.Country
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
// holdForReport holds a copy of what the report's steps need, by the request's
// id. A report only ever reaches that copy, never the request being handled.
func (rq *request) holdForReport() {
held := rq.decision
rq.h.awaitingReport.Add(rq.line.RequestID, &held)
}
// deny answers a question with what the visitor is to get for r.
func (rq *request) deny(r refusal) {
rq.refused = &r
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
// handleReport answers POST wafprotocol.ReportPath: it finishes the request
// the report names, if it holds that request's decision, and ignores the
// report otherwise.
func (h *handler) handleReport() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
return
}
held, found := h.awaitingReport.Peek(report.RequestID)
// Remove reports true to one report only, of several that name the
// id. A question still being handled is held as nil.
if found && held != nil &&
h.awaitingReport.Remove(report.RequestID) {
h.finishWith(r.Context(), held, report)
}
w.WriteHeader(http.StatusNoContent)
}
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
// request to its client's history, the anomaly counters and the metrics, with
// the status and bytes the report gives. A denied request's counting was
// finished at its decision. ctx is the report's own context.
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
) {
rq := &request{
decision: *held,
h: h,
out: &responseWriter{
status: report.Status,
bytes: report.ResponseBytes,
},
requestBytes: report.RequestBytes,
}
rq.line.WAFLine = requestlog.WAFLineReport
rq.line.Status = report.Status
rq.line.RequestBytes = report.RequestBytes
rq.line.ResponseBytes = report.ResponseBytes
rq.line.DurationTotalMS = report.DurationTotalMS
if rq.allowed {
rq.line.DurationUpstreamTotalMS = &report.DurationUpstreamTotalMS
rq.banAndCount(ctx, rq.h.onReport)
rq.h.metrics.RequestReported(
&rq.line,
time.Duration(report.DurationTotalMS*float64(time.Millisecond)),
time.Duration(
report.DurationUpstreamTotalMS*float64(time.Millisecond),
),
)
}
rq.writeLine()
}
+8 -10
View File
@@ -703,9 +703,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t, t,
"", "",
map[string]string{metricsToken: token}, map[string]string{metricsToken: token},
func(server *wafAndParts) { func(server *wafAndParts) { server.ReadTimeout = readTimeout },
server.HTTPServer().ReadTimeout = readTimeout
},
) )
conn := dial(t, s.addr) conn := dial(t, s.addr)
@@ -1836,16 +1834,16 @@ func TestLogLevelHoldsBackNoRequestLine(t *testing.T) {
func TestServerHasTheTimeLimitsSPECGives(t *testing.T) { func TestServerHasTheTimeLimitsSPECGives(t *testing.T) {
t.Parallel() t.Parallel()
server := startWAF(t, "", nil).server.HTTPServer() s := startWAF(t, "", nil)
if server.ReadTimeout != time.Minute || if s.server.ReadTimeout != time.Minute ||
server.WriteTimeout != time.Minute || s.server.WriteTimeout != time.Minute ||
server.IdleTimeout != 2*time.Minute { s.server.IdleTimeout != 2*time.Minute {
t.Errorf( t.Errorf(
"the read, write and idle limits are %v, %v and %v, want 1m, 1m and 2m", "the read, write and idle limits are %v, %v and %v, want 1m, 1m and 2m",
server.ReadTimeout, s.server.ReadTimeout,
server.WriteTimeout, s.server.WriteTimeout,
server.IdleTimeout, s.server.IdleTimeout,
) )
} }
} }