Compare commits

..
Author SHA1 Message Date
sneak 8e3879a0c9 coreruleset: embed the apps' fixes as one directory (closes #430)
check / check (push) Waiting to run
The nine app files move to internal/coreruleset/apps/ and are embedded
once as appFixes. An app's name is its file's name; the per-app
variables go, and apps keeps only gitea's and Forgejo's cookies and
forkOf. Comments and SPEC name each of sww's changes to the Core Rule
Set by what it does instead of by number. The directives Coraza gets
are unchanged.

Model: opus-5-5
2026-11-19 16:35:33 +00:00
40 changed files with 635 additions and 1181 deletions

No files matched your search

-2
View File
@@ -902,8 +902,6 @@ stops the running process (see [`docs/state-files.md`](docs/state-files.md)).
- `internal/rproxy`: the reverse proxy in front of one app.
- `internal/waf`: `waf`'s server, which runs the checks and answers its own
endpoints.
- `internal/wafprotocol`: the question, the answer and the report that `rproxy`
and `waf` send each other.
- `internal/ratelimit`, `internal/bans`, `internal/rules`,
`internal/coreruleset`, `internal/geojs`, `internal/lookup`,
`internal/reputation`, `internal/routewatch`, `internal/anomaly` and
+35 -37
View File
@@ -73,16 +73,13 @@ never as a time.
`go.uber.org/fx`. fx builds the parts of `waf` and `rproxy`, and starts
and stops their background work.
- `internal/config`: reads the settings, the one place they are read.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and answers
`waf`'s own endpoints in `answerOwnEndpoint`. It counts bytes, refusals
and anomalies, and writes the log lines.
- `internal/waf`: `waf`'s server. It runs the checks in `check`, and holds
`waf`'s question, answer and report. It answers `waf`'s own endpoints in
`answerOwnEndpoint`. It counts bytes, refusals and anomalies, and writes
the log lines.
- `internal/rproxy`: the reverse proxy in front of one app that asks `waf`
about each request. It passes a request to the app with the standard
library's `httputil.ReverseProxy`, within the time and size limits.
- `internal/wafprotocol`: the question, the answer and the report that
`rproxy` and `waf` send each other (see "`waf`"). It writes and reads each
one, and declares every header, path and limit both sides use.
`internal/rproxy` imports it, not `internal/waf`.
- `internal/ratelimit`: the table of clients. It counts requests, bytes and
refusals, and keeps each client's history.
- `internal/bans`: the bans. It holds each netblock's bans with their notes,
@@ -820,8 +817,8 @@ The supported apps are:
- `vaultwarden`;
- `wordpress`.
Each app's fixes are in one file, such as `internal/coreruleset/gitea.conf`, and
one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
Each app's fixes are in one file named for the app, such as
`internal/coreruleset/apps/gitea.conf`. gitea's fixes are:
1. Ten parameters are not checked against the Core Rule Set's lists of system
files (930120), shell paths (932160) and command names (932260), on any path:
@@ -854,8 +851,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
3. On gitea's paths, some fields skip the rules that refused ordinary gitea
requests there. They are the fields where gitea sends a search, a file's name
or text, a commit message, a branch name or a repository's name.
`internal/coreruleset/gitea.conf` lists each rule and the request it refused.
The fields are:
`internal/coreruleset/apps/gitea.conf` lists each rule and the request it
refused. The fields are:
- `q` on gitea's searches of code, of issues and pull requests, and of
commits. A code search holds code, such as `<script>`,
`SELECT * FROM users WHERE` or `../`. The searches of issues and pull
@@ -1040,7 +1037,7 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
- The `Content-Range` header of an artifact upload from
`actions/upload-artifact@v3` (920450).
- The passwords in a body, for each rule that random passwords set off. The
numbers are in `internal/coreruleset/gitea.conf`.
numbers are in `internal/coreruleset/apps/gitea.conf`.
- `password` on these paths:
- `/user/login`, `/user/sign_up`, `/user/activate`;
- `/user/link_account_signin` and `/user/link_account_signup`;
@@ -1072,8 +1069,8 @@ one entry of `apps` in `internal/coreruleset/coreruleset.go`. gitea's fixes are:
such as `owner/python3-app` would have that push refused.
WordPress's fixes: on WordPress's paths, some fields skip the rules that refused
ordinary WordPress requests there. `internal/coreruleset/wordpress.conf` lists
each rule and the request it refused. The REST API has four addresses:
ordinary WordPress requests there. `internal/coreruleset/apps/wordpress.conf`
lists each rule and the request it refused. The REST API has four addresses:
- `/wp-json/...`
- `/index.php/wp-json/...`
@@ -1165,9 +1162,9 @@ Elsewhere WordPress's fixes do not apply. The paths are those of WordPress at
the site root.
Drupal's fixes: on Drupal's paths, some fields skip the rules that refused
ordinary Drupal requests there. `internal/coreruleset/drupal.conf` lists each
rule and the request it refused. These query fields skip the lists of system
files and command names (930120, 932120, 932250, 932260):
ordinary Drupal requests there. `internal/coreruleset/apps/drupal.conf` lists
each rule and the request it refused. These query fields skip the lists of
system files and command names (930120, 932120, 932250, 932260):
- On every page under `/admin`: `title`, `name`, `user`, `filename`, `filemime`,
`info`, `subject`, `author_name`, `search`, `q` and `destination`.
@@ -1257,10 +1254,10 @@ first path segment of two or three lowercase letters, optionally followed by a
hyphen and more letters, so `/api` counts too.
MediaWiki's fixes: on MediaWiki's paths, some fields skip the rules that refused
ordinary MediaWiki requests there. `internal/coreruleset/mediawiki.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query or the
body:
ordinary MediaWiki requests there. `internal/coreruleset/apps/mediawiki.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query or
the body:
- On MediaWiki's pages, which are `/index.php` and the paths under `/index.php/`
and `/wiki/`:
@@ -1366,9 +1363,9 @@ Elsewhere these fields keep every rule. The paths are those of MediaWiki at the
site root or under `/w/`.
Mastodon's fixes: on Mastodon's paths, some fields skip the rules that refused
ordinary Mastodon requests there. `internal/coreruleset/mastodon.conf` lists
each rule and the request it refused. These fields skip the lists of system
files and command names (930120, 932120, 932250, 932260), in the query:
ordinary Mastodon requests there. `internal/coreruleset/apps/mastodon.conf`
lists each rule and the request it refused. These fields skip the lists of
system files and command names (930120, 932120, 932250, 932260), in the query:
- On the API's search, accounts and admin accounts, under `/api/v1/` and
`/api/v2/`, the searches and names `q`, `acct`, `username`, `display_name`,
@@ -1529,7 +1526,7 @@ every rule.
Vaultwarden's fixes: on Vaultwarden's paths, some fields skip the rules that
refused ordinary Vaultwarden requests there.
`internal/coreruleset/vaultwarden.conf` lists each rule and the request it
`internal/coreruleset/apps/vaultwarden.conf` lists each rule and the request it
refused. The apps encrypt vault items, folders, collections and sends before
sending them, so those fields skip no rules. The fields below hold plain text.
@@ -1580,8 +1577,8 @@ has a parameter of the same name.
`token` in the body of `/admin`, the admin token typed on the admin page, skips
each rule that random passwords set off. The numbers are in
`internal/coreruleset/vaultwarden.conf`. It applies only when the query has no
`token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`internal/coreruleset/apps/vaultwarden.conf`. It applies only when the query has
no `token` parameter. `' OR '1'='1`, `../../etc/passwd`, `; cat /etc/passwd` and
`<img src=x onerror=x()>` sent as the token reach Vaultwarden;
`<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
@@ -1603,7 +1600,7 @@ Elsewhere these fields keep every rule.
Nextcloud's fixes: on Nextcloud's paths, some fields skip the rules that refused
ordinary Nextcloud requests there, and its WebDAV methods are allowed.
`internal/coreruleset/nextcloud.conf` lists each rule and the request it
`internal/coreruleset/apps/nextcloud.conf` lists each rule and the request it
refused.
The sync client and the Files app use WebDAV on these paths:
@@ -1613,7 +1610,8 @@ The sync client and the Files app use WebDAV on these paths:
- `/remote.php/webdav/`;
- `/public.php/webdav/`.
These methods are allowed there, besides those of the first change:
These methods are allowed there, besides `PUT`, `PATCH` and `DELETE`, which
every app gets:
- `PROPFIND` and `PROPPATCH`;
- `MKCOL`;
@@ -1757,7 +1755,7 @@ gets gitea's fixes too. Forgejo's own fixes hold only what Forgejo sends and
gitea does not.
On Forgejo's paths, some fields skip the rules that refused ordinary Forgejo
requests there. `internal/coreruleset/forgejo.conf` lists each rule and the
requests there. `internal/coreruleset/apps/forgejo.conf` lists each rule and the
request it refused.
`q` skips rules on some of Forgejo's searches. The searches are:
@@ -1803,10 +1801,10 @@ The Core Rule Set reads the request without the `flash` cookie, in which Forgejo
puts what gitea puts in `gitea_flash` (gitea's second fix).
Ghost's fixes: on Ghost's paths, some fields skip the rules that refused
ordinary Ghost requests there. `internal/coreruleset/ghost.conf` lists each rule
and the request it refused. Ghost's admin and integrations call the Admin API
under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API under
`/ghost/api/content/`, and members call `/members/api/`. The admin calls
ordinary Ghost requests there. `internal/coreruleset/apps/ghost.conf` lists each
rule and the request it refused. Ghost's admin and integrations call the Admin
API under `/ghost/api/admin/`. Themes and Ghost's scripts call the Content API
under `/ghost/api/content/`, and members call `/members/api/`. The admin calls
ActivityPub, a service of its own, under `/.ghost/activitypub/`.
Eight rules refused words that name commands and files, such as `python3`,
@@ -1944,9 +1942,9 @@ That name holds `.profile`, a file on the list of system files, so the name
skips 930120 when a profile is saved.
The passwords skip each rule that random passwords set off. The numbers are in
`internal/coreruleset/ghost.conf`. They are `password` when signing in, and the
passwords sent when creating the first staff user, accepting an invitation, and
resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`internal/coreruleset/apps/ghost.conf`. They are `password` when signing in, and
the passwords sent when creating the first staff user, accepting an invitation,
and resetting or changing a password. `' OR '1'='1`, `../../etc/passwd`,
`; cat /etc/passwd` and `<img src=x onerror=x()>` sent as a password reach
Ghost; `<script>alert(1)</script>` and `javascript:alert(1)` are still refused.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -37,9 +37,10 @@
SecRule &TX:app_nextcloud "@eq 0" \
"id:900090,phase:1,pass,nolog,skipAfter:END_NEXTCLOUD"
# The sync client and the Files app use these WebDAV methods besides those
# the first change allows, on /remote.php/dav/ and /public.php/dav/ and on the
# older /remote.php/webdav/ and /public.php/webdav/. 911100 refused each:
# The sync client and the Files app use these WebDAV methods besides PUT,
# PATCH and DELETE, which sww allows for every app, on /remote.php/dav/ and
# /public.php/dav/ and on the older /remote.php/webdav/ and /public.php/webdav/.
# 911100 refused each:
# PROPFIND, PROPPATCH: listing a folder, marking a file a favourite
# MKCOL: making a folder, and starting a chunked upload
# COPY, MOVE: copying, moving and renaming, and ending a chunked upload
File renamed without changes.
+43 -65
View File
@@ -12,11 +12,10 @@
package coreruleset
import (
_ "embed"
"embed"
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/netip"
@@ -35,7 +34,7 @@ import (
// directives are the Core Rule Set as sww runs it, with the
// paranoia level for %d, bodyDirectives for the first %s while
// SWW_WAF_CORE_RULE_SET_BODY_INSPECTION_SIZE is set, the exclusions file for
// the second, and the directives of every app in apps for the third.
// the second, and the fixes of every app in appFixes for the third.
// Each rule sww adds has an id from 900000 to 900999, the ids the
// Core Rule Set keeps for the rules that set it up, which
// SWW_WAF_CORE_RULE_SET_DISABLED_RULES and the exclusions file refuse, so that no
@@ -54,13 +53,12 @@ Include @crs-setup.conf.example
SecAction "id:900000,phase:1,pass,nolog,\
setvar:tx.blocking_paranoia_level=%d"
# The first change: PUT, PATCH and DELETE are allowed besides GET, HEAD,
# POST and OPTIONS.
# PUT, PATCH and DELETE are allowed besides GET, HEAD, POST and OPTIONS.
SecAction "id:900200,phase:1,pass,nolog,\
setvar:'tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE'"
# The second: Expect and Content-Encoding are taken off the Core Rule Set's
# list of the headers it refuses. Content-Encoding goes back on it for a
# Expect and Content-Encoding are taken off the Core Rule Set's list of the
# headers it refuses. Content-Encoding goes back on it for a
# body the Core Rule Set reads (900260 in bodyDirectives).
SecAction "id:900250,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=/proxy/ /lock-token/ /content-range/ \
@@ -86,18 +84,17 @@ SecRule ARGUMENTS_LIMIT_REACHED "@eq 1" "id:900300,phase:2,pass,\
# The apps' fixes come before the rules for the same reason.
%s
# The fifth: only the rules for requests are loaded, and no response is
# inspected.
# Only the rules for requests are loaded, and no response is inspected.
Include @owasp_crs/REQUEST-*.conf
# The third: redirect_uri is not checked for a URL naming an IP address or
# redirect_uri is not checked for a URL naming an IP address or
# localhost. Coraza matches a parameter name here without regard to case.
# ARGS holds the fields of a form data or multipart body Coraza reads as
# well as the query parameters, so a field of this name is left out too.
SecRuleUpdateTargetById 931100 "!ARGS:redirect_uri"
SecRuleUpdateTargetById 934110 "!ARGS:redirect_uri"
# The fourth: Referer is not checked for a Unix command without arguments
# Referer is not checked for a Unix command without arguments
# or for Java starting a process.
SecRuleUpdateTargetById 932340 "!REQUEST_HEADERS:Referer"
SecRuleUpdateTargetById 944110 "!REQUEST_HEADERS:Referer"
@@ -122,9 +119,9 @@ SecRule REQUEST_HEADERS:Content-Type \
"@rx ^(?:application|text)/(?:[a-z0-9.-]+[+])?json" \
"id:900420,phase:1,pass,nolog,t:none,t:lowercase,ctl:requestBodyProcessor=JSON"
# The rest of the second change: Content-Encoding is refused again on a
# body of a kind the Core Rule Set reads, since a compressed body cannot be
# inspected.
# Content-Encoding goes back on the list of the headers the Core Rule Set
# refuses (900250 in directives) for a body of a kind it reads, since a
# compressed body cannot be inspected.
SecRule REQBODY_PROCESSOR "@rx ^(?:URLENCODED|MULTIPART|JSON|XML)$" \
"id:900260,phase:1,pass,nolog,\
setvar:'tx.restricted_headers_basic=%%{tx.restricted_headers_basic} \
@@ -145,15 +142,19 @@ SecRule MULTIPART_STRICT_ERROR "!@eq 0" "id:900450,phase:2,pass,\
severity:'CRITICAL',setvar:'tx.inbound_anomaly_score_pl1=+5'"
`
// app is an app whose ordinary requests the Core Rule Set would refuse,
// with the fixes that let them pass.
// appFixes holds the fixes of each app whose ordinary requests the Core
// Rule Set would refuse, in a file named for the app by the name
// SWW_WAF_APPS gives it, such as apps/gitea.conf. The fixes are Coraza
// exclusions, loaded before the Core Rule Set's rules. Their first rule
// skips to the marker they end with, unless Inspect sets the transaction
// variable named for the app, such as app_gitea. The skip reaches only
// phase 1, so a rule of a later phase must check that variable itself.
//
//go:embed apps/*.conf
var appFixes embed.FS
// app is what an app's file in appFixes cannot say about it.
type app struct {
// directives are Coraza exclusions, loaded before the Core Rule Set's
// rules. Their first rule skips to the marker they end with, unless
// Inspect sets the transaction variable named for the app, such as
// app_gitea. The skip reaches only phase 1, so a rule of a later phase
// must check that variable itself.
directives string
// cookies are the cookies the Core Rule Set reads a request without.
cookies []string
// forkOf names the app this one is a fork of, "" for none. A hostname
@@ -161,60 +162,34 @@ type app struct {
forkOf string
}
//go:embed gitea.conf
var giteaDirectives string
//go:embed wordpress.conf
var wordpressDirectives string
//go:embed drupal.conf
var drupalDirectives string
//go:embed mediawiki.conf
var mediawikiDirectives string
//go:embed mastodon.conf
var mastodonDirectives string
//go:embed vaultwarden.conf
var vaultwardenDirectives string
//go:embed nextcloud.conf
var nextcloudDirectives string
//go:embed forgejo.conf
var forgejoDirectives string
//go:embed ghost.conf
var ghostDirectives string
// apps are the apps whose fixes sww has, by the name SWW_WAF_APPS gives each.
// flash is Forgejo's name for gitea_flash.
// apps are the apps that need more than their file in appFixes, by the
// name SWW_WAF_APPS gives each. flash is Forgejo's name for gitea_flash.
//
//nolint:gochecknoglobals // a constant cannot be a map
var apps = map[string]app{
"drupal": {directives: drupalDirectives},
"forgejo": {
directives: forgejoDirectives,
cookies: []string{"flash"},
forkOf: "gitea",
cookies: []string{"flash"},
forkOf: "gitea",
},
"ghost": {directives: ghostDirectives},
"gitea": {
directives: giteaDirectives,
cookies: []string{"gitea_flash", "redirect_to"},
cookies: []string{"gitea_flash", "redirect_to"},
},
"mastodon": {directives: mastodonDirectives},
"mediawiki": {directives: mediawikiDirectives},
"nextcloud": {directives: nextcloudDirectives},
"vaultwarden": {directives: vaultwardenDirectives},
"wordpress": {directives: wordpressDirectives},
}
// AppNames returns the names of the apps whose fixes sww has, in
// alphabetical order.
func AppNames() []string {
return slices.Sorted(maps.Keys(apps))
// An embedded directory always reads.
files, _ := appFixes.ReadDir("apps")
names := make([]string, len(files))
for i, file := range files {
names[i] = strings.TrimSuffix(file.Name(), ".conf")
}
slices.Sort(names)
return names
}
// Params are what New needs.
@@ -373,8 +348,11 @@ func New(params Params) (*CoreRuleSet, error) {
}
var fixes strings.Builder
for _, name := range AppNames() {
fixes.WriteString(apps[name].directives + "\n")
// An embedded file always reads.
file, _ := appFixes.ReadFile("apps/" + name + ".conf")
fixes.WriteString(string(file) + "\n")
}
text := fmt.Sprintf(directives, params.ParanoiaLevel, body, exclusions,
+10
View File
@@ -7,6 +7,16 @@ import (
"strings"
)
// The headers rproxy sets on the requests it passes to the app. Go writes every
// header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn, and Del removes a visitor's own whatever
// their case; header names are not case-sensitive.
const (
requestIDHeader = "X-Request-ID"
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// peerAddress is the address of the request's TCP peer, normally traefik, without
// any IPv6 zone, which waf refuses.
func peerAddress(r *http.Request) netip.Addr {
+3 -3
View File
@@ -20,7 +20,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// post is the start of a POST whose headers end with what follows it.
@@ -492,7 +492,7 @@ func TestASettingOfOffSetsNoLimit(t *testing.T) {
strings.NewReader("a body"))),
)
if got := get(t, addr, wafprotocol.HealthPath); got.status != http.StatusOK {
if got := get(t, addr, waf.HealthPath); got.status != http.StatusOK {
t.Errorf("answered %+v, want rproxy's own 200", got)
}
})
@@ -1471,7 +1471,7 @@ func TestAVisitorThatStopsALittleShortGetsAnAnswerRProxyGivesAtOnce(
{"a deny", post, http.StatusForbidden},
{
"the health check",
"GET " + wafprotocol.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
"GET " + waf.HealthPath + " HTTP/1.1\r\nHost: site.example\r\n",
http.StatusOK,
},
} {
+13 -13
View File
@@ -12,11 +12,11 @@ import (
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// questionBodyMaxBytes is the most of a body a question carries.
const questionBodyMaxBytes = wafprotocol.QuestionBodyMaxBytes
const questionBodyMaxBytes = waf.QuestionBodyMaxBytes
// chunked is the encoding of a body whose length the visitor did not announce.
const chunked = "chunked"
@@ -42,9 +42,9 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
req.Header.Set(forwardedForHeader, "198.51.100.1")
req.Header.Set(forwardedProtoHeader, "gopher")
req.Header.Set(requestIDHeader, "the-visitor's")
req.Header.Set(wafprotocol.ClientHeader, "198.51.100.2")
req.Header.Set(wafprotocol.PeerHeader, "198.51.100.3")
req.Header.Set(wafprotocol.ContentLengthHeader, "12345")
req.Header.Set(waf.ClientHeader, "198.51.100.2")
req.Header.Set(waf.PeerHeader, "198.51.100.3")
req.Header.Set(waf.ContentLengthHeader, "12345")
// Go's client then sends no User-Agent, so that one in the question would be
// Go's own.
req.Header["User-Agent"] = nil
@@ -73,8 +73,8 @@ func TestQuestionIsTheVisitorsRequestWithRProxysHeaders(t *testing.T) {
want.Set(forwardedForHeader, "198.51.100.1")
want.Set(forwardedProtoHeader, "http")
want.Set(requestIDHeader, asked.header.Get(requestIDHeader))
want.Set(wafprotocol.ClientHeader, localhost)
want.Set(wafprotocol.PeerHeader, localhost)
want.Set(waf.ClientHeader, localhost)
want.Set(waf.PeerHeader, localhost)
if !maps.EqualFunc(asked.header, want, slices.Equal) {
t.Errorf(
@@ -124,8 +124,8 @@ func TestQuestionNamesTheVisitorATrustedProxySpeaksFor(t *testing.T) {
wantGreeting(t, visit(t, req))
asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != localhost ||
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != localhost ||
asked.header.Get(forwardedForHeader) != visitor ||
asked.header.Get(forwardedProtoHeader) != tc.proto {
t.Errorf("the question's headers are %v", asked.header)
@@ -191,7 +191,7 @@ func TestQuestionNamesTheClientAsTheForwardedAddressesSay(t *testing.T) {
req.Header[forwardedForHeader] = tc.header
wantGreeting(t, visit(t, req))
if got := standIn.question(t).header.Get(wafprotocol.ClientHeader); got !=
if got := standIn.question(t).header.Get(waf.ClientHeader); got !=
tc.client {
t.Errorf("the question names %s, want %s", got, tc.client)
}
@@ -282,7 +282,7 @@ func wantQuestionBody(
)
}
values := asked.header.Values(wafprotocol.ContentLengthHeader)
values := asked.header.Values(waf.ContentLengthHeader)
if announced {
if !slices.Equal(values, []string{strconv.Itoa(size)}) ||
@@ -586,8 +586,8 @@ func TestAQuestionNamesNoZone(t *testing.T) {
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
asked := standIn.question(t)
if asked.header.Get(wafprotocol.ClientHeader) != tc.client ||
asked.header.Get(wafprotocol.PeerHeader) != tc.peer {
if asked.header.Get(waf.ClientHeader) != tc.client ||
asked.header.Get(waf.PeerHeader) != tc.peer {
t.Errorf("the question's headers are %v", asked.header)
}
+13 -11
View File
@@ -18,13 +18,17 @@ import (
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// flushAfterEachWrite has ReverseProxy pass on each part of the app's answer as
// soon as it arrives.
const flushAfterEachWrite time.Duration = -1
// reportStatusMax is the highest status waf takes in a report. An app can
// send a higher one, up to 999, and the visitor gets that as it is.
const reportStatusMax = 599
// plainText is the Content-Type of the answers rproxy gives itself.
const plainText = "text/plain; charset=utf-8"
@@ -150,7 +154,7 @@ func (rq *request) readFirstPartOfBody() bool {
_ = rq.rc.SetReadDeadline(rq.requestDeadline())
held, err := io.ReadAll(
io.LimitReader(rq.body, wafprotocol.QuestionBodyMaxBytes),
io.LimitReader(rq.body, waf.QuestionBodyMaxBytes),
)
// From here on, the request timer ends a read of the rest of the body, so that
// the answer can name the side rproxy was waiting on.
@@ -224,17 +228,16 @@ func (rq *request) rewrite(pr *httputil.ProxyRequest) {
setPathAsSent(pr.Out.URL, pr.In)
setForwardedHeaders(pr.In, pr.Out, rq.peer, rq.peerTrusted)
pr.Out.Header.Set(wafprotocol.RequestIDHeader, rq.id)
// Del removes a visitor's own, whatever their case.
pr.Out.Header.Del(wafprotocol.ASNHeader)
pr.Out.Header.Del(wafprotocol.CountryHeader)
pr.Out.Header.Set(requestIDHeader, rq.id)
pr.Out.Header.Del(asnHeader)
pr.Out.Header.Del(countryHeader)
if rq.asn != "" {
pr.Out.Header.Set(wafprotocol.ASNHeader, rq.asn)
pr.Out.Header.Set(asnHeader, rq.asn)
}
if rq.country != "" {
pr.Out.Header.Set(wafprotocol.CountryHeader, rq.country)
pr.Out.Header.Set(countryHeader, rq.country)
}
}
@@ -467,10 +470,9 @@ func (rq *request) finish(ctx context.Context) {
return
}
// An app can send a status up to 999, and the visitor gets it as it is.
report := &wafprotocol.Report{
report := &waf.Report{
RequestID: rq.id,
Status: min(rq.out.status, wafprotocol.ReportStatusMax),
Status: min(rq.out.status, reportStatusMax),
RequestBytes: requestBytes,
ResponseBytes: responseBytes,
DurationTotalMS: requestlog.Milliseconds(end.Sub(rq.start)),
+12 -10
View File
@@ -24,7 +24,7 @@ import (
"time"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// How rproxy keeps connections to the app and to waf open between
@@ -47,6 +47,10 @@ const (
idleTimeout = 120 * time.Second
)
// ownEndpointPrefix starts the path of every request for sww itself, which
// never reaches the app.
const ownEndpointPrefix = "/_sww/"
// allowedMethods lists the methods rproxy passes on, as an Allow header gives
// them. It answers any other method 405 itself.
// The WebDAV ones are for Nextcloud.
@@ -88,9 +92,7 @@ func New(params Params) *Server {
transport: newTransport(),
wafTransport: newWAFTransport(),
reportURL: (&url.URL{
Scheme: backend.Scheme,
Host: backend.Host,
Path: wafprotocol.ReportPath,
Scheme: backend.Scheme, Host: backend.Host, Path: waf.ReportPath,
}).String(),
}
@@ -174,7 +176,7 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer rq.readRestOfBody()
defer rq.finish(r.Context())
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
if r.Method == http.MethodGet && r.URL.Path == waf.HealthPath {
rq.answerHealth()
return
@@ -207,9 +209,9 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
rq.noteAnswer(answer)
switch rq.disposition {
case wafprotocol.DispositionAllow:
case waf.DispositionAllow:
rq.passToApp(r.Context())
case wafprotocol.DispositionClose:
case waf.DispositionClose:
// This panic has Go's server close the connection without an answer,
// and log nothing; the deferred finish still runs.
panic(http.ErrAbortHandler)
@@ -219,8 +221,8 @@ func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
// passToApp passes the request on to the app, unless a limit or its path keeps
// it from the app. waf answers a path under wafprotocol.OwnEndpointPrefix
// itself, so such a request gets here only when
// it from the app. waf answers a path under ownEndpointPrefix itself, so
// such a request gets here only when
// SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION allowed it.
// ctx is the request's own context.
func (rq *request) passToApp(ctx context.Context) {
@@ -229,7 +231,7 @@ func (rq *request) passToApp(ctx context.Context) {
switch {
case rq.refused.Load() != 0 || (maxBytes > 0 && rq.in.ContentLength > maxBytes):
rq.answer(http.StatusRequestEntityTooLarge)
case strings.HasPrefix(rq.in.URL.Path, wafprotocol.OwnEndpointPrefix):
case strings.HasPrefix(rq.in.URL.Path, ownEndpointPrefix):
rq.answer(http.StatusNotFound)
default:
rq.forward(ctx)
+18 -18
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/rproxy"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -56,7 +56,7 @@ const (
// The headers the tests set and look at.
const (
dispositionHeader = wafprotocol.DispositionHeader
dispositionHeader = waf.DispositionHeader
forwardedForHeader = "X-Forwarded-For"
forwardedProtoHeader = "X-Forwarded-Proto"
requestIDHeader = "X-Request-ID"
@@ -72,9 +72,9 @@ const (
// What the stand-in for waf answers.
const (
dispositionAllow = wafprotocol.DispositionAllow
dispositionDeny = wafprotocol.DispositionDeny
dispositionClose = wafprotocol.DispositionClose
dispositionAllow = waf.DispositionAllow
dispositionDeny = waf.DispositionDeny
dispositionClose = waf.DispositionClose
// deniedBody is the body of a deny.
deniedBody = "refused by waf"
)
@@ -175,7 +175,7 @@ type WAF struct {
*httptest.Server
questions chan question
reports chan wafprotocol.Report
reports chan waf.Report
}
// startWAF starts a stand-in for waf.
@@ -184,11 +184,11 @@ func startWAF(t *testing.T, answer http.HandlerFunc) *WAF {
s := &WAF{
questions: make(chan question, 10),
reports: make(chan wafprotocol.Report, 10),
reports: make(chan waf.Report, 10),
}
s.Server = httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) == "" {
if r.Header.Get(waf.ClientHeader) == "" {
s.takeReport(t, w, r)
return
@@ -217,11 +217,11 @@ func (s *WAF) takeReport(
) {
t.Helper()
var report wafprotocol.Report
var report waf.Report
err := json.NewDecoder(r.Body).Decode(&report)
if err != nil || r.Method != http.MethodPost ||
r.URL.Path != wafprotocol.ReportPath {
r.URL.Path != waf.ReportPath {
t.Errorf("%s %s is no report: %v", r.Method, r.URL.Path, err)
}
@@ -242,7 +242,7 @@ func (s *WAF) wantNoReport(t *testing.T) {
}
// report waits for a report.
func (s *WAF) report(t *testing.T) wafprotocol.Report {
func (s *WAF) report(t *testing.T) waf.Report {
t.Helper()
select {
@@ -251,7 +251,7 @@ func (s *WAF) report(t *testing.T) wafprotocol.Report {
case <-time.After(waitLimit):
t.Fatal("no report")
return wafprotocol.Report{}
return waf.Report{}
}
}
@@ -279,7 +279,7 @@ func startWAFAnsweringReports(
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
answer(w, r)
return
@@ -811,7 +811,7 @@ func TestAnAnswerRProxyHoldsWholeIsSentBeforeItsLogLine(t *testing.T) {
}{
{"a deny", "GET /" + host, deniedBody, http.StatusForbidden},
{"rproxy's own answer", "GET x:admin" + host, "Bad Request\n", http.StatusBadRequest},
{"the health check", "GET " + wafprotocol.HealthPath + host, "ok\n", http.StatusOK},
{"the health check", "GET " + waf.HealthPath + host, "ok\n", http.StatusOK},
} {
t.Run(tc.name, func(t *testing.T) {
t.Parallel()
@@ -1576,19 +1576,19 @@ func TestHealthCheckIsAnsweredWhileWAFIsDown(t *testing.T) {
app := startApp(t, greet)
addr, out := startRProxy(t, nowhere, app.URL, nil)
got := get(t, addr, wafprotocol.HealthPath)
got := get(t, addr, waf.HealthPath)
if got.status != http.StatusOK || got.body != "ok\n" {
t.Errorf("answered %+v, want ok", got)
}
line := out.requestLine(t)
if line["path"] != wafprotocol.HealthPath || line["disposition"] != nil {
if line["path"] != waf.HealthPath || line["disposition"] != nil {
t.Errorf("line %v, want one with no disposition", line)
}
// With waf up too, it is not asked.
addr, _ = startRProxy(t, standIn.URL, app.URL, nil)
get(t, addr, wafprotocol.HealthPath)
get(t, addr, waf.HealthPath)
if len(standIn.questions) != 0 {
t.Error("waf was asked about the health check")
@@ -1749,7 +1749,7 @@ func TestWaitForReportsWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
allow(w, r)
return
+82 -34
View File
@@ -1,7 +1,9 @@
package rproxy
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
@@ -10,8 +12,7 @@ import (
"strings"
"time"
"sneak.berlin/go/smallwebwaf/internal/rules"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// answerBodyMaxBytes is the most of the body of waf's answer that is read.
@@ -40,12 +41,15 @@ type wafAnswer struct {
// notes nothing, and no report is due for it: waf holds no decision to report
// on.
func (rq *request) noteAnswer(answer *wafAnswer) {
read, valid := wafprotocol.ReadAnswer(answer.header)
if valid {
rq.disposition = read.Disposition
disposition := answer.header.Get(waf.DispositionHeader)
switch disposition {
case waf.DispositionAllow, waf.DispositionDeny,
waf.DispositionClose:
rq.disposition = disposition
rq.answered = true
rq.asn = read.ASN
rq.country = read.Country
rq.asn = answer.header.Get(asnHeader)
rq.country = answer.header.Get(countryHeader)
}
}
@@ -76,7 +80,7 @@ func (rq *request) handleNoUsableAnswer(ctx context.Context, err error) {
rq.disposition = rq.h.config.WAFBackendNoAnswerDisposition
rq.wafNoAnswer = true
if rq.disposition == wafprotocol.DispositionAllow {
if rq.disposition == waf.DispositionAllow {
rq.passToApp(ctx)
} else {
rq.answer(http.StatusServiceUnavailable)
@@ -116,21 +120,7 @@ func (rq *request) ask(ctx context.Context) (*wafAnswer, error) {
ctx, cancel := context.WithTimeout(ctx, rq.h.config.WAFBackendTimeout)
defer cancel()
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := wafprotocol.NewQuestion(ctx, wafprotocol.QuestionParams{
Visitor: rq.in,
PathAsSent: rules.Path(rq.in),
Body: held,
WAFBackend: rq.h.config.WAFBackend,
Client: rq.client,
Peer: rq.peer,
RequestID: rq.id,
Scheme: scheme(rq.in, rq.peerTrusted),
})
question, err := rq.newQuestion(ctx)
if err != nil {
return nil, err
}
@@ -187,13 +177,69 @@ func canBeAsked(r *http.Request) bool {
len(r.Header.Values("User-Agent")) <= 1
}
// newQuestion returns the question about the request, as "The question" in
// SPEC.md gives it: the visitor's request as the visitor sent it, with the part
// of the body it holds and rproxy's headers.
func (rq *request) newQuestion(ctx context.Context) (*http.Request, error) {
in := rq.in
var held []byte
if rq.body != nil {
held = rq.body.held
}
question, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx,
in.Method,
rq.h.config.WAFBackend.String(),
bytes.NewReader(held),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
setPathAsSent(&target, in)
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of what is held. This line makes
// a body whose length the visitor did not announce go chunked, as "The question"
// in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(waf.ClientHeader, rq.client.String())
question.Header.Set(waf.PeerHeader, rq.peer.String())
question.Header.Del(waf.ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(waf.ContentLengthHeader, announced)
}
question.Header.Set(requestIDHeader, rq.id)
question.Header.Set("X-Forwarded-Proto", scheme(in, rq.peerTrusted))
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// answerWith gives the visitor waf's answer, less its disposition.
func (rq *request) answerWith(answer *wafAnswer) {
rq.prepareOwnAnswer(rq.requestDeadline())
header := rq.out.Header()
maps.Copy(header, answer.header)
header.Del(wafprotocol.DispositionHeader)
header.Del(waf.DispositionHeader)
// Go's server adds a Content-Type it guesses from the body to a response whose
// headers have no Content-Type key. The key with no value sends none.
@@ -207,10 +253,7 @@ func (rq *request) answerWith(answer *wafAnswer) {
// sendReport sends waf report on a goroutine of its own, so that the
// visitor does not wait for it. The report goes on after ctx, the request's own
// context, has ended.
func (h *handler) sendReport(
ctx context.Context,
report *wafprotocol.Report,
) {
func (h *handler) sendReport(ctx context.Context, report *waf.Report) {
h.reportsMu.Lock()
defer h.reportsMu.Unlock()
@@ -228,18 +271,23 @@ func (h *handler) sendReport(
// postReport posts report to waf, and waits for its answer for up to
// SWW_RPROXY_WAF_BACKEND_TIMEOUT_MS.
func (h *handler) postReport(
ctx context.Context,
report *wafprotocol.Report,
) error {
func (h *handler) postReport(ctx context.Context, report *waf.Report) error {
body, err := json.Marshal(report)
if err != nil {
return fmt.Errorf("encode the report: %w", err)
}
ctx, cancel := context.WithTimeout(ctx, h.config.WAFBackendTimeout)
defer cancel()
req, err := wafprotocol.NewReportRequest(ctx, h.reportURL, report)
req, err := http.NewRequestWithContext( //nolint:gosec // a setting's URL
ctx, http.MethodPost, h.reportURL, bytes.NewReader(body))
if err != nil {
return err
return fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
res, err := h.wafTransport.RoundTrip(req)
if err != nil {
return fmt.Errorf("send the report: %w", err)
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/spf13/viper"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// healthCheckTimeout bounds the whole health check.
@@ -116,7 +116,7 @@ func askHealthEndpoint(ctx context.Context, listenAddr string) error {
health := "http://" + net.JoinHostPort(
"127.0.0.1",
port,
) + wafprotocol.HealthPath
) + waf.HealthPath
req, err := http.NewRequestWithContext(
ctx,
+10 -13
View File
@@ -21,7 +21,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -164,7 +164,7 @@ func TestABanThroughOneSiteRefusesTheClientOnTheOther(t *testing.T) {
// One request's three lines carry one id, rproxy's with waf's answer.
line := firstOut.line(t, "path", bannedProbe)
if line["disposition"] != wafprotocol.DispositionDeny ||
if line["disposition"] != waf.DispositionDeny ||
line["client_ip"] != placed {
t.Errorf("rproxy's line is %v", line)
}
@@ -249,7 +249,7 @@ func TestTheSiteAnswersWhileWAFIsStoppedAndStartedAgain(t *testing.T) {
wantStatus(t, site+"/after-stop", placed, http.StatusOK)
line := out.line(t, "path", "/after-stop")
if line["disposition"] != wafprotocol.DispositionAllow ||
if line["disposition"] != waf.DispositionAllow ||
line["waf_backend_no_answer"] != true {
t.Errorf("rproxy's line is %v", line)
}
@@ -437,7 +437,7 @@ func TestAChunkedBodyStaysABodyAtWAF(t *testing.T) {
const smuggledID = "smuggled-request"
smuggled := "GET " + bannedProbe + " HTTP/1.1\r\nHost: site.example\r\n" +
wafprotocol.ClientHeader + ": " + otherClient + "\r\n" +
waf.ClientHeader + ": " + otherClient + "\r\n" +
"X-Request-ID: " + smuggledID + "\r\n\r\n"
status := sendRaw(
@@ -605,11 +605,10 @@ func TestRProxyAsksAWAFThatSpeaksHTTP2OverHTTP1(t *testing.T) {
protocols := make(chan string, 10)
backend := httptest.NewUnstartedServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
if r.Header.Get(waf.ClientHeader) != "" {
protocols <- r.Proto
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionDeny)
w.Header().Set(waf.DispositionHeader, waf.DispositionDeny)
w.WriteHeader(http.StatusForbidden)
}
}))
@@ -691,9 +690,8 @@ func TestRProxyStoppingWaitsForAReportBeingSent(t *testing.T) {
release := make(chan struct{})
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(wafprotocol.ClientHeader) != "" {
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
if r.Header.Get(waf.ClientHeader) != "" {
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
return
}
@@ -804,8 +802,7 @@ func TestRProxyStoppingCutsOffARequestAfterFiveSeconds(t *testing.T) {
backend := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, _ *http.Request) {
w.Header().
Set(wafprotocol.DispositionHeader, wafprotocol.DispositionAllow)
w.Header().Set(waf.DispositionHeader, waf.DispositionAllow)
}))
t.Cleanup(backend.Close)
@@ -938,7 +935,7 @@ func TestMainRunsRProxy(t *testing.T) {
addr, stop := startRProxyProcess(t, wafBackend+"=http://127.0.0.1:9")
req, err := http.NewRequestWithContext(t.Context(), http.MethodGet,
"http://"+addr+wafprotocol.HealthPath, http.NoBody)
"http://"+addr+waf.HealthPath, http.NoBody)
if err != nil {
t.Fatalf("new request: %v", err)
}
+2 -2
View File
@@ -23,7 +23,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/config/configtest"
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -1811,7 +1811,7 @@ func statusFrom(t *testing.T, url, from string) int {
}
req.Header.Set("X-Forwarded-For", from)
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set(waf.ClientHeader, from)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
+8 -14
View File
@@ -20,7 +20,7 @@ import (
_ "time/tzdata"
"sneak.berlin/go/smallwebwaf/internal/sww"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// commandForMain, while set in the environment, has this test binary run Main
@@ -106,7 +106,7 @@ func TestMainRunsWAF(t *testing.T) {
starting := out.lineBeforeExit(t, "starting", exited)
addr, _ := starting["address"].(string)
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The files are written as sww stops.
err = cmd.Process.Signal(syscall.SIGTERM)
@@ -217,7 +217,7 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
t.Fatalf("new request: %v", err)
}
req.Header.Set(wafprotocol.ClientHeader, placed)
req.Header.Set(waf.ClientHeader, placed)
req.Header.Set("X-Request-ID", rand.Text())
transport := &http.Transport{}
@@ -232,11 +232,11 @@ func ask(t *testing.T, addr, path string, status int, disposition string) {
_ = res.Body.Close()
if res.StatusCode != status ||
res.Header.Get(wafprotocol.DispositionHeader) != disposition {
res.Header.Get(waf.DispositionHeader) != disposition {
t.Errorf(
"answered %d with disposition %q, want %d with %q",
res.StatusCode,
res.Header.Get(wafprotocol.DispositionHeader),
res.Header.Get(waf.DispositionHeader),
status,
disposition,
)
@@ -287,18 +287,12 @@ func TestWAFKeepsItsBansAcrossARestartAndNeedsNoApp(t *testing.T) {
}
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusOK, wafprotocol.DispositionAllow)
ask(t, addr, "/", http.StatusOK, waf.DispositionAllow)
// The default rule file bans a probe for this.
ask(
t,
addr,
"/.env",
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
ask(t, addr, "/.env", http.StatusForbidden, waf.DispositionDeny)
})
runWAF(t, env, func(addr string) {
ask(t, addr, "/", http.StatusForbidden, wafprotocol.DispositionDeny)
ask(t, addr, "/", http.StatusForbidden, waf.DispositionDeny)
})
}
+23 -61
View File
@@ -26,7 +26,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/reputation"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -85,7 +84,7 @@ func TestSixteenThousandCookiesAreRefusedBeforeTheCoreRuleSetReadsThem(
got := do(t, req)
wantDisposition(t, got, tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
line := s.out.requestLine(t)
wantLine(t, line, tooLarge, requestlog.ActionRequestHeaderTooLarge)
@@ -165,14 +164,9 @@ func TestARequestLineOverTheLimitIsRefusedWith414BeforeAnyOtherCheck(
line := s.out.requestLine(t)
if tc.status == http.StatusOK {
wantDisposition(
t,
got,
tc.status,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, tc.status, waf.DispositionAllow)
} else {
wantDisposition(t, got, tc.status, wafprotocol.DispositionDeny)
wantDisposition(t, got, tc.status, waf.DispositionDeny)
}
wantLine(t, line, tc.status, tc.action)
@@ -225,14 +219,9 @@ func TestAHeaderLineOverTheLimitIsRefusedWith431(t *testing.T) {
got := do(t, req)
if tc.refused {
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
} else {
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
})
}
@@ -255,7 +244,10 @@ func TestRequestLineRefusalNamesItsHeuristicLengthAndLimit(t *testing.T) {
func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
t.Parallel()
repeated := slices.Repeat([]string{"1"}, headerFieldLimit)
repeated := make([]string, headerFieldLimit)
for i := range repeated {
repeated[i] = "1"
}
for _, tc := range []struct {
name string
@@ -302,12 +294,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
line := s.out.requestLine(t)
if tc.refused {
wantDisposition(
t,
got,
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, tooLarge, waf.DispositionDeny)
wantLine(
t,
line,
@@ -318,12 +305,7 @@ func TestEachHeaderLimitRefusesARequestOverIt(t *testing.T) {
return
}
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
if line.DurationCoreRuleSetMS == nil {
@@ -350,12 +332,7 @@ func TestHeaderLimitsHoldWhateverTheCoreRuleSetSettingsSay(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
})
}
}
@@ -371,12 +348,7 @@ func TestClientInTheAllowNetsSkipsTheHeaderLimits(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
@@ -392,7 +364,7 @@ func TestObserveModeAllowsARequestOverTheHeaderLimitsUnread(t *testing.T) {
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.WouldAction != requestlog.ActionRequestHeaderTooLarge ||
@@ -414,14 +386,14 @@ func TestTheRateLimitsDoNotCountRequestsOverTheHeaderLimits(t *testing.T) {
for range 2 {
req := s.question(siteA, client, "/")
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, req), tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
}
wantDisposition(
t,
s.ask(client, "/"),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
}
@@ -494,12 +466,7 @@ func TestTheHeaderLimitsComeBeforeEveryCheckThatRefuses(t *testing.T) {
req := s.question(siteA, client, tc.path)
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(
t,
do(t, req),
tooLarge,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), tooLarge, waf.DispositionDeny)
wantLine(
t,
s.out.requestLine(t),
@@ -536,16 +503,11 @@ func TestARequestOverTheHeaderLimitsRunsNoOtherCheck(t *testing.T) {
over := s.question(siteA, client, "/")
over.Header.Set(cookieHeader, cookies(cookieLimit+1))
wantDisposition(t, do(t, over), tooLarge, wafprotocol.DispositionDeny)
wantDisposition(t, do(t, over), tooLarge, waf.DispositionDeny)
within := s.question(siteA, client, "/")
within.Header.Set(cookieHeader, cookies(cookieLimit))
wantDisposition(
t,
do(t, within),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, within), http.StatusOK, waf.DispositionAllow)
lines := s.out.requestLines(t, 2)
@@ -602,7 +564,7 @@ func TestNotWaitingForGeoJSNeverLowersAClientsLimit(t *testing.T) {
t,
s.ask(fromDE, "/"),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -639,7 +601,7 @@ func TestAClientGeoJSPlacedInNoCountryCountsAsUnknownWithoutWaiting(
t,
s.ask(client, "/"),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
line := s.out.requestLine(t)
@@ -668,7 +630,7 @@ func TestObserveModeAsksGeoJSNothingForARequestOverTheHeaderLimits(
req.Header.Set(cookieHeader, cookies(cookieLimit+1))
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
if len(asked()) != 0 || got.header.Get("X-Client-Asn") != "" {
t.Errorf(
@@ -710,7 +672,7 @@ func askDirectly(
http.NoBody,
)
req.RemoteAddr = net.JoinHostPort(rproxyPeer, "1234")
req.Header.Set(wafprotocol.ClientHeader, client)
req.Header.Set(waf.ClientHeader, client)
req.Header.Set(requestIDHeader, "the-id")
change(req)
+1 -2
View File
@@ -19,7 +19,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/lookup/lookuptest"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The biased thresholds.
@@ -590,7 +589,7 @@ func serveFromDE(
t.Helper()
req := httptest.NewRequestWithContext(t.Context(), method, "/", body)
req.Header.Set(wafprotocol.ClientHeader, fromDE)
req.Header.Set(waf.ClientHeader, fromDE)
req.Header.Set(requestIDHeader, "the-id")
server.Handler.ServeHTTP(httptest.NewRecorder(), req)
+4 -2
View File
@@ -7,7 +7,6 @@ import (
"slices"
"sneak.berlin/go/smallwebwaf/internal/config"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// peerAddress is the address of the request's TCP peer.
@@ -20,11 +19,14 @@ func peerAddress(r *http.Request) netip.Addr {
return addrPort.Addr().Unmap()
}
// requestIDHeader carries the request's id, from rproxy.
const requestIDHeader = "X-Request-ID"
// requestID is the request's id: the one a question gives, or for any other
// request a new random one.
func requestID(r *http.Request, isQuestion bool) string {
if isQuestion {
return r.Header.Get(wafprotocol.RequestIDHeader)
return r.Header.Get(requestIDHeader)
}
return rand.Text()
+3 -3
View File
@@ -12,7 +12,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/alerts"
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
// The Core Rule Set's settings the tests set, besides SWW_WAF_CORE_RULE_SET_MODE, and
@@ -395,9 +395,9 @@ func TestGiteasFixesApplyOnlyToAHostnameThatServesGitea(t *testing.T) {
})
wantDisposition(t, do(t, s.question(siteA, client, fileName)),
http.StatusOK, wafprotocol.DispositionAllow)
http.StatusOK, waf.DispositionAllow)
wantDisposition(t, do(t, s.question(siteB, client, fileName)),
http.StatusForbidden, wafprotocol.DispositionDeny)
http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2)
wantLine(t, lines[0], http.StatusOK, requestlog.ActionForward)
+3 -8
View File
@@ -5,7 +5,7 @@ import (
"testing"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
@@ -25,13 +25,8 @@ func TestHealthEndpointCountsTowardNoLimit(t *testing.T) {
)
for range healthChecks {
got := s.ownEndpointRequest(
"",
http.MethodGet,
wafprotocol.HealthPath,
"",
http.StatusOK,
)
got := s.ownEndpointRequest("", http.MethodGet, waf.HealthPath, "",
http.StatusOK)
if string(got.body) != "ok\n" ||
got.header.Get("Content-Type") != contentType {
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
func TestHistoryKeepsEachRequestOfTheClient(t *testing.T) {
@@ -119,7 +118,7 @@ func TestOwnEndpointsAreNotInTheHistory(t *testing.T) {
s.ownEndpointRequest(
"",
http.MethodGet,
wafprotocol.HealthPath,
waf.HealthPath,
"",
http.StatusOK,
)
+22
View File
@@ -3,6 +3,7 @@ package waf
import (
"context"
"log/slog"
"net/http"
"net/netip"
"strings"
"time"
@@ -15,6 +16,15 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
)
// The headers in which an answer of allow gives rproxy the client's AS number
// and country, for the app, while SWW_WAF_ADD_LOOKUP_HEADERS is true. Go writes
// every header name in this form, as it sends it and as it receives it, so
// X-Client-ASN arrives as X-Client-Asn; header names are not case-sensitive.
const (
asnHeader = "X-Client-Asn"
countryHeader = "X-Client-Country"
)
// lookupResult is what the lookup gave for a request: whether its client was
// looked up, the answer, and whether the request skipped the wait for GeoJS.
type lookupResult struct {
@@ -156,6 +166,18 @@ func GeoJSFailed(
m.GeoJSFailures.Inc()
}
// setLookupHeaders sets the headers that give the client's AS number and
// country, leaving out one that is unknown.
func setLookupHeaders(header http.Header, asn, country string) {
if asn != "" {
header.Set(asnHeader, asn)
}
if country != "" {
header.Set(countryHeader, country)
}
}
// canBePlaced reports whether a lookup can place addr: private, loopback
// and link-local addresses have no AS number or country.
func canBePlaced(addr netip.Addr) bool {
+1 -2
View File
@@ -16,7 +16,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/geojs"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -67,7 +66,7 @@ func TestMetricsNeedTheTokenAndOtherPathsAreNotFound(t *testing.T) {
{http.MethodPost, waf.MetricsPath, bearer, http.StatusNotFound},
{http.MethodGet, waf.MetricsPath + "/", bearer, http.StatusNotFound},
{http.MethodGet, "/_sww/bans", bearer, http.StatusNotFound},
{http.MethodPost, wafprotocol.HealthPath, "", http.StatusNotFound},
{http.MethodPost, waf.HealthPath, "", http.StatusNotFound},
} {
got := s.ownEndpointRequest(
tc.authorization,
+8
View File
@@ -29,6 +29,14 @@ import (
"sneak.berlin/go/smallwebwaf/internal/rules"
)
// ownEndpointPrefix starts the path of each of waf's own endpoints. The app
// never sees a path under it.
const ownEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// MetricsPath is where the metrics are, for a request that carries
// SWW_WAF_METRICS_TOKEN.
const MetricsPath = "/_sww/metrics"
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
"sneak.berlin/go/smallwebwaf/internal/waf"
)
const (
@@ -248,7 +248,7 @@ func TestTimingsAreInOrder(t *testing.T) {
wantStatus(t, do(t, req), http.StatusForbidden)
refused := out.endedLines(t, 2)[1]
wantStatus(t, get(t, server.Addr, wafprotocol.HealthPath), http.StatusOK)
wantStatus(t, get(t, server.Addr, waf.HealthPath), http.StatusOK)
health := out.endedLines(t, 3)[2]
// A request passed to the app has every timing; one refused, none of
+226 -54
View File
@@ -1,21 +1,75 @@
package waf
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"net/netip"
"strconv"
"strings"
"time"
"unicode/utf8"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// The headers of rproxy's question and of waf's answer, as
// "waf" in SPEC.md gives them.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
const (
// maxAwaitingReport is the most answered requests held until their
// reports come. Past it, the request answered longest ago is dropped.
maxAwaitingReport = 10000
// maxRequestIDBytes is the longest id a question may give.
maxRequestIDBytes = 128
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxHTTPStatus is the highest status a report may give.
maxHTTPStatus = 599
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
// wafTimeout bounds how long a request to waf takes to
// arrive, and its answer to leave, and wafIdleTimeout how long a
// kept-open connection waits for its next request.
@@ -23,6 +77,26 @@ const (
wafIdleTimeout = 120 * time.Second
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// New returns the server `sww waf` runs. It answers
// rproxy's questions, takes rproxy's reports and answers its own endpoints. It
// passes nothing to an app. rproxy applies the size and time limits to the
@@ -58,10 +132,10 @@ func New(params Params) *Server {
}
// ServeHTTP handles one request to waf: a request carrying
// wafprotocol.ClientHeader is rproxy's question, and any other is for waf
// ClientHeader is rproxy's question, and any other is for waf
// itself.
func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if len(r.Header.Values(wafprotocol.ClientHeader)) > 0 {
if len(r.Header.Values(ClientHeader)) > 0 {
h.answerQuestion(w, r)
return
@@ -88,16 +162,19 @@ func (rq *request) refusedWhateverTheAnswer(now time.Time) bool {
// carries, and answers rproxy what to do with it. A denied request's counting
// is finished here, and an allowed one's by its report.
func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
question, ok := h.readQuestion(w, r)
client, body, ok := h.readQuestion(w, r)
if !ok {
return
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
rq := h.startRequest(
w,
question.Visitor,
question.Client,
question.Peer,
asTheVisitorSentIt(r, body),
client.Unmap(),
peer.Unmap(),
true,
)
// The last deferred runs first, so that what is held has the finished log
@@ -110,10 +187,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
rq.line.WAFLine = requestlog.WAFLineDecision
refused := rq.check(r.Context())
rq.checked = time.Now()
underOwnPrefix := strings.HasPrefix(
r.URL.Path,
wafprotocol.OwnEndpointPrefix,
)
underOwnPrefix := strings.HasPrefix(r.URL.Path, ownEndpointPrefix)
if refused == nil && !underOwnPrefix {
// In observe mode, what enforce mode would have refused counts here.
@@ -139,10 +213,7 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
// The app never sees this prefix. waf's own endpoints are at its
// own address, so this answers as a path with no endpoint does.
rq.line.Action = requestlog.ActionOwnEndpoint
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
rq.out.Header().Set(DispositionHeader, DispositionDeny)
http.Error(
rq.out,
http.StatusText(http.StatusNotFound),
@@ -150,19 +221,45 @@ func (h *handler) answerQuestion(w http.ResponseWriter, r *http.Request) {
)
}
// readQuestion reads a question, and reports whether waf takes it. It answers
// a question it does not take with 400 and no disposition. rproxy passes that
// 400 to the visitor, and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does
// not apply. The id of a question waf takes is held from here on, without a
// decision yet, so that another question with that id is refused while this
// one is being handled.
// readQuestion returns the visitor's address and the body a question carries,
// and whether waf takes the question. It answers a question it does not
// take with 400 and no disposition. rproxy passes that 400 to the visitor,
// and SWW_RPROXY_WAF_BACKEND_NO_ANSWER_DISPOSITION does not apply. The id of
// a question waf takes is held from here on, without a decision yet, so
// that another question with that id is refused while this one is being
// handled.
//
// A question whose body stops arriving gets no answer, and is not looked at.
// The body is read before any check, since Go's server does not send an answer
// until it has read the body. A question whose body stops arriving gets no
// answer, and is not looked at.
func (h *handler) readQuestion(
w http.ResponseWriter, r *http.Request,
) (wafprotocol.Question, bool) {
question, err := wafprotocol.ReadQuestion(w, r)
if errors.Is(err, wafprotocol.ErrBodyStoppedArriving) {
) (netip.Addr, []byte, bool) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
h.refuseQuestion(w, ClientHeader+" is not an address without a zone")
return netip.Addr{}, nil, false
}
id := r.Header.Get(requestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
h.refuseQuestion(w, requestIDHeader+" is not a valid id")
return netip.Addr{}, nil, false
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
h.refuseQuestion(w, "the body is over 64 KiB")
return netip.Addr{}, nil, false
}
if err != nil {
h.processLog.Warn(
"a question's body stopped arriving",
"error",
@@ -173,23 +270,17 @@ func (h *handler) readQuestion(
panic(http.ErrAbortHandler)
}
if err != nil {
h.refuseQuestion(w, err.Error())
return question, false
}
held, _ := h.awaitingReport.ContainsOrAdd(question.RequestID, nil)
held, _ := h.awaitingReport.ContainsOrAdd(id, nil)
if held {
h.refuseQuestion(
w,
wafprotocol.RequestIDHeader+" is the id of a request waf holds",
requestIDHeader+" is the id of a request waf holds",
)
return question, false
return netip.Addr{}, nil, false
}
return question, true
return client, body, true
}
// refuseQuestion answers a question waf does not take, for reason.
@@ -198,18 +289,48 @@ func (h *handler) refuseQuestion(w http.ResponseWriter, reason string) {
http.Error(w, reason, http.StatusBadRequest)
}
// allow answers a question with wafprotocol.DispositionAllow and, while
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
// allow answers a question with DispositionAllow and, while
// SWW_WAF_ADD_LOOKUP_HEADERS is true, the client's AS number and country.
func (rq *request) allow() {
rq.allowed = true
answer := wafprotocol.Answer{Disposition: wafprotocol.DispositionAllow}
header := rq.out.Header()
header.Set(DispositionHeader, DispositionAllow)
if rq.h.config.AddLookupHeaders {
answer.ASN = rq.line.ASN
answer.Country = rq.line.Country
setLookupHeaders(header, rq.line.ASN, rq.line.Country)
}
wafprotocol.SetAnswerHeaders(rq.out.Header(), answer)
rq.out.WriteHeader(http.StatusOK)
}
@@ -227,19 +348,13 @@ func (rq *request) deny(r refusal) {
if r.status == 0 {
// SWW_WAF_BAN_RESPONSE is close. Written past rq.out, since the log line's
// status stays 0: the visitor is sent nothing.
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionClose},
)
rq.out.Header().Set(DispositionHeader, DispositionClose)
rq.out.ResponseWriter.WriteHeader(http.StatusOK)
return
}
wafprotocol.SetAnswerHeaders(
rq.out.Header(),
wafprotocol.Answer{Disposition: wafprotocol.DispositionDeny},
)
rq.out.Header().Set(DispositionHeader, DispositionDeny)
http.Error(rq.out, http.StatusText(r.status), r.status)
}
@@ -247,7 +362,7 @@ func (rq *request) deny(r refusal) {
// runs on it: the address is on the private network, where the peers are
// rproxy and the admin's tools, not visitors.
func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPost && r.URL.Path == wafprotocol.ReportPath {
if r.Method == http.MethodPost && r.URL.Path == ReportPath {
h.takeReport(w, r)
return
@@ -258,7 +373,7 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq := h.startRequest(w, r, peer, peer, false)
defer rq.finish()
if r.Method == http.MethodGet && r.URL.Path == wafprotocol.HealthPath {
if r.Method == http.MethodGet && r.URL.Path == HealthPath {
rq.answerHealth()
return
@@ -267,11 +382,11 @@ func (h *handler) answerOwnEndpoint(w http.ResponseWriter, r *http.Request) {
rq.answerAdmin()
}
// takeReport answers POST wafprotocol.ReportPath: it finishes the request the
// report names, if it holds that request's decision, and ignores the report
// takeReport answers POST ReportPath: it finishes the request the report
// names, if it holds that request's decision, and ignores the report
// otherwise.
func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
report, err := wafprotocol.ReadReport(w, r)
report, err := readReport(w, r)
if err != nil {
h.processLog.Warn("a report is refused", "error", err.Error())
http.Error(w, "the body is not a report", http.StatusBadRequest)
@@ -289,6 +404,63 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// readReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func readReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > maxHTTPStatus):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
// finishWith finishes a request with what its report says, and writes the
// request's last log line, which holds its decision's fields and the report's.
// For an allowed request it first runs the heuristics of onReport, and adds the
@@ -298,7 +470,7 @@ func (h *handler) takeReport(w http.ResponseWriter, r *http.Request) {
func (h *handler) finishWith(
ctx context.Context,
held *decision,
report wafprotocol.Report,
report Report,
) {
rq := &request{
decision: *held,
+86 -195
View File
@@ -22,7 +22,6 @@ import (
"sneak.berlin/go/smallwebwaf/internal/ratelimit"
"sneak.berlin/go/smallwebwaf/internal/requestlog"
"sneak.berlin/go/smallwebwaf/internal/waf"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
const (
@@ -107,8 +106,8 @@ func (s *WAF) questionWith(
req := newRequest(s.t, method, s.addr, path, body)
req.Host = host
req.Header.Set(wafprotocol.ClientHeader, from)
req.Header.Set(wafprotocol.PeerHeader, rproxyPeer)
req.Header.Set(waf.ClientHeader, from)
req.Header.Set(waf.PeerHeader, rproxyPeer)
req.Header.Set(requestIDHeader, rand.Text())
return req
@@ -135,18 +134,13 @@ func (s *WAF) allowed(from string) string {
s.t.Helper()
req := s.question(siteA, from, "/")
wantDisposition(
s.t,
do(s.t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(s.t, do(s.t, req), http.StatusOK, waf.DispositionAllow)
return req.Header.Get(requestIDHeader)
}
// report sends rproxy's report and returns the answer.
func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) report(report waf.Report) answer {
s.t.Helper()
body, err := json.Marshal(report)
@@ -161,11 +155,8 @@ func (s *WAF) report(report wafprotocol.Report) answer {
func (s *WAF) reportBody(body string) answer {
s.t.Helper()
return do(
s.t,
newRequest(s.t, http.MethodPost, s.addr, wafprotocol.ReportPath,
strings.NewReader(body)),
)
return do(s.t, newRequest(s.t, http.MethodPost, s.addr, waf.ReportPath,
strings.NewReader(body)))
}
// passedOn asks about a request from the visitor at from on siteA, and reports
@@ -173,7 +164,7 @@ func (s *WAF) reportBody(body string) answer {
func (s *WAF) passedOn(from string) {
s.t.Helper()
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: s.allowed(from),
Status: http.StatusOK,
RequestBytes: bodyBytes,
@@ -219,7 +210,7 @@ func wantDisposition(
t.Helper()
wantStatus(t, got, status)
if answered := got.header.Get(wafprotocol.DispositionHeader); answered !=
if answered := got.header.Get(waf.DispositionHeader); answered !=
disposition {
t.Errorf(
"answered with disposition %q, want %q",
@@ -281,10 +272,10 @@ func TestQuestionAllowedIsAnsweredAndLogged(t *testing.T) {
req := s.question(siteA, client, "/page?q=1&token=visible")
req.Header.Set(requestIDHeader, "the-id")
req.Header.Set("X-Forwarded-Proto", "https")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
req.Header.Set(waf.ContentLengthHeader, "1234")
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
wantBody(t, got, "")
if id := got.header.Get(requestIDHeader); id != "" {
@@ -317,12 +308,7 @@ func TestIPv6VisitorIsCountedInItsGroup(t *testing.T) {
s.allowed("2001:db8::1")
got := s.ask("2001:db8::2", "/")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
@@ -331,14 +317,9 @@ func TestIPv4VisitorInIPv6FormIsTheIPv4Client(t *testing.T) {
s := startWAF(t, "", map[string]string{denyNets: client})
req := s.question(siteA, "::ffff:"+client, "/")
req.Header.Set(wafprotocol.PeerHeader, "::ffff:"+rproxyPeer)
req.Header.Set(waf.PeerHeader, "::ffff:"+rproxyPeer)
wantDisposition(
t,
do(t, req),
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, do(t, req), http.StatusForbidden, waf.DispositionDeny)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer {
@@ -358,13 +339,13 @@ func TestMalformedQuestionGets400AndAWarningAndNoLine(t *testing.T) {
change func(*http.Request)
}{
{"client not an address", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "not an address")
req.Header.Set(waf.ClientHeader, "not an address")
}},
{"client empty", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "")
req.Header.Set(waf.ClientHeader, "")
}},
{"client address with a zone", func(req *http.Request) {
req.Header.Set(wafprotocol.ClientHeader, "2001:db8::1%eth0")
req.Header.Set(waf.ClientHeader, "2001:db8::1%eth0")
}},
{"id absent", func(req *http.Request) {
req.Header.Del(requestIDHeader)
@@ -409,12 +390,7 @@ func TestLongestValidIDIsTaken(t *testing.T) {
req := s.question(siteA, client, "/")
req.Header.Set(requestIDHeader, strings.Repeat("x", 128))
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
@@ -423,18 +399,16 @@ func TestQuestionWithABodyOver64KiBGets400(t *testing.T) {
s := startWAF(t, "", nil)
over := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes+1),
strings.Repeat("x", waf.QuestionBodyMaxBytes+1),
)
got := do(t, s.questionWith(http.MethodPost, siteA, client, "/", over))
wantDisposition(t, got, http.StatusBadRequest, "")
wantProcessMessages(t, s, "a question is refused", 1)
wantNoRequestLines(t, s)
whole := strings.NewReader(
strings.Repeat("x", wafprotocol.QuestionBodyMaxBytes),
)
whole := strings.NewReader(strings.Repeat("x", waf.QuestionBodyMaxBytes))
got = do(t, s.questionWith(http.MethodPost, siteA, client, "/", whole))
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
@@ -450,12 +424,12 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantProcessMessages(t, s, "a question is refused", 1)
// A denied request's id is held as well.
denied := s.question(siteA, otherClient, wafprotocol.HealthPath)
denied := s.question(siteA, otherClient, waf.HealthPath)
wantDisposition(
t,
do(t, denied),
http.StatusNotFound,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
again = s.question(siteA, client, "/")
@@ -463,9 +437,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
wantDisposition(t, do(t, again), http.StatusBadRequest, "")
// The report ends the holding, and the refused questions changed nothing.
got := s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got := s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -474,12 +446,7 @@ func TestQuestionWithAHeldIDIsRefused(t *testing.T) {
again = s.question(siteA, client, "/")
again.Header.Set(requestIDHeader, id)
wantDisposition(
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, again), http.StatusOK, waf.DispositionAllow)
}
func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
@@ -514,7 +481,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
// A report for a question not yet answered is ignored, and that question's
// id is taken.
got := s.report(wafprotocol.Report{RequestID: id, ResponseBytes: 1000})
got := s.report(waf.Report{RequestID: id, ResponseBytes: 1000})
wantStatus(t, got, http.StatusNoContent)
// This would wait on the same lookup were the id not taken.
@@ -529,7 +496,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
select {
case got := <-answered:
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
case <-time.After(waitLimit):
t.Fatal("the question got no answer")
}
@@ -548,9 +515,7 @@ func TestReportNeverReachesAQuestionStillBeingHandled(t *testing.T) {
}
// The true report comes after, and is the one that counts.
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if history := s.history(); history.ResponseBytes != answerBytes {
@@ -568,17 +533,17 @@ func TestPeerAbsentOrUnreadableIsAnsweredAndLoggedInvalid(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Del(wafprotocol.PeerHeader)
req.Header.Del(waf.PeerHeader)
if peer != "" {
req.Header.Set(wafprotocol.PeerHeader, peer)
req.Header.Set(waf.PeerHeader, peer)
}
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
if line := s.out.requestLine(t); line.PeerIP != "invalid IP" {
@@ -606,12 +571,12 @@ func TestContentLengthUnreadableOrNegativeCountsAsAbsent(t *testing.T) {
})
req := s.question(siteA, client, "/")
req.Header.Set(wafprotocol.ContentLengthHeader, announced)
req.Header.Set(waf.ContentLengthHeader, announced)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
if line := s.out.requestLine(t); line.ContentLength != 0 {
@@ -630,18 +595,13 @@ func TestHeaderSentTwiceUsesItsFirstValue(t *testing.T) {
s := startWAF(t, "", nil)
req := s.question(siteA, client, "/")
req.Header.Add(wafprotocol.ClientHeader, otherClient)
req.Header.Add(wafprotocol.PeerHeader, "172.18.0.3")
req.Header.Add(waf.ClientHeader, otherClient)
req.Header.Add(waf.PeerHeader, "172.18.0.3")
req.Header.Set(requestIDHeader, "first")
req.Header.Add(requestIDHeader, "second")
req.Header.Set(wafprotocol.ContentLengthHeader, "1234")
req.Header.Add(wafprotocol.ContentLengthHeader, "5678")
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
req.Header.Set(waf.ContentLengthHeader, "1234")
req.Header.Add(waf.ContentLengthHeader, "5678")
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.ClientIP != client || line.PeerIP != rproxyPeer ||
@@ -670,7 +630,7 @@ func sendQuestionWithPartOfItsBody(t *testing.T, conn net.Conn) {
_, err := fmt.Fprintf(conn,
"POST / HTTP/1.1\r\nHost: %s\r\n%s: %s\r\n%s: an-id\r\n"+
"Content-Length: 100\r\n\r\npart",
siteA, wafprotocol.ClientHeader, client, requestIDHeader)
siteA, waf.ClientHeader, client, requestIDHeader)
if err != nil {
t.Fatalf("send the question: %v", err)
}
@@ -739,7 +699,7 @@ func TestQuestionWhoseBodyStopsArrivingGetsNoAnswer(t *testing.T) {
t,
do(t, again),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
})
}
@@ -757,7 +717,7 @@ func TestQuestionDeniedGetsTheBanResponse(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, status, wafprotocol.DispositionDeny)
wantDisposition(t, got, status, waf.DispositionDeny)
wantBody(t, got, http.StatusText(status)+"\n")
wantLine(
t,
@@ -777,7 +737,7 @@ func TestBanResponseCloseIsAnAnswerOfItsOwn(t *testing.T) {
})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionClose)
wantDisposition(t, got, http.StatusOK, waf.DispositionClose)
wantBody(t, got, "")
// The visitor is sent nothing.
wantLine(t, s.out.requestLine(t), 0, requestlog.ActionClientDenied)
@@ -801,22 +761,12 @@ func TestBlockRuleIsDeniedWith403AndABanRuleWithTheBanResponse(
})
got := s.ask(client, "/blocked")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
wantLine(t, s.out.requestLine(t), http.StatusForbidden,
requestlog.ActionBlockedByBlockRule)
got = s.ask(client, "/.env")
wantDisposition(
t,
got,
http.StatusTooManyRequests,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusTooManyRequests, waf.DispositionDeny)
wantLine(t, s.out.requestLines(t, 2)[1], http.StatusTooManyRequests,
requestlog.ActionTemporarilyBanned)
}
@@ -831,15 +781,10 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
})
for _, path := range []string{
wafprotocol.HealthPath, waf.MetricsPath, wafprotocol.ReportPath,
waf.HealthPath, waf.MetricsPath, waf.ReportPath,
} {
got := s.ask(client, path)
wantDisposition(
t,
got,
http.StatusNotFound,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusNotFound, waf.DispositionDeny)
}
wantLine(
@@ -850,12 +795,7 @@ func TestQuestionUnderTheOwnPrefixIsDeniedWith404AfterTheChecks(
)
got := s.ask(otherClient, waf.MetricsPath)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
@@ -864,7 +804,7 @@ func TestObserveModeAllowsAndLogsWhatEnforceModeWouldHaveDone(t *testing.T) {
s := startWAF(t, "", map[string]string{mode: observe, denyNets: client})
got := s.ask(client, "/")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
wantLine(t, line, http.StatusOK, requestlog.ActionForward)
@@ -882,7 +822,7 @@ func TestObserveModeRefusalAnsweredAllowCountsForTheRefusalLimit(
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
lines := s.out.requestLines(t, 2)
@@ -906,12 +846,7 @@ func TestDeniedQuestionCountsForTheRefusalLimit(t *testing.T) {
for range 2 {
got := s.ask(client, "/blocked")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
lines := s.out.requestLines(t, 2)
@@ -947,12 +882,7 @@ func TestAllowCarriesTheLookupHeadersWhileTheSettingIsSet(t *testing.T) {
})
got := s.ask(fromDE, "/")
wantDisposition(
t,
got,
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
asn, country := got.header.Get(
"X-Client-Asn",
@@ -986,7 +916,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
// 100 bytes are over the limit of 99.
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -1010,12 +940,7 @@ func TestReportCountsTowardAByteLimitWhichThenBans(t *testing.T) {
}
got = s.ask(client, "/")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
@@ -1033,12 +958,12 @@ func TestReportCountsAtThePercentageItsQuestionWasGiven(t *testing.T) {
t,
s.ask(fromKP, "/"),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
s.passedOn(fromDE)
wantDisposition(t, s.ask(fromDE, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny)
waf.DispositionDeny)
}
func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
@@ -1064,7 +989,7 @@ func TestReportAddsTheRequestToItsClientsHistoryAndTheAnomalyCounters(
)
}
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
})
@@ -1096,7 +1021,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
t,
do(t, denied),
http.StatusForbidden,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
counters := s.server.Anomalies.Snapshot()
@@ -1114,7 +1039,7 @@ func TestDeniedRequestIsFinishedAtItsDecision(t *testing.T) {
wantMetric(t, s.metrics(), series, 1)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: denied.Header.Get(
requestIDHeader,
), Status: http.StatusForbidden,
@@ -1157,7 +1082,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
1,
) // the scrape itself
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusBadGateway,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 5, DurationUpstreamTotalMS: 4,
@@ -1175,7 +1100,7 @@ func TestMetricsCountAnAllowedRequestFromItsReport(t *testing.T) {
// check needs a server that has not been scraped yet.
unscraped := startWAF(t, "", map[string]string{metricsToken: token})
got = unscraped.report(wafprotocol.Report{
got = unscraped.report(waf.Report{
RequestID: unscraped.allowed(client), Status: http.StatusBadGateway,
DurationTotalMS: 5,
})
@@ -1193,7 +1118,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: "no-such-request", ResponseBytes: 1000,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1212,9 +1137,7 @@ func TestReportForARequestNotHeldIsIgnored(t *testing.T) {
// A second report of the same request does not count it twice.
id := s.allowed(client)
for range 2 {
got = s.report(
wafprotocol.Report{RequestID: id, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: id, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
}
@@ -1424,7 +1347,7 @@ func TestBytesAtTheirMostEachWayBanRatherThanOverflow(t *testing.T) {
s := startWAF(t, "", map[string]string{bytesLimitPerMinute: byteLimit})
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: s.allowed(client), Status: http.StatusOK,
RequestBytes: reportedBytesLimit, ResponseBytes: reportedBytesLimit,
})
@@ -1518,14 +1441,9 @@ func TestReportWritesTheLastLineWithTheDecisionsAndTheReportsFields(
req.Header.Set("X-Forwarded-Proto", "https")
id := req.Header.Get(requestIDHeader)
wantDisposition(
t,
do(t, req),
http.StatusOK,
wafprotocol.DispositionAllow,
)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: http.StatusCreated,
RequestBytes: bodyBytes, ResponseBytes: answerBytes,
DurationTotalMS: 12.5, DurationUpstreamTotalMS: 10.25,
@@ -1572,7 +1490,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
"close", map[string]string{denyNets: client, banResponse: banResponseClose}, "/",
0, requestlog.ActionClientDenied,
},
{"own prefix", nil, wafprotocol.HealthPath, http.StatusNotFound,
{"own prefix", nil, waf.HealthPath, http.StatusNotFound,
requestlog.ActionOwnEndpoint},
} {
t.Run(tc.name, func(t *testing.T) {
@@ -1584,7 +1502,7 @@ func TestEveryDeniedRequestsReportGetsItsLastLine(t *testing.T) {
id := req.Header.Get(requestIDHeader)
do(t, req)
got := s.report(wafprotocol.Report{
got := s.report(waf.Report{
RequestID: id, Status: tc.status, ResponseBytes: 8, DurationTotalMS: 3,
})
wantStatus(t, got, http.StatusNoContent)
@@ -1624,13 +1542,13 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
t,
readAnswer(res),
http.StatusOK,
wafprotocol.DispositionAllow,
waf.DispositionAllow,
)
}
// The table is full again with next and those, without oldest.
got := s.report(
wafprotocol.Report{RequestID: oldest, ResponseBytes: answerBytes},
waf.Report{RequestID: oldest, ResponseBytes: answerBytes},
)
wantStatus(t, got, http.StatusNoContent)
@@ -1641,9 +1559,7 @@ func TestOldestAnsweredRequestIsDroppedPastTheBound(t *testing.T) {
)
}
got = s.report(
wafprotocol.Report{RequestID: next, ResponseBytes: answerBytes},
)
got = s.report(waf.Report{RequestID: next, ResponseBytes: answerBytes})
wantStatus(t, got, http.StatusNoContent)
if counted := s.history().Requests; counted != 1 {
@@ -1664,20 +1580,10 @@ func TestBanMadeThroughOneHostRefusesTheClientOnAnother(t *testing.T) {
)
got := do(t, s.question(siteA, client, "/.env"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
got = do(t, s.question(siteB, client, "/"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
lines := s.out.requestLines(t, 2)
if lines[0].Host != siteA || lines[1].Host != siteB {
@@ -1715,12 +1621,7 @@ func TestRequestsOnEverySiteCountTowardOneSetOfLimits(t *testing.T) {
s.allowed(client)
got := do(t, s.question(siteB, client, "/"))
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
}
func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
@@ -1735,13 +1636,13 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
path, authorization string
want int
}{
{wafprotocol.HealthPath, "", http.StatusOK},
{waf.HealthPath, "", http.StatusOK},
{waf.MetricsPath, bearer, http.StatusOK},
{waf.MetricsPath, "", http.StatusUnauthorized},
{waf.BansPath, adminBearer, http.StatusOK},
{waf.BansPath, "", http.StatusUnauthorized},
{waf.ClientsPath + client, adminBearer, http.StatusOK},
{wafprotocol.ReportPath, "", http.StatusNotFound},
{waf.ReportPath, "", http.StatusNotFound},
{"/_sww/other", "", http.StatusNotFound},
{"/", "", http.StatusNotFound},
} {
@@ -1758,10 +1659,10 @@ func TestOwnEndpointsAnswerOnlyWithoutTheClientHeader(t *testing.T) {
t,
do(t, req),
http.StatusNotFound,
wafprotocol.DispositionDeny,
waf.DispositionDeny,
)
} else {
wantDisposition(t, do(t, req), http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, do(t, req), http.StatusOK, waf.DispositionAllow)
}
}
}
@@ -1771,11 +1672,11 @@ func TestOwnEndpointsLinesDoNotSayWhichOfTheTwoTheyAre(t *testing.T) {
s := startWAF(t, "", nil)
wantStatus(t, get(t, s.addr, wafprotocol.HealthPath), http.StatusOK)
wantStatus(t, get(t, s.addr, waf.HealthPath), http.StatusOK)
line := s.out.requestLine(t)
if _, there := line.fields["waf_line"]; there ||
line.Path != wafprotocol.HealthPath {
line.Path != waf.HealthPath {
t.Errorf("the health check's log line is %+v", line.fields)
}
}
@@ -1787,7 +1688,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader("a body"))
req.Header.Set(wafprotocol.ContentLengthHeader, "1073741824")
req.Header.Set(waf.ContentLengthHeader, "1073741824")
req.Header.Set(forwardedFor, otherClient)
// 40,000 bytes of headers, over rproxy's 32 KiB, in lines within the
@@ -1797,7 +1698,7 @@ func TestALargeBodyAnnouncedAndLongHeadersAreNoReasonToRefuse(t *testing.T) {
}
got := do(t, req)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
line := s.out.requestLine(t)
if line.ClientIP != client || line.ContentLength != 1<<30 {
@@ -1875,7 +1776,7 @@ carried header:Content-Length block ^6$
strings.NewReader("a body"))
if tc.announced != "" {
req.Header.Set(wafprotocol.ContentLengthHeader, tc.announced)
req.Header.Set(waf.ContentLengthHeader, tc.announced)
}
wantStatus(t, do(t, req), tc.want)
@@ -1893,24 +1794,19 @@ func TestCoreRuleSetReadsTheBodyAQuestionCarries(t *testing.T) {
req := s.questionWith(http.MethodPost, siteA, client, "/",
strings.NewReader(body))
req.Header.Set("Content-Type", contentType)
req.Header.Set(wafprotocol.ContentLengthHeader, "100000")
req.Header.Set(waf.ContentLengthHeader, "100000")
return do(t, req)
}
got := ask(formData, "id=1'%20OR%20'1'='1")
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
// The first 8 KiB of a longer JSON body cannot be parsed, which is no
// attack, so the Core Rule Set does not read it.
cut := `{"text":"` + strings.Repeat("x", 8<<10)
got = ask("application/json", cut)
wantDisposition(t, got, http.StatusOK, wafprotocol.DispositionAllow)
wantDisposition(t, got, http.StatusOK, waf.DispositionAllow)
}
func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
@@ -1919,12 +1815,7 @@ func TestCoreRuleSetAlertNamesTheSite(t *testing.T) {
s := startWAF(t, "", map[string]string{coreRuleSetMode: block})
got := s.ask(client, sqlInjection)
wantDisposition(
t,
got,
http.StatusForbidden,
wafprotocol.DispositionDeny,
)
wantDisposition(t, got, http.StatusForbidden, waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 ||
@@ -1945,7 +1836,7 @@ func TestReputationHitAlertNamesTheSite(t *testing.T) {
loadLists(t, s.server, map[string][]string{dropURL: {client}})
wantDisposition(t, s.ask(client, "/"), http.StatusForbidden,
wafprotocol.DispositionDeny)
waf.DispositionDeny)
waiting := s.queue.Snapshot().Waiting[alerts.DestinationWebhook]
if len(waiting) != 1 || waiting[0].Event != alerts.EventReputationHit ||
-46
View File
@@ -1,46 +0,0 @@
package wafprotocol
import "net/http"
// Answer is what waf's answer to a question tells rproxy in its headers.
// The status, the other headers and the body of an answer of deny are what
// the visitor gets.
type Answer struct {
// Disposition is what rproxy does with the request: DispositionAllow,
// DispositionDeny or DispositionClose.
Disposition string
// ASN and Country are the client's AS number and country, each empty
// when the answer does not give it.
ASN, Country string
}
// SetAnswerHeaders sets the headers of waf's answer that tell rproxy answer,
// leaving out an ASN or Country that is empty.
func SetAnswerHeaders(header http.Header, answer Answer) {
header.Set(DispositionHeader, answer.Disposition)
if answer.ASN != "" {
header.Set(ASNHeader, answer.ASN)
}
if answer.Country != "" {
header.Set(CountryHeader, answer.Country)
}
}
// ReadAnswer returns what the headers of waf's answer tell rproxy, and
// whether they give a valid disposition.
func ReadAnswer(header http.Header) (Answer, bool) {
disposition := header.Get(DispositionHeader)
switch disposition {
case DispositionAllow, DispositionDeny, DispositionClose:
return Answer{
Disposition: disposition,
ASN: header.Get(ASNHeader),
Country: header.Get(CountryHeader),
}, true
}
return Answer{}, false
}
-189
View File
@@ -1,189 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net/http"
"net/netip"
"net/url"
"strconv"
"unicode/utf8"
)
// maxRequestIDBytes is the longest id a question may give.
const maxRequestIDBytes = 128
// ErrBodyStoppedArriving is the error ReadQuestion wraps when the question's
// body stops arriving. Every other error it returns is why waf refuses the
// question.
var ErrBodyStoppedArriving = errors.New("the body stopped arriving")
// Why waf refuses a question. Each is also the body of the refusal.
var (
errClientNotAnAddress = errors.New(
ClientHeader + " is not an address without a zone",
)
errRequestIDNotValid = errors.New(RequestIDHeader + " is not a valid id")
errBodyTooLarge = errors.New("the body is over 64 KiB")
)
// QuestionParams are what NewQuestion needs.
type QuestionParams struct {
// Visitor is the visitor's request.
Visitor *http.Request
// PathAsSent is Visitor's path as the visitor sent it, which the question
// sends unchanged.
PathAsSent string
// Body is the part of Visitor's body the question carries, at most
// QuestionBodyMaxBytes.
Body []byte
// WAFBackend is waf's address. The question takes its scheme and host.
WAFBackend *url.URL
// Client is the visitor's address, and Peer that of rproxy's own TCP
// peer.
Client, Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Scheme is how the visitor reached rproxy, or the trusted proxy in front
// of it.
Scheme string
}
// Question is a question as waf reads it.
type Question struct {
// Client is the visitor's address.
Client netip.Addr
// Peer is the address of rproxy's own TCP peer. It is not valid when the
// question gives none that can be read.
Peer netip.Addr
// RequestID is the request's id.
RequestID string
// Visitor is the visitor's request, as asTheVisitorSentIt gives it.
Visitor *http.Request
}
// NewQuestion returns the question about a visitor's request, as "The
// question" in SPEC.md gives it: the request as the visitor sent it, with the
// part of the body it carries and rproxy's headers.
func NewQuestion(
ctx context.Context,
params QuestionParams,
) (*http.Request, error) {
in := params.Visitor
question, err := http.NewRequestWithContext(
ctx,
in.Method,
params.WAFBackend.String(),
bytes.NewReader(params.Body),
)
if err != nil {
return nil, fmt.Errorf("make the question: %w", err)
}
backend := question.URL
target := *in.URL
target.Scheme = backend.Scheme
target.Host = backend.Host
target.Opaque = params.PathAsSent
question.URL = &target
question.Host = in.Host
// The length NewRequestWithContext set is that of the body carried. This
// line makes a body whose length the visitor did not announce go chunked,
// as "The question" in SPEC.md requires.
question.TransferEncoding = in.TransferEncoding
question.Header = in.Header.Clone()
question.Header.Set(ClientHeader, params.Client.String())
question.Header.Set(PeerHeader, params.Peer.String())
question.Header.Del(ContentLengthHeader)
announced := in.Header.Get("Content-Length")
if announced != "" {
question.Header.Set(ContentLengthHeader, announced)
}
question.Header.Set(RequestIDHeader, params.RequestID)
question.Header.Set("X-Forwarded-Proto", params.Scheme)
// Go's client adds a User-Agent of its own to a request without the key.
_, sent := question.Header["User-Agent"]
if !sent {
question.Header["User-Agent"] = nil
}
return question, nil
}
// ReadQuestion reads the question r, with the body it carries. An error
// wrapping ErrBodyStoppedArriving means the body stopped arriving. Any other
// error is why waf refuses the question, and the question's body is then not
// read. The body is read whole, since Go's server does not send an answer
// until it has read the body.
func ReadQuestion(w http.ResponseWriter, r *http.Request) (Question, error) {
client, err := netip.ParseAddr(r.Header.Get(ClientHeader))
if err != nil || client.Zone() != "" {
return Question{}, errClientNotAnAddress
}
id := r.Header.Get(RequestIDHeader)
if id == "" || len(id) > maxRequestIDBytes || !utf8.ValidString(id) {
return Question{}, errRequestIDNotValid
}
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, QuestionBodyMaxBytes),
)
if _, tooLarge := errors.AsType[*http.MaxBytesError](err); tooLarge {
return Question{}, errBodyTooLarge
}
if err != nil {
return Question{}, fmt.Errorf("%w: %w", ErrBodyStoppedArriving, err)
}
// One that cannot be read leaves the log line's peer_ip invalid.
peer, _ := netip.ParseAddr(r.Header.Get(PeerHeader))
return Question{
Client: client.Unmap(),
Peer: peer.Unmap(),
RequestID: id,
Visitor: asTheVisitorSentIt(r, body),
}, nil
}
// asTheVisitorSentIt returns a copy of r, a question, as the checks and the
// log line are to see it: without the headers rproxy adds for waf,
// with body, which has been read of r, and with the length the visitor
// announced, as -1 when it announced none. The question's own length covers
// only the part of the body it carries.
func asTheVisitorSentIt(r *http.Request, body []byte) *http.Request {
sent := r.Clone(r.Context())
announced := sent.Header.Get(ContentLengthHeader)
sent.Header.Del(ClientHeader)
sent.Header.Del(PeerHeader)
sent.Header.Del(ContentLengthHeader)
sent.Header.Del("Content-Length")
sent.ContentLength = -1
length, err := strconv.ParseInt(announced, 10, 64)
if err == nil && length >= 0 {
sent.ContentLength = length
sent.Header.Set("Content-Length", announced)
}
sent.Body = http.NoBody
if len(body) > 0 {
sent.Body = io.NopCloser(bytes.NewReader(body))
}
return sent
}
-123
View File
@@ -1,123 +0,0 @@
package wafprotocol
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
)
const (
// reportBodyMaxBytes is the most of a report's body that is read. It is
// far above any report's.
reportBodyMaxBytes = 1 << 20
// maxReportedBytes is the most bytes a report may give for each way, 1
// PiB, so that adding them up cannot overflow.
maxReportedBytes = 1 << 50
// maxReportedMilliseconds is the longest duration a report may give,
// about 31 years, so that it fits a time.Duration.
maxReportedMilliseconds = 1e12
)
var (
errReportFieldMissing = errors.New("a field is missing")
errReportValue = errors.New("a value is out of range")
)
// Report is the JSON object rproxy posts to ReportPath once the visitor's
// answer has ended, its fields named as the request log's of the same
// meaning. RequestID is the question's id. Status is what the visitor got, 0
// if nothing. The durations are in milliseconds.
//
//nolint:tagliatelle // docs/request-log.md names the fields in snake_case
type Report struct {
RequestID string `json:"request_id"`
Status int `json:"status"`
RequestBytes int64 `json:"request_bytes"`
ResponseBytes int64 `json:"response_bytes"`
DurationTotalMS float64 `json:"duration_total_ms"`
DurationUpstreamTotalMS float64 `json:"duration_upstream_total_ms"`
}
// NewReportRequest returns the request that posts report to reportURL, which
// is waf's ReportPath.
func NewReportRequest(
ctx context.Context,
reportURL string,
report *Report,
) (*http.Request, error) {
body, err := json.Marshal(report)
if err != nil {
return nil, fmt.Errorf("encode the report: %w", err)
}
req, err := http.NewRequestWithContext(
ctx, http.MethodPost, reportURL, bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("make the request: %w", err)
}
req.Header.Set("Content-Type", "application/json")
return req, nil
}
// ReadReport reads a report from r's body, which must be one JSON object with
// all of Report's fields, in the ranges SPEC.md gives.
func ReadReport(w http.ResponseWriter, r *http.Request) (Report, error) {
var report Report
body, err := io.ReadAll(
http.MaxBytesReader(w, r.Body, reportBodyMaxBytes),
)
if err != nil {
return report, fmt.Errorf("read the body: %w", err)
}
var fields map[string]json.RawMessage
err = json.Unmarshal(body, &fields)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
for _, name := range []string{
"request_id", "status", "request_bytes", "response_bytes",
"duration_total_ms", "duration_upstream_total_ms",
} {
value, there := fields[name]
if !there || string(value) == "null" {
return report, fmt.Errorf("%w: %s", errReportFieldMissing, name)
}
}
err = json.Unmarshal(body, &report)
if err != nil {
return report, fmt.Errorf("decode the body: %w", err)
}
return report, report.checkRanges()
}
// checkRanges returns an error for a value of the report that cannot be true.
func (report *Report) checkRanges() error {
switch {
case report.Status != 0 &&
(report.Status < http.StatusContinue || report.Status > ReportStatusMax):
return fmt.Errorf("%w: status", errReportValue)
case report.RequestBytes < 0 || report.RequestBytes > maxReportedBytes:
return fmt.Errorf("%w: request_bytes", errReportValue)
case report.ResponseBytes < 0 || report.ResponseBytes > maxReportedBytes:
return fmt.Errorf("%w: response_bytes", errReportValue)
case report.DurationTotalMS < 0 || report.DurationTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_total_ms", errReportValue)
case report.DurationUpstreamTotalMS < 0 ||
report.DurationUpstreamTotalMS > maxReportedMilliseconds:
return fmt.Errorf("%w: duration_upstream_total_ms", errReportValue)
}
return nil
}
-61
View File
@@ -1,61 +0,0 @@
// Package wafprotocol holds the three messages rproxy and waf send each
// other, as "`waf`" in SPEC.md gives them. rproxy asks waf a question about
// each visitor's request, and waf answers it. Once the visitor's answer has
// ended, rproxy sends waf a report. This package writes and reads each
// message, and declares every name and limit both sides must agree on.
package wafprotocol
// The headers of the question and of the answer. Go writes every header name
// in this form, as it sends it and as it receives it, so X-Client-ASN arrives
// as X-Client-Asn; header names are not case-sensitive.
const (
// ClientHeader marks a request as rproxy's question, and gives the
// visitor's address.
ClientHeader = "X-Sww-Client"
// PeerHeader gives the address of rproxy's own TCP peer.
PeerHeader = "X-Sww-Peer"
// ContentLengthHeader gives the body length the visitor announced.
ContentLengthHeader = "X-Sww-Content-Length"
// RequestIDHeader gives the request's id, which rproxy makes. rproxy
// also sets it on the request it passes to the app.
RequestIDHeader = "X-Request-ID"
// DispositionHeader gives waf's answer.
DispositionHeader = "X-Sww-Disposition"
// ASNHeader and CountryHeader give the client's AS number and country in
// an answer of allow. rproxy passes them to the app.
ASNHeader = "X-Client-Asn"
CountryHeader = "X-Client-Country"
)
// The values of DispositionHeader.
const (
// DispositionAllow tells rproxy to pass the request to the app.
DispositionAllow = "allow"
// DispositionDeny tells rproxy to give the visitor the answer's status,
// headers and body.
DispositionDeny = "deny"
// DispositionClose tells rproxy to close the visitor's connection
// without an answer, which SWW_WAF_BAN_RESPONSE asks for when it is close.
// waf closing the question's connection instead would read as
// waf being down, and rproxy would then let the request through.
DispositionClose = "close"
)
// OwnEndpointPrefix starts the path of every request for sww itself: waf's
// own endpoints and rproxy's health endpoint. The app never sees a path under
// it.
const OwnEndpointPrefix = "/_sww/"
// HealthPath is the health endpoint of waf and of rproxy, which the
// container's health check asks.
const HealthPath = "/_sww/healthz"
// ReportPath is where rproxy sends the report of each request waf
// answered.
const ReportPath = "/_sww/report"
// QuestionBodyMaxBytes is the most of a visitor's body a question carries.
const QuestionBodyMaxBytes = 64 << 10
// ReportStatusMax is the highest status a report may give.
const ReportStatusMax = 599
-204
View File
@@ -1,204 +0,0 @@
package wafprotocol_test
import (
"io"
"net/http"
"net/http/httptest"
"net/netip"
"net/url"
"strings"
"testing"
"sneak.berlin/go/smallwebwaf/internal/wafprotocol"
)
// requestID is the id the tests' questions and reports give.
const requestID = "an-id"
// questionRead is what the server startQuestionReader starts read of a
// question: the question, and the body it carries.
type questionRead struct {
question wafprotocol.Question
body string
}
// startQuestionReader starts a server that reads each question it gets with
// ReadQuestion. It returns the server's URL and the channel what it reads goes
// to.
func startQuestionReader(t *testing.T) (*url.URL, <-chan questionRead) {
t.Helper()
read := make(chan questionRead, 1)
server := httptest.NewServer(http.HandlerFunc(
func(w http.ResponseWriter, r *http.Request) {
question, err := wafprotocol.ReadQuestion(w, r)
if err != nil {
t.Errorf("read the question: %v", err)
}
var body []byte
if question.Visitor != nil {
body, _ = io.ReadAll(question.Visitor.Body)
}
read <- questionRead{question, string(body)}
},
))
t.Cleanup(server.Close)
serverURL, err := url.Parse(server.URL)
if err != nil {
t.Fatalf("parse the server's URL: %v", err)
}
return serverURL, read
}
// TestQuestionReadsAsWritten sends a question NewQuestion writes over HTTP,
// and checks that ReadQuestion gives back the visitor's request and what
// rproxy added.
func TestQuestionReadsAsWritten(t *testing.T) {
t.Parallel()
const (
target = "/a%2Fb?q=1"
body = "name=value"
held = "name"
)
client := netip.MustParseAddr("2001:db8::1")
peer := netip.MustParseAddr("192.0.2.1")
backend, read := startQuestionReader(t)
visitor := httptest.NewRequestWithContext(
t.Context(),
http.MethodPost,
"http://site.example.com"+target,
strings.NewReader(body),
)
visitor.Header.Set("Content-Length", "10")
visitor.Header.Set("User-Agent", "a visitor")
visitor.Header.Set(wafprotocol.PeerHeader, "the visitor's own")
question, err := wafprotocol.NewQuestion(
t.Context(),
wafprotocol.QuestionParams{
Visitor: visitor,
PathAsSent: "/a%2Fb",
Body: []byte(held),
WAFBackend: backend,
Client: client,
Peer: peer,
RequestID: requestID,
Scheme: "https",
},
)
if err != nil {
t.Fatalf("write the question: %v", err)
}
res, err := http.DefaultTransport.RoundTrip(question)
if err != nil {
t.Fatalf("send the question: %v", err)
}
_ = res.Body.Close()
got := <-read
if got.question.Visitor == nil {
t.FailNow()
}
if got.question.Client != client || got.question.Peer != peer ||
got.question.RequestID != requestID {
t.Errorf("read %+v", got.question)
}
sent := got.question.Visitor
if sent.Host != "site.example.com" || sent.RequestURI != target ||
sent.ContentLength != int64(len(body)) || got.body != held {
t.Errorf(
"read %s at %s, of length %d, carrying %q",
sent.RequestURI, sent.Host, sent.ContentLength, got.body,
)
}
for name, want := range map[string]string{
"Content-Length": "10",
"User-Agent": "a visitor",
"X-Forwarded-Proto": "https",
wafprotocol.RequestIDHeader: requestID,
wafprotocol.ClientHeader: "",
wafprotocol.PeerHeader: "",
wafprotocol.ContentLengthHeader: "",
} {
if sent.Header.Get(name) != want {
t.Errorf("%s is %q, want %q", name, sent.Header.Get(name), want)
}
}
}
// TestAnswerReadsAsWritten checks that ReadAnswer gives back what
// SetAnswerHeaders sets, and that headers without a valid disposition give
// nothing.
func TestAnswerReadsAsWritten(t *testing.T) {
t.Parallel()
for _, want := range []wafprotocol.Answer{
{Disposition: wafprotocol.DispositionAllow, ASN: "64496", Country: "DE"},
{Disposition: wafprotocol.DispositionAllow, Country: "DE"},
{Disposition: wafprotocol.DispositionDeny},
{Disposition: wafprotocol.DispositionClose},
} {
header := http.Header{}
wafprotocol.SetAnswerHeaders(header, want)
got, valid := wafprotocol.ReadAnswer(header)
if !valid || got != want {
t.Errorf("read %+v (valid: %t), want %+v", got, valid, want)
}
}
header := http.Header{}
header.Set(wafprotocol.DispositionHeader, "maybe")
header.Set(wafprotocol.ASNHeader, "64496")
got, valid := wafprotocol.ReadAnswer(header)
if valid || got != (wafprotocol.Answer{}) {
t.Errorf("read %+v (valid: %t), want nothing", got, valid)
}
}
// TestReportReadsAsWritten checks that ReadReport gives back the report
// NewReportRequest posts.
func TestReportReadsAsWritten(t *testing.T) {
t.Parallel()
want := wafprotocol.Report{
RequestID: requestID,
Status: http.StatusOK,
RequestBytes: 10,
ResponseBytes: 20,
DurationTotalMS: 1.5,
DurationUpstreamTotalMS: 0.5,
}
req, err := wafprotocol.NewReportRequest(
t.Context(),
"http://waf.example.com"+wafprotocol.ReportPath,
&want,
)
if err != nil {
t.Fatalf("write the report: %v", err)
}
if req.Method != http.MethodPost ||
req.URL.Path != wafprotocol.ReportPath {
t.Errorf("the report is %s %s", req.Method, req.URL.Path)
}
got, err := wafprotocol.ReadReport(httptest.NewRecorder(), req)
if err != nil || got != want {
t.Errorf("read %+v (%v), want %+v", got, err, want)
}
}